DeepSWE v1.1
DataCurve / Coding · Changes repositories and fixes bugs in a common assistant environment to compare the underlying coding models.
What it measures and how
Only pass@1 is used. A representative configuration is selected within the common harness using predetermined reasoning-tier priorities.
How this evidence is used
Receives 3.6% of the coding-and-design budget with the controlled system and version fixed.
Evaluation results
Fixed rules select one representative configuration per public model. Anonymous test models are omitted. Original ranks are retained, with up to 30 models per evaluation.
| Source rank | Source model name | Raw result | Representative configuration |
|---|---|---|---|
| 2 | gemini-3-8-flashGoogle | 73.8% | High reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 3 | claude-opus-5Anthropic | 73.6% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 4 | gpt-6-astraOpenAI | 73.2% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 9 | gpt-5-6-solOpenAI | 72.7% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 13 | claude-fable-5Anthropic | 69.7% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 14 | gpt-5-6-terraOpenAI | 69.6% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 16 | glm-5-3Z.ai | 69.0% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 19 | kimi-k3Moonshot AI | 68.5% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 21 | gpt-5-6-lunaOpenAI | 67.2% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 22 | gpt-5-5OpenAI | 67.0% | xHigh reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 24 | grok-4-6xAI | 66.7% | xHigh reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 27 | gemini-3-7-flashGoogle | 65.3% | High reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 30 | glm-5-3-flashZ.ai | 63.4% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 31 | deepseek-v4-proDeepSeek | 62.8% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 35 | claude-opus-4-8Anthropic | 59.0% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 37 | qwen3-8-maxAlibaba | 57.5% | xHigh reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 39 | muse-spark-1-2Meta | 54.9% | xHigh reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 42 | claude-sonnet-5Anthropic | 53.8% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 43 | grok-4-5xAI | 53.8% | High reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 46 | deepseek-v4-flashDeepSeek | 53.3% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 46 | muse-spark-1-1Meta | 53.3% | xHigh reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 48 | gpt-5-4OpenAI | 51.8% | xHigh reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 53 | gemini-3-6-flashGoogle | 46.7% | High reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 56 | glm-5-2Z.ai | 43.8% | Max reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 61 | gemini-3-5-flashGoogle | 36.1% | High reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 63 | kimi-k2-7-codeMoonshot AI | 30.5% | Full system · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 65 | claude-sonnet-4-6Anthropic | 29.9% | High reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 68 | gemini-3-1-pro-previewGoogle | 11.7% | High reasoning · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
Evaluation limits & attribution
Measures the combined model and mini-swe-agent system. Major version changes require separate snapshots.
Data license: Official structured results are public. Private administrator observation only, pending license review before public display.
Results are published by DataCurve. Original scores and the NexGateHub consensus index use different scales and cannot be added directly.