Share promptsSign inCreate team library
← All benchmarks

Coding AI benchmarks

Fixing real GitHub issues and solving fresh programming problems. 2 public leaderboards below.

SWE-bench Verified

Fixing real GitHub issues in open-source Python projects; a fix counts if the project tests pass.

Run by Vals AI (benchmark by Princeton/OpenAI). Leaderboard as of 1 Sep 2026 · retrieved 21 Oct 2026. Showing top 25 of 88 models listed.

  • Anthropic
  • OpenAI
  • Google
  • xAI
  • Meta
  • DeepSeek
  • Alibaba
  • Z.ai
  • Moonshot
  • Other
SWE-bench Verified: Resolved vs Cost per test (USD)Scatter plot of 24 models. Leader on Resolved: Claude Opus 5 at 97.0%. Full values in the table below.$.01$.1$1$1075%80%85%90%95%100%Cost per test (USD) (log scale) →↑ ResolvedGemini 3.8 Flash (Google): Cost per test (USD) $2.19, Resolved 80.0%GGemini 3.7 Flash (Google): Cost per test (USD) $1.44, Resolved 80.8%GClaude Opus 4.7 (Anthropic): Cost per test (USD) $2.42, Resolved 82.0%AMuse Spark 1.1 (Meta): Cost per test (USD) $0.35, Resolved 82.0%MInkling Small (Other): Cost per test (USD) $0.13, Resolved 82.2%·GPT 5.5 (OpenAI): Cost per test (USD) $1.36, Resolved 82.6%OGLM 5.2 (Z.ai): Cost per test (USD) $0.71, Resolved 82.8%ZQwen 3.8 Max (Alibaba): Cost per test (USD) $1.13, Resolved 85.6%QClaude Opus 4.8 (Anthropic): Cost per test (USD) $0.67, Resolved 85.8%AQwen 3.8 27B (Alibaba): Cost per test (USD) $1.08, Resolved 86.0%QMuse Spark 1.2 (Meta): Cost per test (USD) $0.55, Resolved 86.6%MGrok 4.5 (xAI): Cost per test (USD) $0.54, Resolved 86.6%XClaude Opus 4.8 (Anthropic): Cost per test (USD) $1.92, Resolved 88.6%ADeepSeek V4 Flash 0731 (DeepSeek): Cost per test (USD) $0.010, Resolved 88.8%DGLM 5.3 Flash (Z.ai): Cost per test (USD) $0.020, Resolved 92.0%ZGPT-5.6 Luna (OpenAI): Cost per test (USD) $0.040, Resolved 93.0%OKimi K3 (Moonshot): Cost per test (USD) $0.76, Resolved 93.4%KClaude Fable 5 (Anthropic): Cost per test (USD) $2.05, Resolved 95.0%AGLM 5.3 (Z.ai): Cost per test (USD) $0.34, Resolved 95.4%ZGPT-5.6 Terra (OpenAI): Cost per test (USD) $0.40, Resolved 95.4%OGrok 4.6 (xAI): Cost per test (USD) $0.78, Resolved 95.6%XGPT-5.6 Sol (OpenAI): Cost per test (USD) $1.15, Resolved 96.2%ODeepSeek V4 Pro 0813 (DeepSeek): Cost per test (USD) $0.10, Resolved 96.4%DClaude Opus 5 (Anthropic): Cost per test (USD) $1.29, Resolved 97.0%AClaude Opus 5DeepSeek V4 Pro 0813GPT-5.6 Sol
SWE-bench Verified: top 15 by ResolvedRanked bars. Leader: Claude Opus 5 at 97.0%. Full values in the table below.0%20%40%60%80%100%Claude Opus 5 (Anthropic): 97.0%AClaude Opus 597.0%DeepSeek V4 Pro 0813 (DeepSeek): 96.4%DDeepSeek V4 Pro 081396.4%GPT-5.6 Sol (OpenAI): 96.2%OGPT-5.6 Sol96.2%Grok 4.6 (xAI): 95.6%XGrok 4.695.6%GPT-5.6 Terra (OpenAI): 95.4%OGPT-5.6 Terra95.4%GLM 5.3 (Z.ai): 95.4%ZGLM 5.395.4%Claude Fable 5 (Anthropic): 95.0%AClaude Fable 595.0%Kimi K3 (Moonshot): 93.4%KKimi K393.4%GPT-5.6 Luna (OpenAI): 93.0%OGPT-5.6 Luna93.0%GLM 5.3 Flash (Z.ai): 92.0%ZGLM 5.3 Flash92.0%DeepSeek V4 Flash 0731 (DeepSeek): 88.8%DDeepSeek V4 Flash 073188.8%Claude Opus 4.8 (Anthropic): 88.6%AClaude Opus 4.888.6%Grok 4.5 (xAI): 86.6%XGrok 4.586.6%Muse Spark 1.2 (Meta): 86.6%MMuse Spark 1.286.6%Qwen 3.8 27B (Alibaba): 86.0%QQwen 3.8 27B86.0%
Resolved
Share of issues fixed so that the hidden tests pass.
Cost per test (USD)
Average API cost per issue, as published.
SWE-bench Verified scores
RankModelProviderResolvedCost per test (USD)HarnessPrice in / out ($/1M tokens)
1 Claude Opus 5Anthropic97.0%$1.29Mini-SWE-agent$5 / $25
2 DeepSeek V4 Pro 0813DeepSeek96.4%$0.10Mini-SWE-agent$1.32 / $3.96
3 GPT-5.6 SolOpenAI96.2%$1.15Mini-SWE-agent$4 / $20
4 Grok 4.6xAI95.6%$0.78Mini-SWE-agent$2 / $6
5 GPT-5.6 TerraOpenAI95.4%$0.40Mini-SWE-agent$2 / $12
6 GLM 5.3Z.ai95.4%$0.34Mini-SWE-agent$1.4 / $4.4
7 Claude Fable 5Anthropic95.0%$2.05Mini-SWE-agent$10 / $50
8 Kimi K3Moonshot93.4%$0.76Mini-SWE-agent$3 / $15
9 GPT-5.6 LunaOpenAI93.0%$0.040Mini-SWE-agent$0.2 / $1.2
10 GLM 5.3 FlashZ.ai92.0%$0.020Mini-SWE-agent$0.075 / $0.25
Show 15 more rows
RankModelProviderResolvedCost per test (USD)HarnessPrice in / out ($/1M tokens)
11 DeepSeek V4 Flash 0731DeepSeek88.8%$0.010Mini-SWE-agent$0.44 / $1.32
12 Claude Opus 4.8Anthropic88.6%$1.92Mini-SWE-agent$5 / $25
13 Grok 4.5xAI86.6%$0.54Mini-SWE-agent$2 / $6
14 Muse Spark 1.2Meta86.6%$0.55Mini-SWE-agent$1.25 / $4.25
15 Qwen 3.8 27BAlibaba86.0%$1.08Mini-SWE-agent$0.5 / $3
16 Claude Opus 4.8Anthropic85.8%$0.67Claude CodeN/A
17 Qwen 3.8 MaxAlibaba85.6%$1.13Mini-SWE-agent$2 / $6
18 GLM 5.2Z.ai82.8%$0.71Mini-SWE-agent$1.4 / $4.4
19 GPT 5.5OpenAI82.6%$1.36Mini-SWE-agent$5 / $30
20 Inkling SmallOther82.2%$0.13Mini-SWE-agent$0.3 / $1.2
21 Muse Spark 1.1Meta82.0%$0.35Mini-SWE-agent$1.25 / $4.25
22 Claude Opus 4.7Anthropic82.0%$2.42Mini-SWE-agent$5 / $25
23 Gemini 3.7 FlashGoogle80.8%$1.44Mini-SWE-agent$1.5 / $7.5
24 Gemini 3.8 FlashGoogle80.0%$2.19Mini-SWE-agent$1.5 / $7.5
25 Composer 2.5Other79.6%–Cursor CLI$0.5 / $2.5

Source: Vals AI leaderboard ↗ · SWE-bench project ↗

Top model here: Claude Opus 5 (Anthropic). Try prompts for Claude →

LiveCodeBench

Recent competitive-programming problems, collected after model training cut-offs to limit memorisation.

Run by Vals AI (benchmark by LiveCodeBench team). Leaderboard as of 1 Sep 2026 · retrieved 21 Oct 2026. Showing top 25 of 143 models listed.

  • Anthropic
  • OpenAI
  • Google
  • xAI
  • DeepSeek
  • Alibaba
  • Moonshot
  • Other
LiveCodeBench: top 15 by AccuracyRanked bars. Leader: Claude Fable 5.1 at 90.52%. Full values in the table below.0%20%40%60%80%100%Claude Fable 5.1 (Anthropic): 90.52%AClaude Fable 5.190.52%Claude Fable 5 (Anthropic): 89.78%AClaude Fable 589.78%Gemini 3.8 Flash (Google): 89.48%GGemini 3.8 Flash89.48%Claude Opus 5 (Anthropic): 89.03%AClaude Opus 589.03%Gemini 3.7 Flash (Google): 88.65%GGemini 3.7 Flash88.65%Gemini 3.1 Pro Preview (02/26) (Google): 88.48%GGemini 3.1 Pro Preview (02/26)88.48%Grok 4.6 (xAI): 88.22%XGrok 4.688.22%Gemini 3.6 Flash (Google): 88.08%GGemini 3.6 Flash88.08%GPT 5.2 Codex (OpenAI): 87.99%OGPT 5.2 Codex87.99%Qwen 3.8 Max (Alibaba): 87.85%QQwen 3.8 Max87.85%Claude Opus 4.8 (Anthropic): 87.82%AClaude Opus 4.887.82%Gemini 3.5 Flash (Google): 87.60%GGemini 3.5 Flash87.60%DeepSeek V4 Pro 0813 (DeepSeek): 87.53%DDeepSeek V4 Pro 081387.53%DeepSeek V4 (DeepSeek): 87.48%DDeepSeek V487.48%Grok 4.5 (xAI): 87.35%XGrok 4.587.35%
Accuracy
Share of problems solved (all tests pass).
LiveCodeBench scores
RankModelProviderAccuracyPrice in / out ($/1M tokens)
1 Claude Fable 5.1Anthropic90.52%$10 / $50
2 Claude Fable 5Anthropic89.78%$10 / $50
3 Gemini 3.8 FlashGoogle89.48%$1.5 / $7.5
4 Claude Opus 5Anthropic89.03%$5 / $25
5 Gemini 3.7 FlashGoogle88.65%$1.5 / $7.5
6 Gemini 3.1 Pro Preview (02/26)Google88.48%$2 / $12
7 Grok 4.6xAI88.22%$2 / $6
8 Gemini 3.6 FlashGoogle88.08%$1.5 / $7.5
9 GPT 5.2 CodexOpenAI87.99%$1.75 / $14
10 Qwen 3.8 MaxAlibaba87.85%$2 / $6
Show 15 more rows
RankModelProviderAccuracyPrice in / out ($/1M tokens)
11 Claude Opus 4.8Anthropic87.82%$5 / $25
12 Gemini 3.5 FlashGoogle87.60%$1.5 / $9
13 DeepSeek V4 Pro 0813DeepSeek87.53%$1.32 / $3.96
14 DeepSeek V4DeepSeek87.48%$1.32 / $3.96
15 Grok 4.5xAI87.35%$2 / $6
16 GPT 5.3 CodexOpenAI87.31%$1.75 / $14
17 DeepSeek V4 Flash 0731DeepSeek87.26%$0.44 / $1.32
18 Kimi K3Moonshot87.19%$3 / $15
19 Qwen 3.7 MaxAlibaba87.06%$2.5 / $7.5
20 Kimi K2.6Moonshot86.77%$0.95 / $4
21 GPT 5 MiniOpenAI86.61%$0.25 / $2
22 GPT 5.1OpenAI86.49%$1.25 / $10
23 Gemini 3 Pro (11/25)Google86.41%$2 / $12
24 Nemotron 3 UltraOther85.98%N/A
25 Qwen 3.6 PlusAlibaba85.95%$0.5 / $3

Source: Vals AI leaderboard ↗ · LiveCodeBench project ↗

Top model here: Claude Fable 5.1 (Anthropic). Try prompts for Claude →

ChatJudge does not run these tests. Scores are copied as published by each source, with a link and the date we read them, and only change when we refresh from the source. Different leaderboards use different settings, so compare models within one chart, not across charts. Spotted a mistake? Tell us.