← Private benchmarks

Published 2026-11-16. Synthetic example tasks written by ChatJudge; no customer data. This is the report buyers receive, generated by the same code from a real run of the models (actual spend on this run: US$0.70 in model calls plus US$0.31 for grading, and US$0.37 for the grader cross-check below).

Download the report (PDF) Download (HTML) The example tasks (JSONL)

ChatJudge sample report on synthetic example tasks · run 2026-11-16 12:49 SGT

Sample report: six example tasks, ranked by quality, cost and speed

6 tasks × 3 models × 5 runs each, graded blind against what you said a good answer looks like. A run is successful when every hard rule passes and it meets at least 80% of the checks.

Recommendation

Best quality: a tie between GPT-6 Astra and Gemini 4 Carbon, which met 97.3% and 96.3% of your checks on average; the gap is within run-to-run noise. GPT-6 Astra succeeded on 30 of 30 runs (100%).

Best value: also GPT-6 Astra, at $0.0078 per successful task, counting the cost of failed attempts.

Fastest: GPT-6 Astra, with a median of 3.6 s per answer (9 in 10 answers within 4.8 s).

Claude Opus 5.5 succeeded on too few runs to be considered for value or speed (under 80% of the best pass rate).

Recommendation: GPT-6 Astra. It is at the top on quality (tied) and costs least per successful task, and it is also the fastest.

Best qualityGPT-6 Astra97.3% of checks met · 100% of runs successful
Best valueGPT-6 Astra$0.0078 per successful task
FastestGPT-6 Astramedian 3.6 s · p90 4.8 s

This applies to these tasks only. It is not a general ranking of the models.

Ranking

#ModelQuality (checks met)Successful runsCost per successful taskCost per attemptLatency, median / p90Same result every run
1GPT-6 Astra
OpenAI · gpt-6-astra
97.3%
runs ranged 80–100%
30/30 (100%)$0.0078$0.00783.6 s / 4.8 s6 of 6 tasks
2Gemini 4 Carbon
Google · gemini-4-carbon
96.3%
runs ranged 80–100%
30/30 (100%)$0.0083$0.008315 s / 27 s6 of 6 tasks
3Claude Opus 5.5
Anthropic · claude-opus-5-5
92.6%
runs ranged 50–100%
23/30 (77%)$0.0092$0.00713.9 s / 5.2 s3 of 6 tasks

Quality is the average, over your tasks, of the share of checks each answer met (every task counts equally). Cost per successful task = everything spent on that model's attempts, failed ones included, divided by the successful attempts: what you would pay per usable answer if you retried until one passed. Latency is the time for one full answer through the API gateway. "Same result every run" counts tasks where all 5 runs agreed on pass or fail.

Task by task

Each dot is one run, placed by the share of checks it met; filled dots succeeded. The gold line is your pass mark (80%).

1. Support reply: late refund

Checks: Includes “LM-20417” · Includes “Sam, Lumenly Support” · Does not include “guarantee” · At most 150 words · Apologises for the delay and for her having to write twice · Confirms the full US$480 refund was approved and sent to her original card · Says card refunds take 5 to 10 business days to appear · Does not promise a specific arrival date or anything the notes do not support · Polite, plain tone with no blame on the customer · Makes no factual claims that the supplied context does not support · Best here: GPT-6 Astra

ModelRunsSuccessfulQuality, mean (range)Cost per successfulMedian latencyMost-missed check
GPT-6 AstraRun 1: 90%Run 2: 90%Run 3: 80%Run 4: 80%Run 5: 80%5/584% (80–90)$0.00974.5 sDoes not promise a specific arrival date or anything the notes do not support (3×)
Gemini 4 CarbonRun 1: 80%Run 2: 80%Run 3: 80%Run 4: 80%Run 5: 80%5/580% (80–80)$0.01218 sDoes not promise a specific arrival date or anything the notes do not support (5×)
Claude Opus 5.5Run 1: 70%Run 2: 80%Run 3: 80%Run 4: 70%Run 5: 70%2/574% (70–80)$0.0285.3 sDoes not promise a specific arrival date or anything the notes do not support (5×)
GPT-6 Astra: a typical answer (80% of checks, successful)
Hi Priya,

I’m sorry you’ve had to contact us twice about your US$480 refund.

You cancelled within our 14-day refund window, so you qualified for a full refund. Your refund for order LM-20417 was approved on 12 November and sent to your original card.

Card refunds take 5–10 business days to appear, so it’s still within the expected processing window. I’m sorry we didn’t communicate the status sooner. If it hasn’t appeared after 10 business days, please reply here so we can look into it.

Sam, Lumenly Support

✓ Includes “LM-20417”
✓ Includes “Sam, Lumenly Support”
✓ Does not include “guarantee”
✓ At most 150 words
✓ Apologises for the delay and for her having to write twice
✓ Confirms the full US$480 refund was approved and sent to her original card
✓ Says card refunds take 5 to 10 business days to appear
✗ Does not promise a specific arrival date or anything the notes do not support — Claims Lumenly did not communicate the status sooner; the notes do not establish that.
✓ Polite, plain tone with no blame on the customer
✗ Makes no factual claims that the supplied context does not support — The claim that Lumenly did not communicate the status sooner is unsupported.

Unsupported claims flagged: Lumenly did not communicate the refund status sooner.

Gemini 4 Carbon: a typical answer (80% of checks, successful)
Subject: Update on your Lumenly refund (Order LM-20417)

Hi Priya,

I sincerely apologize for the delay and the frustration of having to reach out to us twice. 

Since you cancelled within our 14-day refund window, you qualified for a full refund. I have checked order LM-20417 and can confirm that your US$480 refund was approved and sent back to your original payment card on 12 November. 

Standard card refunds take 5 to 10 business days from the date they are issued to appear on your statement, depending on your bank's processing times. 

If you do not see the funds in your account after 10 business days from 12 November, please reply directly to this email and I will look into it immediately.

Best regards,

Sam, Lumenly Support

✓ Includes “LM-20417”
✓ Includes “Sam, Lumenly Support”
✓ Does not include “guarantee”
✓ At most 150 words
✓ Apologises for the delay and for her having to write twice
✓ Confirms the full US$480 refund was approved and sent to her original card
✓ Says card refunds take 5 to 10 business days to appear
✗ Does not promise a specific arrival date or anything the notes do not support — Promises to investigate immediately, a commitment not supported by the notes.
✓ Polite, plain tone with no blame on the customer
✗ Makes no factual claims that the supplied context does not support — Claims a personal order check and attributes refund timing to bank processing; neither is established by the context.

Unsupported claims flagged: The agent personally checked order LM-20417.; Refund timing depends on the bank's processing times.

Claude Opus 5.5: a typical answer (70% of checks, not successful)
Subject: Your refund for order LM-20417

Hi Priya,

I'm sorry you've had to write to us twice about this. You should have had a clear answer the first time.

Your cancellation on 3 November was within our 14-day refund window, so you qualify for a full refund. We approved and sent your US$480 refund to your original card on 12 November.

Card refunds usually take 5 to 10 business days to appear on your statement, depending on your bank. If the refund isn't on your statement by the end of that window, please reply to this email and I'll follow up with our payments team and your bank's details right away.

Thank you for your patience, and again, I'm sorry for the frustration.

Best regards,
Sam, Lumenly Support

✓ Includes “LM-20417”
✓ Includes “Sam, Lumenly Support”
✓ Does not include “guarantee”
✓ At most 150 words
✗ Apologises for the delay and for her having to write twice — Apologises for her having to write twice, but does not explicitly apologise for the refund delay.
✓ Confirms the full US$480 refund was approved and sent to her original card
✓ Says card refunds take 5 to 10 business days to appear
✗ Does not promise a specific arrival date or anything the notes do not support — Promises follow-up with a payments team and bank details, which the notes do not support.
✓ Polite, plain tone with no blame on the customer
✗ Makes no factual claims that the supplied context does not support — Adds unsupported details about bank-dependent timing, the first reply, and follow-up with a payments team and bank details.

Unsupported claims flagged: The first response failed to provide a clear answer.; Refund timing depends on Priya’s bank.; Sam can follow up with a payments team and Priya’s bank details right away.

2. SQL: revenue per customer

Checks: Includes “/\bjoin\b/i” · Includes “/limit\s+5/i” · Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i” · Filters to status = 'paid' · Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026) · Converts cents to dollars with 2 decimal places without integer division · Groups by customer so two customers with the same name are not merged · Orders by revenue descending and limits to 5 · Agrees with the reference answer on every point that matters (wording can differ) · Best here: Claude Opus 5.5

ModelRunsSuccessfulQuality, mean (range)Cost per successfulMedian latencyMost-missed check
GPT-6 AstraRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.0134.8 s–
Gemini 4 CarbonRun 1: 88.9%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/597.8% (88.9–100)$0.008423 sAgrees with the reference answer on every point that matters (wording can differ) (1×)
Claude Opus 5.5Run 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00893.8 s–
GPT-6 Astra: a typical answer (100% of checks, successful)
SELECT
    c.name,
    c.country,
    ROUND(SUM(o.total_cents)::numeric / 100, 2) AS revenue_dollars
FROM customers AS c
JOIN orders AS o ON o.customer_id = c.id
WHERE o.status = 'paid'
  AND o.created_at >= TIMESTAMPTZ '2025-01-01 00:00:00'
  AND o.created_at < TIMESTAMPTZ '2026-01-01 00:00:00'
GROUP BY c.id, c.name, c.country
ORDER BY SUM(o.total_cents) DESC
LIMIT 5;

✓ Includes “/\bjoin\b/i”
✓ Includes “/limit\s+5/i”
✓ Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i”
✓ Filters to status = 'paid'
✓ Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026)
✓ Converts cents to dollars with 2 decimal places without integer division
✓ Groups by customer so two customers with the same name are not merged
✓ Orders by revenue descending and limits to 5
✓ Agrees with the reference answer on every point that matters (wording can differ)

Gemini 4 Carbon: a typical answer (100% of checks, successful)
SELECT
    c.name,
    c.country,
    ROUND(SUM(o.total_cents) / 100.0, 2) AS revenue_dollars
FROM
    customers c
JOIN
    orders o ON c.id = o.customer_id
WHERE
    o.status = 'paid'
    AND o.created_at >= '2025-01-01'
    AND o.created_at < '2026-01-01'
GROUP BY
    c.id,
    c.name,
    c.country
ORDER BY
    revenue_dollars DESC
LIMIT 5;

✓ Includes “/\bjoin\b/i”
✓ Includes “/limit\s+5/i”
✓ Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i”
✓ Filters to status = 'paid'
✓ Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026)
✓ Converts cents to dollars with 2 decimal places without integer division
✓ Groups by customer so two customers with the same name are not merged
✓ Orders by revenue descending and limits to 5
✓ Agrees with the reference answer on every point that matters (wording can differ)

Claude Opus 5.5: a typical answer (100% of checks, successful)
```sql
SELECT
    c.name,
    c.country,
    ROUND(SUM(o.total_cents)::numeric / 100, 2) AS revenue_dollars
FROM customers c
JOIN orders o ON o.customer_id = c.id
WHERE o.status = 'paid'
  AND o.created_at >= TIMESTAMPTZ '2025-01-01 00:00:00'
  AND o.created_at <  TIMESTAMPTZ '2026-01-01 00:00:00'
GROUP BY c.id, c.name, c.country
ORDER BY SUM(o.total_cents) DESC, c.id
LIMIT 5;
```

✓ Includes “/\bjoin\b/i”
✓ Includes “/limit\s+5/i”
✓ Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i”
✓ Filters to status = 'paid'
✓ Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026)
✓ Converts cents to dollars with 2 decimal places without integer division
✓ Groups by customer so two customers with the same name are not merged
✓ Orders by revenue descending and limits to 5
✓ Agrees with the reference answer on every point that matters (wording can differ)

3. Summary: incident report

Checks: Includes “INC-311” · Exactly 3 bullet points · States the cause (connection limit cut from 2,000 to 200 by a load balancer change) · States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected) · Mentions both follow-ups with owners · Agrees with the reference answer on every point that matters (wording can differ) · Makes no factual claims that the supplied context does not support · Best here: GPT-6 Astra

ModelRunsSuccessfulQuality, mean (range)Cost per successfulMedian latencyMost-missed check
GPT-6 AstraRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.0123.6 s–
Gemini 4 CarbonRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.01824 s–
Claude Opus 5.5Run 1: 85.7%Run 2: 100%Run 3: 100%Run 4: 85.7%Run 5: 85.7%2/591.4% (85.7–100)$0.0294.7 sIncludes “INC-311” (3×)
GPT-6 Astra: a typical answer (100% of checks, successful)
- **Incident and impact:** INC-311 occurred on 4 November 2025, from 09:12 to 10:47 SGT. About 18% of Nimbus Store web checkout requests failed with HTTP 502 errors, resulting in an estimated 1,350 failed checkout attempts and 74 related support tickets. Mobile apps were unaffected; no payment data was exposed.
- **Cause and resolution:** A 09:05 load balancer configuration change reduced the connection limit from 2,000 to 200. It passed review because the diff showed the value without units. The on-call engineer rolled back the change at 10:41; error rates returned to normal by 10:47.
- **Follow-ups:** Infra team to add a minimum-value check for connection limits by 14 November; Platform team to show units in configuration diffs by 21 November.

✓ Includes “INC-311”
✓ Exactly 3 bullet points
✓ States the cause (connection limit cut from 2,000 to 200 by a load balancer change)
✓ States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected)
✓ Mentions both follow-ups with owners
✓ Agrees with the reference answer on every point that matters (wording can differ)
✓ Makes no factual claims that the supplied context does not support

Gemini 4 Carbon: a typical answer (100% of checks, successful)
* **Incident & Impact:** On Tuesday, 4 November 2025, from 09:12 to 10:47 SGT (Incident INC-311), about 18% of Nimbus Store web app checkout requests failed with HTTP 502 errors, resulting in an estimated 1,350 failed checkout attempts and 74 customer support tickets; mobile apps were not affected and no payment data was exposed.
* **Cause & Resolution:** The issue was caused by a 09:05 load balancer configuration change that reduced the connection limit from 2,000 to 200 (passing review because the diff showed the value without units), and was resolved when the on-call engineer rolled back the change at 10:41, returning error rates to normal by 10:47.
* **Follow-ups:** The Infra team is assigned to add a minimum-value check for connection limits by 14 November, and the Platform team is assigned to show units in config diffs by 21 November.

✓ Includes “INC-311”
✓ Exactly 3 bullet points
✓ States the cause (connection limit cut from 2,000 to 200 by a load balancer change)
✓ States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected)
✓ Mentions both follow-ups with owners
✓ Agrees with the reference answer on every point that matters (wording can differ)
✓ Makes no factual claims that the supplied context does not support

Claude Opus 5.5: a typical answer (85.7% of checks, not successful)
- **What happened:** On Tuesday 4 November 2025, from 09:12 to 10:47 SGT, about 18% of checkout requests on the Nimbus Store web app failed with HTTP 502 errors. This caused an estimated 1,350 failed checkout attempts and 74 related customer support tickets. Mobile apps were not affected, and no payment data was exposed.
- **Cause and resolution:** At 09:05, a load balancer configuration change reduced the connection limit from 2,000 to 200. It passed review because the diff showed the value without units. The on-call engineer rolled back the change at 10:41, and error rates returned to normal by 10:47.
- **Follow-ups:** The Infra team will add a minimum-value check for connection limits (due 14 November). The Platform team will show units in config diffs (due 21 November).

✗ Includes “INC-311”
✓ Exactly 3 bullet points
✓ States the cause (connection limit cut from 2,000 to 200 by a load balancer change)
✓ States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected)
✓ Mentions both follow-ups with owners
✓ Agrees with the reference answer on every point that matters (wording can differ)
✓ Makes no factual claims that the supplied context does not support

4. Classify: support ticket

Checks: Answer is exactly “bug” · Best here: Claude Opus 5.5

ModelRunsSuccessfulQuality, mean (range)Cost per successfulMedian latencyMost-missed check
GPT-6 AstraRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00101.7 s–
Gemini 4 CarbonRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00187.0 s–
Claude Opus 5.5Run 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00061.4 s–
GPT-6 Astra: a typical answer (100% of checks, successful)
bug

✓ Answer is exactly “bug”

Gemini 4 Carbon: a typical answer (100% of checks, successful)
bug

✓ Answer is exactly “bug”

Claude Opus 5.5: a typical answer (100% of checks, successful)
bug

✓ Answer is exactly “bug”

5. Extract: purchase order to JSON

Checks: Includes “PO-008812” · Valid JSON · po_number is "PO-008812" with the leading zeros · order_date is 2025-10-07 · total is 2016.5 (including GST), not 1850 · line_count is 3 · currency is SGD · Agrees with the reference answer on every point that matters (wording can differ) · Best here: Claude Opus 5.5

ModelRunsSuccessfulQuality, mean (range)Cost per successfulMedian latencyMost-missed check
GPT-6 AstraRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00541.9 s–
Gemini 4 CarbonRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.007112 s–
Claude Opus 5.5Run 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00371.7 s–
GPT-6 Astra: a typical answer (100% of checks, successful)
{"po_number":"PO-008812","supplier":"Harbourline Packaging Pte. Ltd.","order_date":"2025-10-07","currency":"SGD","total":2016.50,"line_count":3}

✓ Includes “PO-008812”
✓ Valid JSON
✓ po_number is "PO-008812" with the leading zeros
✓ order_date is 2025-10-07
✓ total is 2016.5 (including GST), not 1850
✓ line_count is 3
✓ currency is SGD
✓ Agrees with the reference answer on every point that matters (wording can differ)

Gemini 4 Carbon: a typical answer (100% of checks, successful)
{
  "po_number": "PO-008812",
  "supplier": "Harbourline Packaging Pte. Ltd.",
  "order_date": "2025-10-07",
  "currency": "SGD",
  "total": 2016.50,
  "line_count": 3
}

✓ Includes “PO-008812”
✓ Valid JSON
✓ po_number is "PO-008812" with the leading zeros
✓ order_date is 2025-10-07
✓ total is 2016.5 (including GST), not 1850
✓ line_count is 3
✓ currency is SGD
✓ Agrees with the reference answer on every point that matters (wording can differ)

Claude Opus 5.5: a typical answer (100% of checks, successful)
```json
{
  "po_number": "PO-008812",
  "supplier": "Harbourline Packaging Pte. Ltd.",
  "order_date": "2025-10-07",
  "currency": "SGD",
  "total": 2016.50,
  "line_count": 3
}
```

✓ Includes “PO-008812”
✓ Valid JSON
✓ po_number is "PO-008812" with the leading zeros
✓ order_date is 2025-10-07
✓ total is 2016.5 (including GST), not 1850
✓ line_count is 3
✓ currency is SGD
✓ Agrees with the reference answer on every point that matters (wording can differ)

6. Pushback (beta): missing information beta: pushback

Checks: Does not invent a region name or a growth figure · Points out that the task is impossible, contradictory or missing information, instead of inventing an answer · Best here: Gemini 4 Carbon

ModelRunsSuccessfulQuality, mean (range)Cost per successfulMedian latencyMost-missed check
GPT-6 AstraRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00523.4 s–
Gemini 4 CarbonRun 1: 100%Run 2: 100%Run 3: 100%Run 4: 100%Run 5: 100%5/5100% (100–100)$0.00247.9 s–
Claude Opus 5.5Run 1: 100%Run 2: 100%Run 3: 50%Run 4: 100%Run 5: 100%4/590% (50–100)$0.00814.1 sDoes not invent a region name or a growth figure (1×)
GPT-6 Astra: a typical answer (100% of checks, successful)
I don’t see an attached spreadsheet. Please upload it with Q2 and Q3 sales by region, and I’ll identify which region had the highest quarter-on-quarter percentage growth and report both the percentage and sales increase.

✓ Does not invent a region name or a growth figure
✓ Points out that the task is impossible, contradictory or missing information, instead of inventing an answer

Gemini 4 Carbon: a typical answer (100% of checks, successful)
It looks like you forgot to attach or paste the Q3 sales spreadsheet! 

Please upload the file or paste the data (specifically the Q2 and Q3 sales figures broken down by region), and I will gladly calculate which region had the biggest quarter-on-quarter growth for you.

✓ Does not invent a region name or a growth figure
✓ Points out that the task is impossible, contradictory or missing information, instead of inventing an answer

Claude Opus 5.5: a typical answer (100% of checks, successful)
I don't see a spreadsheet attached to your message. Only text came through on my end. Could you try uploading it again, or paste the data directly into the chat?

To calculate quarter-on-quarter growth, I'll need sales figures for **both Q2 and Q3** for each region. If the file only has Q3, please include the Q2 numbers too.

Once I have the data, I'll calculate growth for each region as:

**QoQ Growth (%) = (Q3 Sales − Q2 Sales) ÷ Q2 Sales × 100**

Then I'll identify the region with the biggest increase, in both percentage and absolute terms, since those can point to different regions.

✓ Does not invent a region name or a growth figure
✓ Points out that the task is impossible, contradictory or missing information, instead of inventing an answer

Hallucination and pushback checks beta

Graded by the same grading model, so treat these as indicators. Pushback tasks are ones you marked as impossible or missing information; a good answer says so instead of inventing one. Grounding applies to tasks where you supplied context.

ModelPushed back when it shouldStayed within the contextUnsupported claims flagged
GPT-6 Astra5/5 runs7/10 runs3
Gemini 4 Carbon5/5 runs5/10 runs9
Claude Opus 5.55/5 runs5/10 runs12

Method

ChatJudge · chatjudge.ai · Advark Tech Pte Ltd, Singapore · hello@advark.ai

Grader cross-check

To see how much the result depends on the grading model, we re-graded every graded answer in this sample with a second grader from a different provider, Claude Sonnet 5.5 (claude-sonnet-5-5). The two graders gave the same verdict on 371 of 390 checklist items (95%) and the same successful / not successful result on 71 of 75 answers.

ModelQuality, main graderQuality, second graderSuccessful, main graderSuccessful, second grader
Claude Opus 5.592.6%95.6%77%90%
GPT-6 Astra97.3%97.7%100%100%
Gemini 4 Carbon96.3%98%100%100%

Order by quality: main grader GPT-6 Astra > Gemini 4 Carbon > Claude Opus 5.5; second grader Gemini 4 Carbon > GPT-6 Astra > Claude Opus 5.5. Re-grading cost US$0.37.

Create my benchmark Start with my own task