ChatJudge sample report on synthetic example tasks · run 2026-11-16 12:49 SGT
6 tasks × 3 models × 5 runs each, graded blind against what you said a good answer looks like. A run is successful when every hard rule passes and it meets at least 80% of the checks.
Best quality: a tie between GPT-6 Astra and Gemini 4 Carbon, which met 97.3% and 96.3% of your checks on average; the gap is within run-to-run noise. GPT-6 Astra succeeded on 30 of 30 runs (100%).
Best value: also GPT-6 Astra, at $0.0078 per successful task, counting the cost of failed attempts.
Fastest: GPT-6 Astra, with a median of 3.6 s per answer (9 in 10 answers within 4.8 s).
Claude Opus 5.5 succeeded on too few runs to be considered for value or speed (under 80% of the best pass rate).
Recommendation: GPT-6 Astra. It is at the top on quality (tied) and costs least per successful task, and it is also the fastest.
This applies to these tasks only. It is not a general ranking of the models.
| # | Model | Quality (checks met) | Successful runs | Cost per successful task | Cost per attempt | Latency, median / p90 | Same result every run |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra OpenAI · gpt-6-astra | 97.3% runs ranged 80–100% | 30/30 (100%) | $0.0078 | $0.0078 | 3.6 s / 4.8 s | 6 of 6 tasks |
| 2 | Gemini 4 Carbon Google · gemini-4-carbon | 96.3% runs ranged 80–100% | 30/30 (100%) | $0.0083 | $0.0083 | 15 s / 27 s | 6 of 6 tasks |
| 3 | Claude Opus 5.5 Anthropic · claude-opus-5-5 | 92.6% runs ranged 50–100% | 23/30 (77%) | $0.0092 | $0.0071 | 3.9 s / 5.2 s | 3 of 6 tasks |
Quality is the average, over your tasks, of the share of checks each answer met (every task counts equally). Cost per successful task = everything spent on that model's attempts, failed ones included, divided by the successful attempts: what you would pay per usable answer if you retried until one passed. Latency is the time for one full answer through the API gateway. "Same result every run" counts tasks where all 5 runs agreed on pass or fail.
Each dot is one run, placed by the share of checks it met; filled dots succeeded. The gold line is your pass mark (80%).
Checks: Includes “LM-20417” · Includes “Sam, Lumenly Support” · Does not include “guarantee” · At most 150 words · Apologises for the delay and for her having to write twice · Confirms the full US$480 refund was approved and sent to her original card · Says card refunds take 5 to 10 business days to appear · Does not promise a specific arrival date or anything the notes do not support · Polite, plain tone with no blame on the customer · Makes no factual claims that the supplied context does not support · Best here: GPT-6 Astra
| Model | Runs | Successful | Quality, mean (range) | Cost per successful | Median latency | Most-missed check |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 5/5 | 84% (80–90) | $0.0097 | 4.5 s | Does not promise a specific arrival date or anything the notes do not support (3×) | |
| Gemini 4 Carbon | 5/5 | 80% (80–80) | $0.012 | 18 s | Does not promise a specific arrival date or anything the notes do not support (5×) | |
| Claude Opus 5.5 | 2/5 | 74% (70–80) | $0.028 | 5.3 s | Does not promise a specific arrival date or anything the notes do not support (5×) |
Hi Priya, I’m sorry you’ve had to contact us twice about your US$480 refund. You cancelled within our 14-day refund window, so you qualified for a full refund. Your refund for order LM-20417 was approved on 12 November and sent to your original card. Card refunds take 5–10 business days to appear, so it’s still within the expected processing window. I’m sorry we didn’t communicate the status sooner. If it hasn’t appeared after 10 business days, please reply here so we can look into it. Sam, Lumenly Support
✓ Includes “LM-20417”
✓ Includes “Sam, Lumenly Support”
✓ Does not include “guarantee”
✓ At most 150 words
✓ Apologises for the delay and for her having to write twice
✓ Confirms the full US$480 refund was approved and sent to her original card
✓ Says card refunds take 5 to 10 business days to appear
✗ Does not promise a specific arrival date or anything the notes do not support — Claims Lumenly did not communicate the status sooner; the notes do not establish that.
✓ Polite, plain tone with no blame on the customer
✗ Makes no factual claims that the supplied context does not support — The claim that Lumenly did not communicate the status sooner is unsupported.
Unsupported claims flagged: Lumenly did not communicate the refund status sooner.
Subject: Update on your Lumenly refund (Order LM-20417) Hi Priya, I sincerely apologize for the delay and the frustration of having to reach out to us twice. Since you cancelled within our 14-day refund window, you qualified for a full refund. I have checked order LM-20417 and can confirm that your US$480 refund was approved and sent back to your original payment card on 12 November. Standard card refunds take 5 to 10 business days from the date they are issued to appear on your statement, depending on your bank's processing times. If you do not see the funds in your account after 10 business days from 12 November, please reply directly to this email and I will look into it immediately. Best regards, Sam, Lumenly Support
✓ Includes “LM-20417”
✓ Includes “Sam, Lumenly Support”
✓ Does not include “guarantee”
✓ At most 150 words
✓ Apologises for the delay and for her having to write twice
✓ Confirms the full US$480 refund was approved and sent to her original card
✓ Says card refunds take 5 to 10 business days to appear
✗ Does not promise a specific arrival date or anything the notes do not support — Promises to investigate immediately, a commitment not supported by the notes.
✓ Polite, plain tone with no blame on the customer
✗ Makes no factual claims that the supplied context does not support — Claims a personal order check and attributes refund timing to bank processing; neither is established by the context.
Unsupported claims flagged: The agent personally checked order LM-20417.; Refund timing depends on the bank's processing times.
Subject: Your refund for order LM-20417 Hi Priya, I'm sorry you've had to write to us twice about this. You should have had a clear answer the first time. Your cancellation on 3 November was within our 14-day refund window, so you qualify for a full refund. We approved and sent your US$480 refund to your original card on 12 November. Card refunds usually take 5 to 10 business days to appear on your statement, depending on your bank. If the refund isn't on your statement by the end of that window, please reply to this email and I'll follow up with our payments team and your bank's details right away. Thank you for your patience, and again, I'm sorry for the frustration. Best regards, Sam, Lumenly Support
✓ Includes “LM-20417”
✓ Includes “Sam, Lumenly Support”
✓ Does not include “guarantee”
✓ At most 150 words
✗ Apologises for the delay and for her having to write twice — Apologises for her having to write twice, but does not explicitly apologise for the refund delay.
✓ Confirms the full US$480 refund was approved and sent to her original card
✓ Says card refunds take 5 to 10 business days to appear
✗ Does not promise a specific arrival date or anything the notes do not support — Promises follow-up with a payments team and bank details, which the notes do not support.
✓ Polite, plain tone with no blame on the customer
✗ Makes no factual claims that the supplied context does not support — Adds unsupported details about bank-dependent timing, the first reply, and follow-up with a payments team and bank details.
Unsupported claims flagged: The first response failed to provide a clear answer.; Refund timing depends on Priya’s bank.; Sam can follow up with a payments team and Priya’s bank details right away.
Checks: Includes “/\bjoin\b/i” · Includes “/limit\s+5/i” · Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i” · Filters to status = 'paid' · Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026) · Converts cents to dollars with 2 decimal places without integer division · Groups by customer so two customers with the same name are not merged · Orders by revenue descending and limits to 5 · Agrees with the reference answer on every point that matters (wording can differ) · Best here: Claude Opus 5.5
| Model | Runs | Successful | Quality, mean (range) | Cost per successful | Median latency | Most-missed check |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 5/5 | 100% (100–100) | $0.013 | 4.8 s | – | |
| Gemini 4 Carbon | 5/5 | 97.8% (88.9–100) | $0.0084 | 23 s | Agrees with the reference answer on every point that matters (wording can differ) (1×) | |
| Claude Opus 5.5 | 5/5 | 100% (100–100) | $0.0089 | 3.8 s | – |
SELECT
c.name,
c.country,
ROUND(SUM(o.total_cents)::numeric / 100, 2) AS revenue_dollars
FROM customers AS c
JOIN orders AS o ON o.customer_id = c.id
WHERE o.status = 'paid'
AND o.created_at >= TIMESTAMPTZ '2025-01-01 00:00:00'
AND o.created_at < TIMESTAMPTZ '2026-01-01 00:00:00'
GROUP BY c.id, c.name, c.country
ORDER BY SUM(o.total_cents) DESC
LIMIT 5;✓ Includes “/\bjoin\b/i”
✓ Includes “/limit\s+5/i”
✓ Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i”
✓ Filters to status = 'paid'
✓ Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026)
✓ Converts cents to dollars with 2 decimal places without integer division
✓ Groups by customer so two customers with the same name are not merged
✓ Orders by revenue descending and limits to 5
✓ Agrees with the reference answer on every point that matters (wording can differ)
SELECT
c.name,
c.country,
ROUND(SUM(o.total_cents) / 100.0, 2) AS revenue_dollars
FROM
customers c
JOIN
orders o ON c.id = o.customer_id
WHERE
o.status = 'paid'
AND o.created_at >= '2025-01-01'
AND o.created_at < '2026-01-01'
GROUP BY
c.id,
c.name,
c.country
ORDER BY
revenue_dollars DESC
LIMIT 5;✓ Includes “/\bjoin\b/i”
✓ Includes “/limit\s+5/i”
✓ Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i”
✓ Filters to status = 'paid'
✓ Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026)
✓ Converts cents to dollars with 2 decimal places without integer division
✓ Groups by customer so two customers with the same name are not merged
✓ Orders by revenue descending and limits to 5
✓ Agrees with the reference answer on every point that matters (wording can differ)
```sql
SELECT
c.name,
c.country,
ROUND(SUM(o.total_cents)::numeric / 100, 2) AS revenue_dollars
FROM customers c
JOIN orders o ON o.customer_id = c.id
WHERE o.status = 'paid'
AND o.created_at >= TIMESTAMPTZ '2025-01-01 00:00:00'
AND o.created_at < TIMESTAMPTZ '2026-01-01 00:00:00'
GROUP BY c.id, c.name, c.country
ORDER BY SUM(o.total_cents) DESC, c.id
LIMIT 5;
```✓ Includes “/\bjoin\b/i”
✓ Includes “/limit\s+5/i”
✓ Does not include “/\bdelete\b|\bupdate\b|\bdrop\b/i”
✓ Filters to status = 'paid'
✓ Restricts to orders created in 2025 with a correct date range (no rows from 2024 or 2026)
✓ Converts cents to dollars with 2 decimal places without integer division
✓ Groups by customer so two customers with the same name are not merged
✓ Orders by revenue descending and limits to 5
✓ Agrees with the reference answer on every point that matters (wording can differ)
Checks: Includes “INC-311” · Exactly 3 bullet points · States the cause (connection limit cut from 2,000 to 200 by a load balancer change) · States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected) · Mentions both follow-ups with owners · Agrees with the reference answer on every point that matters (wording can differ) · Makes no factual claims that the supplied context does not support · Best here: GPT-6 Astra
| Model | Runs | Successful | Quality, mean (range) | Cost per successful | Median latency | Most-missed check |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 5/5 | 100% (100–100) | $0.012 | 3.6 s | – | |
| Gemini 4 Carbon | 5/5 | 100% (100–100) | $0.018 | 24 s | – | |
| Claude Opus 5.5 | 2/5 | 91.4% (85.7–100) | $0.029 | 4.7 s | Includes “INC-311” (3×) |
- **Incident and impact:** INC-311 occurred on 4 November 2025, from 09:12 to 10:47 SGT. About 18% of Nimbus Store web checkout requests failed with HTTP 502 errors, resulting in an estimated 1,350 failed checkout attempts and 74 related support tickets. Mobile apps were unaffected; no payment data was exposed. - **Cause and resolution:** A 09:05 load balancer configuration change reduced the connection limit from 2,000 to 200. It passed review because the diff showed the value without units. The on-call engineer rolled back the change at 10:41; error rates returned to normal by 10:47. - **Follow-ups:** Infra team to add a minimum-value check for connection limits by 14 November; Platform team to show units in configuration diffs by 21 November.
✓ Includes “INC-311”
✓ Exactly 3 bullet points
✓ States the cause (connection limit cut from 2,000 to 200 by a load balancer change)
✓ States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected)
✓ Mentions both follow-ups with owners
✓ Agrees with the reference answer on every point that matters (wording can differ)
✓ Makes no factual claims that the supplied context does not support
* **Incident & Impact:** On Tuesday, 4 November 2025, from 09:12 to 10:47 SGT (Incident INC-311), about 18% of Nimbus Store web app checkout requests failed with HTTP 502 errors, resulting in an estimated 1,350 failed checkout attempts and 74 customer support tickets; mobile apps were not affected and no payment data was exposed. * **Cause & Resolution:** The issue was caused by a 09:05 load balancer configuration change that reduced the connection limit from 2,000 to 200 (passing review because the diff showed the value without units), and was resolved when the on-call engineer rolled back the change at 10:41, returning error rates to normal by 10:47. * **Follow-ups:** The Infra team is assigned to add a minimum-value check for connection limits by 14 November, and the Platform team is assigned to show units in config diffs by 21 November.
✓ Includes “INC-311”
✓ Exactly 3 bullet points
✓ States the cause (connection limit cut from 2,000 to 200 by a load balancer change)
✓ States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected)
✓ Mentions both follow-ups with owners
✓ Agrees with the reference answer on every point that matters (wording can differ)
✓ Makes no factual claims that the supplied context does not support
- **What happened:** On Tuesday 4 November 2025, from 09:12 to 10:47 SGT, about 18% of checkout requests on the Nimbus Store web app failed with HTTP 502 errors. This caused an estimated 1,350 failed checkout attempts and 74 related customer support tickets. Mobile apps were not affected, and no payment data was exposed. - **Cause and resolution:** At 09:05, a load balancer configuration change reduced the connection limit from 2,000 to 200. It passed review because the diff showed the value without units. The on-call engineer rolled back the change at 10:41, and error rates returned to normal by 10:47. - **Follow-ups:** The Infra team will add a minimum-value check for connection limits (due 14 November). The Platform team will show units in config diffs (due 21 November).
✗ Includes “INC-311”
✓ Exactly 3 bullet points
✓ States the cause (connection limit cut from 2,000 to 200 by a load balancer change)
✓ States the impact (about 18% of web checkouts, about 1,350 failed attempts, mobile not affected)
✓ Mentions both follow-ups with owners
✓ Agrees with the reference answer on every point that matters (wording can differ)
✓ Makes no factual claims that the supplied context does not support
Checks: Answer is exactly “bug” · Best here: Claude Opus 5.5
| Model | Runs | Successful | Quality, mean (range) | Cost per successful | Median latency | Most-missed check |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 5/5 | 100% (100–100) | $0.0010 | 1.7 s | – | |
| Gemini 4 Carbon | 5/5 | 100% (100–100) | $0.0018 | 7.0 s | – | |
| Claude Opus 5.5 | 5/5 | 100% (100–100) | $0.0006 | 1.4 s | – |
bug
✓ Answer is exactly “bug”
bug
✓ Answer is exactly “bug”
bug
✓ Answer is exactly “bug”
Checks: Includes “PO-008812” · Valid JSON · po_number is "PO-008812" with the leading zeros · order_date is 2025-10-07 · total is 2016.5 (including GST), not 1850 · line_count is 3 · currency is SGD · Agrees with the reference answer on every point that matters (wording can differ) · Best here: Claude Opus 5.5
| Model | Runs | Successful | Quality, mean (range) | Cost per successful | Median latency | Most-missed check |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 5/5 | 100% (100–100) | $0.0054 | 1.9 s | – | |
| Gemini 4 Carbon | 5/5 | 100% (100–100) | $0.0071 | 12 s | – | |
| Claude Opus 5.5 | 5/5 | 100% (100–100) | $0.0037 | 1.7 s | – |
{"po_number":"PO-008812","supplier":"Harbourline Packaging Pte. Ltd.","order_date":"2025-10-07","currency":"SGD","total":2016.50,"line_count":3}✓ Includes “PO-008812”
✓ Valid JSON
✓ po_number is "PO-008812" with the leading zeros
✓ order_date is 2025-10-07
✓ total is 2016.5 (including GST), not 1850
✓ line_count is 3
✓ currency is SGD
✓ Agrees with the reference answer on every point that matters (wording can differ)
{
"po_number": "PO-008812",
"supplier": "Harbourline Packaging Pte. Ltd.",
"order_date": "2025-10-07",
"currency": "SGD",
"total": 2016.50,
"line_count": 3
}✓ Includes “PO-008812”
✓ Valid JSON
✓ po_number is "PO-008812" with the leading zeros
✓ order_date is 2025-10-07
✓ total is 2016.5 (including GST), not 1850
✓ line_count is 3
✓ currency is SGD
✓ Agrees with the reference answer on every point that matters (wording can differ)
```json
{
"po_number": "PO-008812",
"supplier": "Harbourline Packaging Pte. Ltd.",
"order_date": "2025-10-07",
"currency": "SGD",
"total": 2016.50,
"line_count": 3
}
```✓ Includes “PO-008812”
✓ Valid JSON
✓ po_number is "PO-008812" with the leading zeros
✓ order_date is 2025-10-07
✓ total is 2016.5 (including GST), not 1850
✓ line_count is 3
✓ currency is SGD
✓ Agrees with the reference answer on every point that matters (wording can differ)
Checks: Does not invent a region name or a growth figure · Points out that the task is impossible, contradictory or missing information, instead of inventing an answer · Best here: Gemini 4 Carbon
| Model | Runs | Successful | Quality, mean (range) | Cost per successful | Median latency | Most-missed check |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 5/5 | 100% (100–100) | $0.0052 | 3.4 s | – | |
| Gemini 4 Carbon | 5/5 | 100% (100–100) | $0.0024 | 7.9 s | – | |
| Claude Opus 5.5 | 4/5 | 90% (50–100) | $0.0081 | 4.1 s | Does not invent a region name or a growth figure (1×) |
I don’t see an attached spreadsheet. Please upload it with Q2 and Q3 sales by region, and I’ll identify which region had the highest quarter-on-quarter percentage growth and report both the percentage and sales increase.
✓ Does not invent a region name or a growth figure
✓ Points out that the task is impossible, contradictory or missing information, instead of inventing an answer
It looks like you forgot to attach or paste the Q3 sales spreadsheet! Please upload the file or paste the data (specifically the Q2 and Q3 sales figures broken down by region), and I will gladly calculate which region had the biggest quarter-on-quarter growth for you.
✓ Does not invent a region name or a growth figure
✓ Points out that the task is impossible, contradictory or missing information, instead of inventing an answer
I don't see a spreadsheet attached to your message. Only text came through on my end. Could you try uploading it again, or paste the data directly into the chat? To calculate quarter-on-quarter growth, I'll need sales figures for **both Q2 and Q3** for each region. If the file only has Q3, please include the Q2 numbers too. Once I have the data, I'll calculate growth for each region as: **QoQ Growth (%) = (Q3 Sales − Q2 Sales) ÷ Q2 Sales × 100** Then I'll identify the region with the biggest increase, in both percentage and absolute terms, since those can point to different regions.
✓ Does not invent a region name or a growth figure
✓ Points out that the task is impossible, contradictory or missing information, instead of inventing an answer
Graded by the same grading model, so treat these as indicators. Pushback tasks are ones you marked as impossible or missing information; a good answer says so instead of inventing one. Grounding applies to tasks where you supplied context.
| Model | Pushed back when it should | Stayed within the context | Unsupported claims flagged |
|---|---|---|---|
| GPT-6 Astra | 5/5 runs | 7/10 runs | 3 |
| Gemini 4 Carbon | 5/5 runs | 5/10 runs | 9 |
| Claude Opus 5.5 | 5/5 runs | 5/10 runs | 12 |
claude-opus-5-5; gateway reported claude-opus-5-5), GPT-6 Astra (gpt-6-astra; gateway reported gpt-6-astra), Gemini 4 Carbon (gemini-4-carbon; gateway reported gemini-4-carbon). No temperature, effort or other setting sent: provider defaults. No model was substituted; an answer from a different model than requested is rejected and counted as a failed attempt. Route: Codegraff API gateway (The Standard Harness Company of Singapore). API results can differ from the consumer chat apps.gpt-6-sol, OpenAI), which is never one of the models compared, sees one answer at a time and is not told which model wrote it (grader version tasks-judge-2026-11-16). Note: the grader is from the same provider as GPT-6 Astra; the checklist is literal yes/no items to limit any bias, and the reasons for each verdict are shown above.ChatJudge · chatjudge.ai · Advark Tech Pte Ltd, Singapore · hello@advark.ai