ChatJudge sample report on synthetic documents · run 2026-11-15 22:42 SGT
Which model extracts the fields you need, and what does it cost per document? Tested on the same five documents and the same field list for all three models.
Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon all captured every field you marked essential on these five documents, so they tie on your requirements.
Of those, Gemini 4 Carbon costs least, at $0.0081 per document against $0.0184 for Claude Opus 5.5 and $0.0239 for GPT-6 Astra.
This applies to these five documents only. It is not a general ranking of the models.
| Essential field | Claude Opus 5.5 | GPT-6 Astra | Gemini 4 Carbon |
|---|---|---|---|
| Invoice number | 5/5 | 5/5 | 5/5 |
| Invoice date | 5/5 | 5/5 | 5/5 |
| Supplier name | 5/5 | 5/5 | 5/5 |
| Purchase-order number | 5/5 | 5/5 | 5/5 |
| Total amount due | 5/5 | 5/5 | 5/5 |
| All essential values | 25/25 · met | 25/25 · met | 25/25 · met |
| Model | All values right | Missed | Wrong | Unexpected | Ambiguous | Cost per document | Cost, 5 documents | Time, 5 calls |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 Anthropic · claude-opus-5-5 | 45/45 (100%) | 0 | 0 | 0 | 0 | $0.0184 | $0.0918 | 32.1 s |
| GPT-6 Astra OpenAI · gpt-6-astra | 45/45 (100%) | 0 | 0 | 0 | 0 | $0.0239 | $0.1193 | 23.7 s |
| Gemini 4 Carbon Google · gemini-4-carbon | 45/45 (100%) | 0 | 0 | 0 | 0 | $0.0081 | $0.0404 | 45.7 s |
Cost = tokens reported by the gateway × list price per token (claude-opus-5-5: $4.40 in / $22.00 out per million tokens; gpt-6-astra: $11.00 in / $55.00 out per million tokens; gemini-4-carbon: $2.20 in / $11.00 out per million tokens). Marked unknown when the gateway reported no usage.
No errors: every model got every value right.
| Field | Type | Claude Opus 5.5 | GPT-6 Astra | Gemini 4 Carbon |
|---|---|---|---|---|
| Invoice number (essential) | Identifier | 5/5 | 5/5 | 5/5 |
| Invoice date (essential) | Date | 5/5 | 5/5 | 5/5 |
| Due date | Date | 5/5 | 5/5 | 5/5 |
| Supplier name (essential) | Name | 5/5 | 5/5 | 5/5 |
| Customer name | Name | 5/5 | 5/5 | 5/5 |
| Purchase-order number (essential) | Identifier | 5/5 | 5/5 | 5/5 |
| Subtotal | Amount with currency | 5/5 | 5/5 | 5/5 |
| Tax amount | Amount with currency | 5/5 | 5/5 | 5/5 |
| Total amount due (essential) | Amount with currency | 5/5 | 5/5 | 5/5 |
cmp-extract-v1), the same document text and the same output format, one document per call.claude-opus-5-5; gateway reported claude-opus-5-5), GPT-6 Astra (gpt-6-astra; gateway reported gpt-6-astra), Gemini 4 Carbon (gemini-4-carbon; gateway reported gemini-4-carbon). No model was substituted.ChatJudge · chatjudge.ai · Advark Tech Pte Ltd, Singapore.