← Private model comparison

Published 2026-11-15. Synthetic documents written by ChatJudge; no customer data. This is the report format buyers receive, generated by the same code from a real run of the three models.

Citation

ChatJudge (2026-11-15). Sample model comparison on five synthetic invoices: Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon all captured every field you marked essential on these five documents, so they tie on your requirements. Run 2026-11-15 22:42 SGT. https://chatjudge.ai/private-comparison/sample-2026-11-15

Download the report (PDF) Download (HTML) The 5 synthetic documents Answer key (CSV)

ChatJudge sample report on synthetic documents · run 2026-11-15 22:42 SGT

Sample report: which model extracts these invoice fields, and what does it cost?

Which model extracts the fields you need, and what does it cost per document? Tested on the same five documents and the same field list for all three models.

Decision

Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon all captured every field you marked essential on these five documents, so they tie on your requirements.

Of those, Gemini 4 Carbon costs least, at $0.0081 per document against $0.0184 for Claude Opus 5.5 and $0.0239 for GPT-6 Astra.

This applies to these five documents only. It is not a general ranking of the models.

Essential fields

Essential fieldClaude Opus 5.5GPT-6 AstraGemini 4 Carbon
Invoice number5/55/55/5
Invoice date5/55/55/5
Supplier name5/55/55/5
Purchase-order number5/55/55/5
Total amount due5/55/55/5
All essential values25/25 · met25/25 · met25/25 · met

Overall accuracy and cost

ModelAll values rightMissedWrongUnexpectedAmbiguousCost per documentCost, 5 documentsTime, 5 calls
Claude Opus 5.5
Anthropic · claude-opus-5-5
45/45 (100%)0000$0.0184$0.091832.1 s
GPT-6 Astra
OpenAI · gpt-6-astra
45/45 (100%)0000$0.0239$0.119323.7 s
Gemini 4 Carbon
Google · gemini-4-carbon
45/45 (100%)0000$0.0081$0.040445.7 s

Cost = tokens reported by the gateway × list price per token (claude-opus-5-5: $4.40 in / $22.00 out per million tokens; gpt-6-astra: $11.00 in / $55.00 out per million tokens; gemini-4-carbon: $2.20 in / $11.00 out per million tokens). Marked unknown when the gateway reported no usage.

Representative errors

No errors: every model got every value right.

Per-field results

FieldTypeClaude Opus 5.5GPT-6 AstraGemini 4 Carbon
Invoice number (essential)Identifier5/55/55/5
Invoice date (essential)Date5/55/55/5
Due dateDate5/55/55/5
Supplier name (essential)Name5/55/55/5
Customer nameName5/55/55/5
Purchase-order number (essential)Identifier5/55/55/5
SubtotalAmount with currency5/55/55/5
Tax amountAmount with currency5/55/55/5
Total amount due (essential)Amount with currency5/55/55/5

Method

ChatJudge · chatjudge.ai · Advark Tech Pte Ltd, Singapore.

Field definitions the models received

You extract specific fields from one business document. Use only the document text you are given.

Fields to extract (every field below must appear in your answer):
1. "invoice_number" — Invoice number (Identifier). The identifier the supplier gives this invoice, usually labelled "Invoice No.", "Invoice #", "Invoice number" or "Tax invoice no.". Not the purchase-order number, customer account number, or tax registration number. Format: Copy the identifier exactly as printed: keep every character, including leading zeros, letters, prefixes, hyphens and slashes. Do not reformat it.
2. "invoice_date" — Invoice date (Date). The date the invoice was issued (labelled "Invoice date", "Date" or "Date of issue"). Not the due date or the delivery date. Format: Give the date as YYYY-MM-DD. If the document only shows an ambiguous numeric date (such as 03/04/2026) and nothing tells you the order, copy it exactly as printed instead.
3. "due_date" — Due date (Date). The date payment is due, if the invoice states one (labelled "Due date" or "Payment due"). If the invoice gives only payment terms such as "Net 30" and no date, there is no due date. Format: Give the date as YYYY-MM-DD. If the document only shows an ambiguous numeric date (such as 03/04/2026) and nothing tells you the order, copy it exactly as printed instead.
4. "supplier_name" — Supplier name (Name). The legal or trading name of the company issuing the invoice (the seller). Format: Give the name as printed, without adding or dropping words. Do not include addresses or titles.
5. "customer_name" — Customer name (Name). The name of the company or person being billed (labelled "Bill to", "Customer" or "Sold to"). Format: Give the name as printed, without adding or dropping words. Do not include addresses or titles.
6. "po_number" — Purchase-order number (Identifier). The buyer's purchase-order number that this invoice refers to (labelled "PO", "PO number", "Your order", "Order ref"). Not the invoice number. Many invoices have none. Format: Copy the identifier exactly as printed: keep every character, including leading zeros, letters, prefixes, hyphens and slashes. Do not reformat it.
7. "subtotal" — Subtotal (Amount with currency). The total before tax, with its currency. Format: Give the ISO 4217 currency code, a space, then the number with no thousands separators and a dot for decimals, for example "USD 1250.00". If the document shows only a symbol such as "$" and no country or code, copy the symbol and number as printed.
8. "tax_amount" — Tax amount (Amount with currency). The total tax charged (GST, VAT or sales tax), with its currency. If the invoice shows no tax line, there is no tax amount. Format: Give the ISO 4217 currency code, a space, then the number with no thousands separators and a dot for decimals, for example "USD 1250.00". If the document shows only a symbol such as "$" and no country or code, copy the symbol and number as printed.
9. "total_amount" — Total amount due (Amount with currency). The final amount payable including tax, with its currency (labelled "Total", "Amount due", "Balance due"). Format: Give the ISO 4217 currency code, a space, then the number with no thousands separators and a dot for decimals, for example "USD 1250.00". If the document shows only a symbol such as "$" and no country or code, copy the symbol and number as printed.

Rules:
- If a field is not stated in the document, set its "value" to null. Do not guess, infer or calculate a value that is not printed.
- For each field also give "quote": the shortest exact passage from the document that shows the value, or null when the value is null.
- Return only JSON in exactly this shape, with every field key present:
{"fields": {"invoice_number": {"value": ..., "quote": ...}, "invoice_date": {"value": ..., "quote": ...}, "due_date": {"value": ..., "quote": ...}, "supplier_name": {"value": ..., "quote": ...}, "customer_name": {"value": ..., "quote": ...}, "po_number": {"value": ..., "quote": ...}, "subtotal": {"value": ..., "quote": ...}, "tax_amount": {"value": ..., "quote": ...}, "total_amount": {"value": ..., "quote": ...}}}

Compare models on my documents · US$299