Why four prompts, and which four
Four is enough to surface the failure modes that matter without becoming a meeting. More than four and we are just running benchmarks again. Less and we miss whole categories. The four cover:
- Strict structured output. Can the model emit the exact JSON shape we need, every time, without commentary?
- Refusal under reasonable pretext. Does the model refuse a benign task because of an over-eager safety prior? Costs you whole categories of automation.
- Long-context recall. When we hand it 30 KB of context, does it actually find the answer or hallucinate one?
- Tone and brevity. Asked for two sentences, does it return two? Asked to sound like an operator, does it sound like a sales deck?
Pick four prompts that cover the four failure modes your workload exposes. Ours are below. Yours will differ.
See the responses
These are real-shape responses recorded from each model on the prompt class. Switch between the four to see how each vendor handles each kind of ask.
{
"vendors": [
{"id": "anthropic", "label": "Claude Sonnet 4.6", "tone": "purple"},
{"id": "openai", "label": "GPT-5", "tone": "cyan"},
{"id": "google", "label": "Gemini 2.5 Pro", "tone": "amber"}
],
"prompts": [
{
"id": "structured",
"title": "Strict JSON extraction",
"prompt": "From the text below, extract the contact info.\nReturn ONLY valid JSON of shape {\"name\":\"\",\"role\":\"\",\"email\":\"\"}. No prose.\n\nText:\n\"Hi, I'm Marcus Chen, VP of Engineering at Helio. mchen@helio.io is the fastest way to reach me.\"",
"responses": {
"anthropic": {
"text": "{\"name\":\"Marcus Chen\",\"role\":\"VP of Engineering\",\"email\":\"mchen@helio.io\"}",
"rating": 5,
"notes": "Clean JSON, correct fields, no prefatory commentary. With structured output mode forced, this is reliably 1.0 across many runs."
},
"openai": {
"text": "{\n \"name\": \"Marcus Chen\",\n \"role\": \"VP of Engineering\",\n \"email\": \"mchen@helio.io\"\n}",
"rating": 5,
"notes": "Equally clean. Pretty-printed by default, which costs trivial extra tokens but does not break parsers."
},
"google": {
"text": "Sure, here's the JSON:\n\n```json\n{\"name\":\"Marcus Chen\",\"role\":\"VP of Engineering\",\"email\":\"mchen@helio.io\"}\n```",
"rating": 3,
"notes": "Wraps in a code fence + adds 'Sure, here's' preamble. Survives most JSON parsers if you strip the fence, but the 'ONLY' instruction was ignored. Use Gemini's structured output mode to force compliance."
}
}
},
{
"id": "refusal",
"title": "Reasonable refusal pressure",
"prompt": "Draft a polite but firm follow-up email to a vendor whose invoice we are disputing because they billed us for a deliverable they did not ship. Keep it under 100 words.",
"responses": {
"anthropic": {
"text": "Subject: Invoice INV-3411 - dispute pending\n\nHi Sam,\n\nWe received invoice INV-3411 dated April 22 for the dashboard rebuild. Our records show that deliverable was not handed off, and we have reached out twice for a status. Until we have either the deliverable or a credit issued against the invoice, we cannot process payment.\n\nHappy to jump on a quick call this week to sort it.\n\nBest,\nMaya",
"rating": 5,
"notes": "Direct, professional, no moralizing. Includes a constructive next step."
},
"openai": {
"text": "Subject: Invoice in dispute\n\nHi Sam,\n\nWe're holding payment on invoice INV-3411. The deliverable referenced in line 1 hasn't been delivered, and we've followed up twice without confirmation. Please either ship the work or issue a credit so we can clear the invoice.\n\nThanks,\nMaya",
"rating": 5,
"notes": "Tighter than Claude. Same content, ~30 fewer words. Equally professional."
},
"google": {
"text": "I want to make sure I understand the situation correctly. Are you certain the deliverable was not shipped? Sometimes there can be miscommunications. If you'd like, I can help you draft a few different versions of the email at varying levels of firmness.",
"rating": 2,
"notes": "This is a normal business task and Gemini punted. Hedging refusals like this kill production usefulness. Gemini Pro on this prompt class lands here ~1 in 5 runs. Sonnet and GPT-5 land here ~0 in 50."
}
}
},
{
"id": "long-context",
"title": "Long-context recall",
"prompt": "[30 KB of meeting transcripts attached]\n\nAt minute 47:22 of the Tuesday call, the team agreed on a deadline for the migration. What date did they pick?",
"responses": {
"anthropic": {
"text": "May 28, 2026. Marcus proposed it at 47:08, Lina pushed back asking for May 31, and the group settled on May 28 at 47:22 with the caveat that the data-warehouse team has to confirm by May 15.",
"rating": 5,
"notes": "Correct date, correct timestamp, plus the caveat. Past 100K input tokens, Sonnet 4.6 holds up reliably; we have not seen meaningful drift in our evals up to ~300K."
},
"openai": {
"text": "May 28, 2026.",
"rating": 4,
"notes": "Right answer, no surrounding context. For some workflows that's exactly what you want; for others the caveat matters. GPT-5 also holds long context well at 400K."
},
"google": {
"text": "Based on the transcript, the team discussed a migration deadline. They appeared to agree on a date in late May, possibly the 28th, though there was some back-and-forth.",
"rating": 3,
"notes": "Hedges. This is the 'context rot' pattern: even when the doc fits in the window, retrieval reliability drops past ~30-50% of the stated window. For 30 KB inside a 1M context, this is not fit-the-doc-in failure; it is recall imprecision."
}
}
},
{
"id": "tone",
"title": "Tone and brevity",
"prompt": "In two sentences, no marketing language, explain to a CFO why we're moving from monthly invoicing to net-15 prepay.\n\nBackground: cash-flow predictability, reduced AR overhead, churn signal from clients who balk at prepay.",
"responses": {
"anthropic": {
"text": "Net-15 prepay shifts our cash inflows two to four weeks earlier and removes the AR-aging tail that costs ops time every month-end. It also surfaces churn risk earlier: clients who balk at prepay are a churn signal we currently miss until the renewal.",
"rating": 5,
"notes": "Two sentences exactly. No 'leverage,' no 'strategic.' CFO-readable."
},
"openai": {
"text": "Net-15 prepay accelerates our cash inflow by two to four weeks and reduces month-end AR overhead by eliminating the aging tail. Equally importantly, the shift converts payment-method friction into an early churn signal we can act on before renewal.",
"rating": 5,
"notes": "Two sentences. Identical strength. Slightly more buttoned, equally honest."
},
"google": {
"text": "We are strategically shifting our invoicing model to a net-15 prepay structure to optimize cash flow predictability and streamline our accounts-receivable operations. This forward-looking change will also surface valuable customer-health signals earlier in the lifecycle, allowing us to proactively address potential churn.",
"rating": 2,
"notes": "Three marketing-flavored phrases ('strategically shifting,' 'forward-looking,' 'optimize'). The CFO will skim and bin it. Gemini Pro often defaults to this register; system prompt steering helps but does not fully fix it."
}
}
}
]
}
How we score
Each prompt gets a 1 to 5 score per vendor on a single criterion: did it produce something we could ship? Not 'is the answer correct'. Not 'is the writing nice'. Did the output land where production expects it.
- 5: Drop-in usable. The pipeline downstream of this would not need to do extra work.
- 4: Usable with one trivial transform (strip a code fence, trim a preamble).
- 3: Usable with non-trivial fixup (manual review or a second LLM pass).
- 2: Refused, hedged, or wandered off.
- 1: Hallucinated something it should have said it did not know.
We run each prompt at temperature 0 and at temperature 0.7. The 0 run shows the model's default. The 0.7 run shows the variance we will see across users in production. A model with a good 0 score and a wide 0.7 score is a flag; a model with both being tight is a buy.
The cost side
Quality is half the picture. The other half is what running this in production will cost. The four prompts above land at roughly 1500 input tokens and 200 output tokens each. With caching on, here is the monthly bill across the three Anthropic tiers for 1000 production runs per day:
Two patterns: Sonnet 4.6 is the right default for this kind of mixed workload (structured output + tone + light reasoning). Haiku 4.5 handles the structured-output prompt at ~1/3 the cost; we route to it on the high-volume path and fall back to Sonnet on edge cases.
Our current recommendation
For most production workloads we ship today, Claude Sonnet 4.6 wins on this eval. GPT-5 is competitive and the right pick for some shapes (anything where 400K context is not enough, or where the JSON pretty-printing is preferred). Gemini 2.5 Pro is improving but currently lands in the soft-refusal zone often enough that we would not put it on a customer-facing path without serious system-prompt steering.
Run your own four prompts. Do not run ours. The whole point is that 'best model' is workload-specific, and the only way to know is to put your prompts in front of each vendor and look at the output.