Claude API vs GPT-4o-mini JSON Hallucination Rate Experiment

Structured JSON output broke in production last quarter. Not because the code was wrong — because the model lied confidently inside a valid schema. That’s the problem engineers are finally measuring in 2026, and the numbers between Claude API and GPT-4o-mini are more divergent than most teams expect.

Key Takeaways

• Claude’s structured output mode shows measurably lower hallucination rates on constrained JSON schemas compared to GPT-4o-mini, particularly on nested object generation with enum constraints.
• GPT-4o-mini delivers faster token throughput (~85 tokens/sec vs. Claude Haiku 3.5’s ~60 tokens/sec) but trades off semantic accuracy under strict schema validation in production benchmarks.
• Hallucination rates vary significantly by task type — extraction tasks favor Claude; classification tasks with small enum sets favor GPT-4o-mini on cost-per-call.
• Teams running high-volume pipelines (>500K calls/month) face a real cost-vs-accuracy fork: GPT-4o-mini at ~$0.15/1M input tokens vs. Claude Haiku 3.5 at ~$0.80/1M input tokens.
• Schema complexity is the single biggest predictor of which model degrades faster — not raw benchmark scores.

Why JSON Mode Hallucinations Are a 2026 Problem

Two years ago, structured output was a hack. You’d prompt-engineer your way to a JSON blob, validate it client-side, and retry on failure. Good enough.

That changed fast. As LLM pipelines matured — feeding outputs directly into databases, triggering downstream API calls, populating user-facing dashboards — “retry on failure” became a liability. The industry needed guaranteed schema conformance, not just probabilistic JSON.

Both Anthropic and OpenAI responded. OpenAI shipped response_format: json_schema for GPT-4o-mini in late 2024, adding constrained decoding via a grammar-based sampler. Anthropic followed with tool_use-based structured output refinements and tighter JSON mode enforcement across the Claude 3.5 and subsequent Haiku/Sonnet 3.7 line through early 2026.

The result: both APIs now claim near-100% schema conformance. What they don’t advertise is the semantic hallucination rate — cases where the model returns a structurally valid JSON object containing fabricated, wrong, or nonsensical field values. A {"invoice_total": 0.00} that parses clean but is completely invented. That’s the real benchmark. And that’s exactly what recent independent experiments are measuring.

Schema Conformance vs. Semantic Accuracy: Two Different Problems

This distinction matters more than most teams realize.

Schema conformance is binary — the output either matches the schema or it doesn’t. Modern constrained decoding makes this close to a solved problem for both providers.

Semantic accuracy is the harder question. Given a source document and a target schema, does the model extract the correct values, or does it hallucinate plausible-sounding ones?

According to a structured output comparison published on Medium by Gluk (2025), Anthropic’s Claude models outperformed OpenAI’s on nested schema extraction tasks involving multi-field documents — particularly when fields required cross-referencing information from different sections of a source text. The hallucination rate gap was most visible on schemas with 6+ fields and conditional required properties. GPT-4o-mini showed a consistent pattern of “confident infill” — populating missing fields with statistically likely values rather than returning null or triggering a validation error.

This matters enormously for use cases like invoice processing, medical record parsing, or contract extraction. A model that returns null on ambiguous fields is far safer than one that fills them with plausible fiction.

Where GPT-4o-mini Wins: Speed, Cost, Simple Schemas

GPT-4o-mini isn’t the wrong choice. It’s the situationally wrong choice when applied to the wrong task.

On classification tasks — sentiment tagging, category assignment, entity type labeling with small closed enum sets — GPT-4o-mini performs comparably to Claude Haiku 3.5 on semantic accuracy while costing approximately 5x less per million input tokens (OpenAI pricing as of Q1 2026: ~$0.15/1M input tokens; Anthropic Claude Haiku 3.5: ~$0.80/1M, per official pricing pages).

Throughput also favors GPT-4o-mini in high-concurrency scenarios. At constrained output lengths under 256 tokens, its latency profile is lower. For real-time user-facing features where 400ms matters, that’s not trivial.

The problem surfaces when teams start with a simple schema and then add complexity over time. By the time the schema has grown to 12 fields with nested arrays, the hallucination rate has quietly climbed — and nobody noticed because the JSON kept parsing.

That’s the failure mode worth worrying about. Not the dramatic crash. The silent drift.

The Hallucination Rate Breakdown by Task Type

Published comparisons converge on a consistent pattern:

Rates derived from published structured output benchmarks (Gluk, Medium, 2025) and community reproducible experiments shared on the Anthropic developer forum, Q1 2026. These are approximate ranges, not absolute guarantees.

The gap isn’t uniform. It’s specifically large on complex extraction — which is exactly the task where a hallucinated value causes the most downstream damage.

What “JSON Mode” Actually Means Under the Hood

Both providers use constrained decoding, but the implementation differs. OpenAI’s approach grammar-samples tokens against the schema definition at inference time, which strongly enforces structural validity. Anthropic’s Claude combines fine-tuned instruction following with constrained sampling, and also surfaces structured output behavior through the tool_use API pattern.

In practice, Claude tends to perform better when schema field names carry semantic meaning — "customer_billing_address" gives it more signal than "field_22b". GPT-4o-mini’s constrained sampler is more schema-agnostic, which helps in some contexts and hurts in others.

Neither approach is unconditionally superior. The right model depends on whether your task demands semantic grounding or just structural enforcement — and those are separate capabilities.

How to Actually Choose

The $0.65/1M token question isn’t really about cost. It’s about which failure mode you can tolerate.

For teams running document intelligence pipelines — legal, finance, healthcare — the semantic accuracy gap at complex schemas makes Claude the defensible choice. A hallucinated invoice field that passes schema validation and flows into a payment system costs far more to fix than the token price difference ever justified.

For teams running classification at scale — content moderation tagging, support ticket routing, simple entity extraction — GPT-4o-mini is the rational call. Hallucination rates are comparable, the cost savings are real, and schema complexity rarely grows into the range where the gap becomes dangerous.

Three signals worth watching over the next six months:

1. OpenAI’s structured output improvements — OpenAI has signaled investment in semantic grounding for JSON mode, not just structural enforcement. If they close the semantic gap, the cost argument becomes decisive.
2. Anthropic’s batch API pricing — Claude’s cost disadvantage shrinks significantly on async batch jobs. Pricing updates could shift the accuracy-per-dollar math meaningfully.
3. Third-party benchmark standardization — There’s no agreed-upon eval framework for structured output semantic accuracy yet. Whoever publishes a credible standard first will shape how the industry measures this going forward.

What Comes Next

The structured output hallucination story isn’t settled. Both providers made real progress on schema conformance — that problem is largely solved. The semantic accuracy gap on complex extraction tasks remains meaningful, and it runs in Claude’s favor based on current published benchmarks.

Four things worth carrying forward:

• Schema complexity is the gating variable. Run your own eval before committing to either model at scale.
• Cost math is real but task-dependent. GPT-4o-mini wins on simple classification at volume; Claude wins where extraction accuracy is mission-critical.
• Both providers are actively improving. Q2 2026 model updates from both Anthropic and OpenAI are expected to push semantic accuracy further.
• “JSON mode” isn’t one thing. Constrained decoding and semantic grounding are separate capabilities. Check which one your task actually requires.

Run a 500-call benchmark on your specific schema before deciding. Don’t outsource that decision to a leaderboard.

Your current failure mode — structural or semantic — is the single question that determines which direction to go.

References

1. ChatGPT-4o vs Claude 4: Comprehensive Report and Comparison
2. Structured Output Comparison across popular LLM providers — OpenAI, Gemini, Anthropic, Mistral and A
3. OpenAI’s GPT-5.5 vs Claude Opus 4.7: Which is better? | Mashable

Photo by Bernd 📷 Dittrich on Unsplash