Structured Outputs: How to Make LLMs Return Data Your Systems Can Trust

The reliable way to get machine-readable data out of a language model is constrained decoding against a schema, not prompting for JSON and parsing what comes back. Native structured output modes, offered by the major providers and by open-source libraries that constrain token sampling directly, make syntactic validity a guarantee rather than a probability: the model is prevented at the sampling layer from emitting a token that would break the schema, so the parse cannot fail. Teams that move from prompt-and-parse to a schema guarantee typically delete an entire class of retry logic, error handling, and alerting along with it. The important consequence is what this does not solve. A schema guarantee makes the output parseable, not correct. The model can return a perfectly valid object in which the amount is off by a factor of ten, the enum is the wrong member, the required field was filled with a plausible guess rather than left absent, and every downstream system will accept it without complaint because it validates. The engineering problem worth attention starts precisely there, and this guide covers the choice of approach, the schema design that determines most of the outcome, the validation stack that catches what the schema cannot, and the cases where forcing structure degrades the answer itself.

Key Takeaways

• Constrained decoding provides a syntactic guarantee that prompt-and-parse cannot, at any prompt quality. If the provider or serving stack offers a schema-enforced mode, using anything weaker for production data extraction is choosing a failure class voluntarily.
• Schema design is the actual craft, and it decides more about output quality than prompt wording does. Flat beats nested, enums beat free text, and field descriptions inside the schema function as prompts the model reliably reads.
• Optionality is where silent failures concentrate. A required field the model cannot support from the input becomes a fabrication, because the schema left it no way to say "not present", which makes nullable-plus-reason the safer default for anything not guaranteed by the input.
• Syntactic validity is guaranteed; semantic validity is not. Range checks, referential checks against systems of record, cross-field consistency, and a judge pass on high-stakes fields are separate layers, and none of them are optional at scale.
• Forcing schema adherence on reasoning-heavy tasks measurably degrades the reasoning. The two-pass pattern, reason free-form and then extract into the schema with a second cheap call, recovers the quality at modest cost.
• Structured outputs are best understood as the boundary between the probabilistic and deterministic halves of a system. Everything upstream is a distribution; everything downstream assumes a contract. The schema is where the contract is declared and where it must be enforced.

The Four Approaches

Prompt and parse

Ask for JSON in the prompt, parse the string, handle the exceptions. This is where nearly every team starts and where a surprising number remain.

The failure modes are well known and cumulatively expensive: prose wrapped around the object, markdown code fences, trailing commas, single quotes, unescaped newlines inside strings, truncation at the token limit mid-object, and the model helpfully explaining what it did before doing it. Each is individually fixable with a regex or a lenient parser, which is exactly why teams stay: the fix always looks like ten minutes of work, and the accumulated repair layer becomes a permanent maintenance burden that nobody owns.

The honest assessment is that prompt-and-parse is acceptable for prototypes, for one-off analysis, and for models or endpoints that offer nothing better. It is not an appropriate production choice when a schema-enforced mode is available, because no amount of prompt engineering converts a probability into a guarantee.

Function and tool calling

The model is given a tool definition with a typed parameter schema and returns a structured call rather than free text. This was the first widely available mechanism for reliable structure and it remains the correct choice whenever the semantic intent is genuinely "the model should invoke something", including in agent systems where tool schemas travel over a protocol such as the Model Context Protocol.

Adherence is materially better than prompting, because the provider is applying its own constraint machinery, but the guarantee varies by provider and by model tier, and the mechanism carries a semantic mismatch when used purely for extraction: nothing is being called, and the model is being asked to pretend an extraction is an action. In practice this matters less than the ergonomics, which are good, and tool calling remains the most widely deployed structured output mechanism in production.

JSON mode

A flag that constrains the model to emit syntactically valid JSON. It eliminates the fence-and-prose class of failure entirely and guarantees the parse succeeds.

The critical limitation is routinely misread: basic JSON mode guarantees valid JSON, not JSON matching a specific schema. The model can return a valid object with fields renamed, omitted, added, or nested differently than intended. Code that flags JSON mode and then accesses fields positionally is relying on an unenforced convention, which works until a model version changes and then fails in production on a Tuesday. JSON mode is a strict improvement over prompting and a strict downgrade from schema enforcement.

Grammar-constrained decoding with a schema guarantee

The strongest mechanism. A schema is compiled into a grammar or state machine, and at each decoding step the sampler is masked so that only tokens which can continue a schema-valid document remain available. Invalid output is not corrected after the fact; it is unreachable.

This is what the providers' schema-enforced structured output modes implement, and what open-source libraries such as Outlines implement against locally served models. Instructor wraps the provider mechanisms with Pydantic models, which is the ergonomic pattern most Python teams converge on: define the type once, get validation and the schema from the same declaration. The engineering cost is modest and the operational payoff is the deletion of an entire error path.

Two honest caveats. Compiling a very large or deeply recursive schema carries measurable overhead, and some serving stacks handle this better than others. And a constrained model that has painted itself into a corner will emit something schema-valid, because the sampler leaves it no alternative, which is the mechanism behind the most important failure mode in this entire guide.

Schema Design Is the Real Craft

Once the parse is guaranteed, output quality is determined mostly by the schema, and schema design gets a fraction of the attention that prompt wording receives.

Flat beats nested. Deeply nested objects degrade adherence and make partial failure hard to isolate. A model that must hold four levels of structure while also reasoning about content spends capacity on the structure. Flattening with compound field names, or splitting one deep extraction into two shallow calls, is almost always the better trade, and it makes validation errors point at a specific field instead of a subtree.

Enums beat free text, everywhere they can be applied. A free-text status field will eventually contain "pending", "Pending", "pending review", and "awaiting approval" for the same state. An enum makes the wrong values unreachable at the sampling layer. Every field with a finite value set should be an enum, and the exercise of enumerating the set frequently surfaces that the downstream system's own state model was never written down.

Field descriptions are prompts. Descriptions in the schema are passed to the model and are read. This is the most underused lever available: a description that says "the invoice total including tax, as it appears on the document; do not compute it from line items" resolves an ambiguity that would otherwise require prompt text, and it lives next to the field it governs rather than in a prompt paragraph three screens away. Schema descriptions are also more maintainable than prompt instructions, because they cannot drift away from the field they describe.

Optionality is where the silent failures hide. This is the single highest-value paragraph in this guide. When a field is marked required and the source material does not support it, a constrained model cannot decline. The sampler has removed every token that would produce an absent field, so the model produces its best guess, formatted perfectly, indistinguishable from an extraction. The failure is invisible precisely because the schema worked.

The mitigations are straightforward once the mechanism is understood. Mark a field required only when the input format genuinely guarantees it. Prefer nullable fields with an explicit companion reason, or a sentinel enum member such as not_present, so that absence is a value the model can legitimately express. And treat a sudden drop in null rates on an optional field as a production alert, because it usually means an upstream input change is being papered over with fabrication.

Order fields so that context precedes conclusions. Generation is sequential, and a field emitted early cannot depend on one emitted later. Placing a short evidence or reasoning string before the classification it supports gives the model tokens to think in and measurably improves the classification. Reversing that order asks for the conclusion first and the justification afterward, which produces post-hoc rationalization of an answer already committed to.

The Validation Stack

Schema conformance is layer one of four, and it is the only one the provider gives away.

Layer one, syntactic and structural. Guaranteed by constrained decoding. Types are right, required fields are present, enums are members. Nothing to build.

Layer two, semantic and range validation. Ordinary deterministic code. Is the invoice date within a plausible window rather than in 1970 or 2087? Is the percentage between zero and one hundred? Do the line items sum to the stated total? Is the currency one the business actually transacts in? These checks are cheap, catch a large share of real errors, and are frequently skipped because the object validated. Cross-field consistency checks belong here and are the highest-yield subset: individual fields are usually plausible in isolation, and their relationships are where extraction errors show up.

Layer three, referential validation against systems of record. Does the extracted customer identifier exist? Does the supplier name resolve to a known vendor? Does the referenced order number match an open order? This layer converts a plausible hallucination into a hard failure, and it is the strongest available defense against the confidently-wrong-but-valid output, because the model cannot fabricate its way into a foreign key that exists.

Layer four, judge passes on high-stakes fields. A second model call evaluating whether the extraction is supported by the source, applied selectively to the fields where an error is expensive. This is the most costly layer and should be targeted rather than blanket: run it on the amount and the counterparty, not on the document type. Judge passes are also the layer most prone to false confidence, since a judge sharing the extractor's blind spots will confirm its errors, which argues for a different model or at minimum a different prompt framing on the judge.

Retry and Repair

With constrained decoding, retries are no longer about parse failures. They are about validation failures at layers two through four, and the strategy matters for both cost and correctness.

The cheapest effective pattern is a targeted repair call: return the specific validation error to the model along with the original input and the previous output, and ask for a corrected object. This succeeds often, because the model usually has the information and made a localized mistake. It costs one additional call on the failing fraction only.

Two disciplines keep this from becoming a cost problem. Cap retries at one or two, because a third attempt on the same input rarely succeeds and usually indicates the input is genuinely ambiguous or the schema is wrong. And escalate rather than loop: route persistent failures to a larger model once, then to a human queue, which is the same escalation ladder described in the guide to model routing in production and follows the same economics. The metric to watch is cost per successfully validated record, not cost per call, because a cheap extraction that fails validation half the time is not cheap.

Repair rate is also the best available leading indicator of upstream change. A stable pipeline that suddenly needs repairs on fifteen percent of documents is telling you the documents changed, and that signal is more actionable than any model-quality metric.

When Structure Hurts

Constrained decoding is not free of consequences for the content of the answer, and the honest version of this guide has to say so.

Forcing a model to emit into a rigid schema on a task requiring genuine reasoning degrades the reasoning, and the mechanism is not mysterious. Chain-of-thought works because the model has tokens in which to compute. A schema that jumps straight to conclusion removes that space, and the model must commit to an answer with no room to derive it. The effect is largest on multi-step arithmetic, on classification requiring the weighing of competing evidence, and on any task where the correct answer is not directly stated in the input.

The fix is the two-pass pattern, and it is worth adopting as a default for anything analytical. First call: unconstrained, reason freely, produce an answer in prose. Second call: cheap, constrained, extract that prose into the schema. The extraction pass is a mechanical task well within a small model's capability, so the second call adds little cost, and this pairs naturally with a routing setup where the reasoning goes to a capable model and the extraction goes to a cheap one.

A related and less obvious cost is that a constrained model producing output in a domain far from its training distribution has been given no escape hatch. It cannot say "this document is not an invoice"; it can only produce an invoice-shaped object. Every extraction schema should therefore include a way to express failure, whether a top-level extraction_failed boolean with a reason or a confidence field the pipeline thresholds on. Systems without one convert every out-of-distribution input into a confident fabrication, and this remains true regardless of which model generation is in use, including the endpoints available during the week of August 31.

Structured Outputs as a System Boundary

The most useful mental model is architectural. A production system that uses a language model has a probabilistic half and a deterministic half, and structured output is the boundary between them. Upstream, everything is a distribution over possible responses. Downstream, code assumes a contract: this field exists, this type holds, this enum has these members. The schema is where that contract is declared, and the validation stack is where it is enforced.

Reading it this way settles several design questions that otherwise get argued from taste. The schema belongs in version control next to the code that consumes it, versioned like any other interface, because it is one. Schema changes are interface changes and deserve the same compatibility discipline that a data contract or an API version gets. Validation failures are boundary violations and should be logged with the full input, the output, and the failing check, because that record is the only way to distinguish a model regression from an upstream data change. And the boundary should be as narrow as the use case allows: every field in the schema is a field that can be wrong, and extracting eleven fields when the downstream system consumes four is three additional silent failure modes bought for nothing.

The same boundary logic explains why structured outputs pair so naturally with retrieval systems. A retrieval-augmented generation pipeline is already an exercise in assembling trustworthy context; structured output is the matching discipline on the way out. And once these calls run inside agent loops rather than single request-response cycles, the validation failures become trajectory events that need tracing, which is the observability problem covered in the guide to debugging agents in production.

A Deployment Sequence

The ordering matters. Teams that begin with judge passes are buying an expensive check on a schema that is still generating avoidable errors, and teams that skip stage two find that stage five's judge spends its time catching problems a better enum would have prevented for free.

Frequently Asked Questions

What is the difference between JSON mode and structured outputs?

JSON mode guarantees the model emits syntactically valid JSON. Structured outputs, meaning schema-enforced constrained decoding, additionally guarantee that the JSON conforms to a specific schema: the declared fields are present, the types are right, and enum values are members of the enumeration. Code written against JSON mode that assumes particular field names is relying on an unenforced convention, which is a common source of production failures after a model version change.

Does constrained decoding make the model less accurate?

For extraction and formatting tasks, no meaningfully. For reasoning-heavy tasks, yes, and measurably, because the schema removes the intermediate tokens the model would otherwise use to compute an answer. The standard remedy is the two-pass pattern: one unconstrained call to reason in prose, then a cheap constrained call to extract that prose into the schema. The second call is mechanical and can be routed to a small model.

Why does a schema-valid output still contain wrong data?

Because constrained decoding enforces structure, not truth. When a required field is not supported by the input, the sampler has removed every token that would let the model omit it, so the model produces its most plausible guess in correct format. This is the dominant silent failure mode in structured extraction, and the mitigations are marking fields optional unless the input format guarantees them, providing an explicit way to express absence or failure, and validating against systems of record.

Which library should a team use?

For Python teams calling hosted models, Instructor with Pydantic is the common choice, since the Pydantic model produces both the schema and the runtime validation from one declaration. For self-hosted or open-weight models where the serving stack is under the team's control, Outlines constrains sampling directly against a schema or grammar. Provider-native structured output modes are the right default whenever available, with a library on top for ergonomics rather than as a replacement.

How should validation failures be handled in production?

Attempt one targeted repair call that returns the specific validation error alongside the original input, then escalate rather than loop: a larger model once, then a human queue. Cap retries at one or two, because a third attempt on the same input usually indicates genuinely ambiguous input or an incorrect schema rather than a transient model error. Track cost per successfully validated record, and treat a rising repair rate as a signal that the upstream inputs changed.