Most tutorials on calling a language model end with a triumphant JSON.parse(response.content). That single line works beautifully for the first ten test cases. Then you ship. Sometime around request four hundred, the model invents a date, returns eight array items when your schema permits only five, or produces a perfectly valid JSON object with a field quietly missing.
I hit this wall while building Temploracraft, a résumé tool that converts an uploaded document into structured data for editing. The input is genuinely unpredictable: two-column PDFs, pseudo-tables, and dates in roughly fourteen different formats. The output must be strict because every extracted field lands in a form where a person will scrutinize it. When the model misreads a date, the user flips out in about two seconds.
This article explores the layer between "the model returned some text" and "my application has data it can trust." We'll examine the three primary mechanisms for constraining output, why designing the schema first beats writing a longer prompt, how to build a retry loop that doesn't burn through your budget, and what to do about failures no retry will ever fix.
Table of Contents
Prerequisites
This guide assumes familiarity with Python and basic API calls to a large language model—probably OpenAI, but the principles apply to any provider. We'll work with JSON Schema and Pydantic for validation, and we'll reference the OpenAI Python SDK. By mid-2026, the ecosystem has matured significantly: most major providers support native structured outputs, and OpenRouter aggregates them behind a unified interface. If you're starting fresh, check whether your provider natively supports structured outputs—a feature that enforces schema compliance at the API level, reducing the need for complex post-processing. This is particularly valuable in production, where reliability trumps flexibility.
Why Prompting for JSON Isn't Enough
Before diving into solutions, let's understand the problem. When you ask a model to "return JSON," several things can go wrong, even with a well-crafted prompt:
- Format errors: The model returns markdown with
```jsonfences, or trailing commas that crashJSON.parse. - Schema violations: Even valid JSON might not match your expected structure—missing fields, wrong types, or extra properties.
- Content hallucinations: The model fills in plausible-sounding but incorrect data, especially for dates or numeric IDs.
- Inconsistency: Minor prompt changes can lead to wildly different output formats, making downstream code fragile.
Prompting is a soft constraint. It guides, but it doesn't guarantee. For production systems, you need hard constraints. That's where the three mechanisms come in.
The Three Ways to Constrain Output
By 2026, the industry has converged on three primary approaches to force models into a structured format. Each has trade-offs in complexity, flexibility, and provider support.
1. Prompt Engineering Plus Parsing Hacks
The oldest approach: write a detailed prompt, then use regex or string manipulation to extract the JSON block, clean it up, and parse it. This is brittle and error-prone, but it works with any model and requires no special API features. If you're stuck with a model that doesn't support structured output, this is your only option. You'll want a robust extraction function that handles code fences, leading text, and trailing garbage.
2. Function Calling (or Tool Use)
Introduced in 2023 and much improved since, function calling lets you define a function schema and force the model to output arguments that match it. The API returns a structured object with the arguments as a JSON string. This is a significant step up from prompt hacking. You define the parameters with JSON Schema, and the model is constrained to produce only those parameters. The output is still a string, but it's more likely to be valid.
3. Native Structured Outputs
The state of the art in 2026. Providers like OpenAI, Anthropic, and Google now offer structured output modes where the API guarantees the response matches a given JSON Schema. The model is constrained at a fundamental level—either through constrained decoding or a disciplined post-processing pipeline that validates and resamples until compliance. For OpenAI, this is the response_format parameter with type: "json_schema". You provide a schema, and the API handles the rest. This is the recommended default for new projects. It dramatically reduces the need for retries and stabilizes the output format.
Here's a decision matrix for choosing:
- Use native structured outputs if your provider supports it. It's the most reliable and requires minimal code.
- Use function calling if you need compatibility with a narrower API or are already using tools for other purposes.
- Rely on prompt tricks only as a fallback for legacy models or unusual requirements.
Start with the Schema, Not the Prompt
Once you've chosen your constraint mechanism, the next step is designing the schema. This is more important than the prompt itself. The schema is a contract that the model must fulfill, and it shapes the model's behavior more than any instructional text.
For Temploracraft, I define a Pydantic model for every structured output. This gives me two things: a clear definition of the expected data and a built-in validation layer.
Example schema for a résumé section:
from pydantic import BaseModel, Field
from typing import List
class EmploymentEntry(BaseModel):
company: str = Field(description="Company name")
title: str = Field(description="Job title")
start_date: str = Field(description="ISO format YYYY-MM-DD")
end_date: str | None = Field(description="ISO format, null if present")
class Resume(BaseModel):
full_name: str
employment_history: List[EmploymentEntry] = Field(min_length=0, max_length=10)
Writing the schema first forces you to think about your data requirements before prompting. If you define start_date as a string, you accept any format. If you define it with a regex pattern for YYYY-MM-DD, you constrain the model's output and make validation easier. The richer your schema, the better the model can comply. Descriptions matter—they act as hints for the model, especially when field names are ambiguous.
But here's the trap: don't write a long prompt. A verbose prompt that re-explain the schema often confuses the model. Your prompt should be minimal, delegating structural enforcement to the schema itself. For example, a good prompt might be:
"Extract the employment history from this résumé. Follow the schema exactly."
No more. The schema carries the weight.
Validation Is Two Jobs, Not One
Even with native structured outputs, you need a validation step. Validation serves two distinct purposes, and conflating them is a common mistake.
Syntactic Validation
This checks the structure: Is it valid JSON? Does it match the schema? Are all required fields present? Are types correct? This is what Pydantic does when you instantiate your model. It's deterministic and should be handled automatically. With native structured outputs, byte-level compliance is often guaranteed, so this step might be a formality—but it's a cheap sanity check.
Semantic Validation
This checks the meaning: Is the extracted date plausible? Is the trimmed text still meaningful? Does a field that should be an email actually look like an email? Semantic validation requires business logic. For example, a résumé parser might reject a start_date of "2050-01-01" as implausible, even though it's syntactically valid. Or it might check that end_date is after start_date.
Here's a concrete example from Temploracraft: we accept any string for a candidate's name, but we strip whitespace and check length. If the name is 200 characters long, it's probably a hallucinated dump, so we reject it.
Validation is your last line of defense. It's where you catch the model lying, even when the format is perfect. If you skip semantic validation, you'll end up with structured garbage—data that looks fine to a parser but is nonsense to a human.
Building a Retry Loop That Doesn't Burn Tokens
When validation fails, your instinct is to retry—the model isn't perfect, after all. But naive retries are expensive, both in latency and cost. A well-designed retry loop needs to be smart.
The first principle is to fail fast. If the response is structurally invalid (doesn't parse, doesn't match schema), there's no point in trying to salvage it. You can retry, but you should change something. Adding a single error message to the prompt can work wonders:
"Your previous response did not match the schema. Ensure all fields follow the descriptions."
This gives the model feedback, often fixing subtle format issues on the next attempt.
The second principle is to bounded retries. In 2026, the cost of tokens is lower, but it's not zero. And more attempts mean higher latency. Set a maximum—three attempts is a reasonable default—and after that, fail gracefully. A human should not be blocked, but the system should log the failure for debugging.
The third principle is to reduce risk on retry. If the first attempt fails, consider loosening constraints temporarily. For example, if the model can't produce a strict ISO date, fall back to a more permissive pattern, extract the date with a parser, and then normalize it. This adds complexity but improves resilience.
Here's a sample loop pseudocode:
for attempt in range(max_attempts):
response = client.chat.completions.create(
model=model_name,
messages=messages,
response_format={"type": "json_schema", "json_schema": schema},
)
try:
data = validate_semantically(response.content)
return data
except ValidationError as e:
messages.append({"role": "user", "content": f"Error: {e}. Fix it."})
raise SystemError("Max retries exceeded")
Notice the error message is specific. Instead of saying "JSON invalid," you'd say "'start_date' is not in format YYYY-MM-DD." This granularity makes the retry effective.
Streaming Structured Output
Sometimes you can't wait for a complete response—especially with long documents. Streaming allows you to show the user partial results while the model finishes. But streaming complicates structured output because you're receiving fragments.
By 2026, providers have improved support, but you still need to handle incremental parsing. The approach is to accumulate the fragments and attempt validation only when you detect a probable boundary. For example, you might parse incremental JSON using a library like json-stream or wait for a special token indicating the end. The key is to buffer and only validate at the end, while displaying a lower-confidence version in the UI.
For our résumé tool, we stream the extracted fields into a sidebar as they're parsed, but the final database write only happens after full validation. This gives perceived speed without compromising reliability.
Implementing a streaming solution correctly is a project in itself—worth it for user experience, but not if you're just starting out. For v1, generate the full response, then stream it to the UI for display.
The Failures You Can't Retry Away
No amount of schema engineering, validation, or retrying fixes some failures. You need to design for the long tail of bad inputs.
The most common un-fixable failure is ambiguous input. A résumé with no company names, or a date "2/3" with no year, is inherently ambiguous. The model must hallucinate a guess, and you should detect that uncertainty and ask the user to confirm. In my tool, when the confidence score is low for a field, we render it in yellow with a prompt to correct it.
The second class is model limitations. Sometime the model just can't do the reasoning—like determining the correct chronological order of jobs on a poorly formatted résumé. Retrying doesn't help because the model can't improve. Instead, you should gracefully degrade: extract what you can, leave placeholders, and flag for manual review.
Finally, there are cost ceilings. If the input is massive and complex, you might hit token limits. No retry fixes that. You need to pre-process the input, chunk it, or use a more efficient model. Meanwhile, your validation should be prepared for timeouts.
The key is to classify failures: retryable (format errors, minor schema mismatches) vs. non-retryable (semantic ambiguity, model limitations). Non-retryable failures should go to a human via a review queue, not on a loop.
Practical Checklist
Here's a checklist to apply to your own system:
- Use a rich schema. Add descriptions, formats, and constraints to every field.
- Prefer native structured outputs. If your provider supports it, use it.
- Separate syntactic and semantic validation. Use tools like Pydantic for the former, and write custom logic for the latter.
- Design a smart retry loop. Fail fast, provide specific feedback, and bound the number of attempts.
- Stream if you must, but validate fully before persisting.
- Plan for non-retryable failures. Detect uncertainty, degrade gracefully, and route to humans.
Getting structured data out of an LLM isn't just about parsing a response. It's about the entire pipeline: constraining the output, modeling your data, validating for meaning, and handling the edge cases. Invest in this layer, and your application will feel reliable—even when the model isn't.
via FreeCodeCamp