Get structured JSON out of an LLM — reliably

You ask a model for JSON. It replies:

Sure! Here's the JSON you requested:

\`\`\`json
{"name": "Priya", "age": 29}
\`\`\`

Let me know if you need anything else!

json.loads chokes. So you write a regex to pull out the braces, and it works — until the day the model returns trailing commas, or a field you never asked for, or an age of "twenty-nine".

There is a proper fix, and it is not a better prompt. It is a schema.

Why prompting alone is fragile

Asking politely for JSON leaves three failure modes open:

1. Wrapper text — explanations before and after
2. Wrong types — "29" instead of 29, "yes" instead of true
3. Missing or extra fields — no age, or a bonus notes key

Each is rare. Across thousands of calls, rare becomes constant. And every one of them lands as a crash in production rather than a warning in development.

Step 1 — Define the shape you want

Write the schema before writing the prompt. JSON SchemaA standard way to describe the exact shape of a JSON object is the vocabulary both you and the model understand:

INVOICE_SCHEMA = {
    "type": "object",
    "properties": {
        "vendor": {"type": "string", "description": "Company that issued the invoice"},
        "invoice_number": {"type": "string"},
        "total": {"type": "number", "description": "Total amount, digits only"},
        "currency": {"type": "string", "enum": ["INR", "USD", "EUR", "GBP"]},
        "line_items": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "description": {"type": "string"},
                    "amount": {"type": "number"},
                },
                "required": ["description", "amount"],
            },
        },
    },
    "required": ["vendor", "invoice_number", "total", "currency"],
}

Two details do most of the work. The description on each field is read by the model — treat it as instruction, not documentation. And enum removes an entire class of variation: currency can only ever be one of four strings.

Step 2 — Use tools instead of asking nicely

Every serious model API supports tool use, and it is the reliable path to structured output. You describe a "tool" whose input is your schema, then force the model to call it:

import os
import json
import anthropic

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

def extract_invoice(text):
    message = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1500,
        tools=[{
            "name": "record_invoice",
            "description": "Record the structured details of an invoice",
            "input_schema": INVOICE_SCHEMA,
        }],
        tool_choice={"type": "tool", "name": "record_invoice"},
        messages=[{"role": "user", "content": text}],
    )

    for block in message.content:
        if block.type == "tool_use":
            return block.input

    raise ValueError("Model did not call the tool")

tool_choice is the important line. It removes the model's option to write prose instead, so the response arrives as a parsed Python dict — no code fences, no "Sure!", no regex.

Step 3 — Validate anyway

A schema guides the model; it does not physically prevent every mistake. Validate at the boundary, where a bad value can still be caught cheaply:

python -m pip install pydantic

from typing import Literal
from pydantic import BaseModel, Field, ValidationError

class LineItem(BaseModel):
    description: str
    amount: float

class Invoice(BaseModel):
    vendor: str
    invoice_number: str
    total: float = Field(gt=0)
    currency: Literal["INR", "USD", "EUR", "GBP"]
    line_items: list[LineItem] = []


def parse_invoice(raw):
    try:
        return Invoice.model_validate(raw)
    except ValidationError as error:
        print("Validation failed:", error)
        return None

PydanticA Python library that checks data against a typed model and converts it gives you real Python objects with autocompletion, plus rules a JSON Schema cannot express as neatly — Field(gt=0) rejects a negative total outright.

Pydantic can also generate the schema, so the two never drift apart:

INVOICE_SCHEMA = Invoice.model_json_schema()

Define the shape once, in one place. This is the version to use in real projects.

Step 4 — Retry on failure, with the error attached

When validation fails, do not retry blindly. Tell the model what was wrong:

def extract_with_retry(text, attempts=3):
    last_error = None

    for attempt in range(attempts):
        prompt = text
        if last_error:
            prompt = f"{text}\n\nYour previous attempt failed validation:\n{last_error}\nFix it."

        raw = extract_invoice(prompt)
        try:
            return Invoice.model_validate(raw)
        except ValidationError as error:
            last_error = str(error)
            print(f"Attempt {attempt + 1} failed, retrying")

    raise ValueError(f"Gave up after {attempts} attempts: {last_error}")

Models correct themselves well when shown the specific error. Two attempts fix the overwhelming majority of failures.

Step 5 — Fall back to text parsing when you must

For an API without tool support, harden your parsing:

import json
import re

def extract_json(text):
    fenced = re.search(r"```(?:json)?\s*(\{.*?\}|\[.*?\])\s*```", text, re.S)
    if fenced:
        candidate = fenced.group(1)
    else:
        start = min((i for i in (text.find("{"), text.find("[")) if i != -1), default=-1)
        if start == -1:
            raise ValueError("No JSON found")
        end = max(text.rfind("}"), text.rfind("]"))
        candidate = text[start:end + 1]

    candidate = re.sub(r",\s*([}\]])", r"\1", candidate)   # trailing commas
    return json.loads(candidate)

Prefer the fenced block when present, fall back to outermost braces, strip trailing commas. It is a workaround, not a solution — use tools when you can.

Step 6 — Set temperature to 0 for extraction

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1500,
    temperature=0,
    tools=[...],
    tool_choice={"type": "tool", "name": "record_invoice"},
    messages=[{"role": "user", "content": text}],
)

TemperatureHow much randomness the model uses when choosing words should be 0 for extraction and classification. Creativity is a feature when writing and a bug when parsing an invoice.

A complete extraction pipeline

def process_documents(paths):
    results, failures = [], []

    for path in paths:
        text = open(path, encoding="utf-8").read()
        try:
            invoice = extract_with_retry(text)
            results.append(invoice.model_dump())
        except Exception as error:
            failures.append({"path": path, "error": str(error)})

    print(f"{len(results)} succeeded, {len(failures)} failed")
    return results, failures

Collect failures rather than crashing on the first one. Batch jobs should always finish and report.

Common problems (and fixes)

FAQ

How do I get reliable JSON output from an LLM?

Define a JSON Schema and pass it as a tool with tool_choice set to that tool. The response then arrives as a parsed object rather than text you have to scrape.

Why does the model add text around my JSON?

Because a plain prompt allows it to. Forcing a tool call removes the option to produce prose, which is why tools are far more reliable than prompt instructions alone.

Should I use Pydantic with LLM output?

Yes. Generate the schema from your Pydantic model so there is a single definition, then validate every response before it reaches the rest of your code.

What temperature should I use for structured extraction?

Zero. Extraction and classification want determinism; randomness only helps when you are generating creative text.

How should I handle a response that fails validation?

Retry with the validation error included in the prompt so the model can correct it. Two attempts resolve almost all failures; log the rest instead of crashing the batch.

Next reading on Sythra Articles

• Talk to an AI from Python in about 20 lines
• Build a chatbot that answers from your own PDFs
• Stop memorizing Python syntax