Pydantic models and Pydantic AI / docs / embedding

A pydantic model as the question sheet, and a Pydantic AI validator that judges every response before it ships.

bash

pip install "vegaml[structured]"

python

from typing import Literal
from pydantic import BaseModel, Field
from vegaml.structured import decide_model

class Ticket(BaseModel):
    team: Literal["billing", "technical", "sales"] = Field(
        description="Which team handles this?")
    churn: bool = Field(description="Does the customer threaten to leave?")
    severity: int = Field(ge=1, le=3, description="How severe?",
                          json_schema_extra={"criteria": ["cosmetic", "degraded", "outage"]})

ticket, result = decide_model(v, "I was billed twice, refund me or I cancel.", Ticket)

ticket.team                   # "billing", validated
result["answers"]["team"]     # probabilities, prediction set, abstain: still all there

How fields map#

python type	becomes	notes
`bool`	a boolean question	the field description is the statement
`Literal["a", "b"]`	a choice over those values	str Enums work the same way
`int`	a score	needs ge and le; the levels span the range

Criteria for a score come from json_schema_extra, or are numbered when the range speaks for itself. The field description is required, because it is the instruction the model reads: a field with no description is a question with no question in it.

What you keep and what you gain#

The instance comes through pydantic validation, so an answer outside the field's type is a ValidationError rather than a quiet corruption. The second return value is the decide() dictionary untouched, so going typed loses none of the calibrated detail.

python

ticket, result = decide_model(v, text, Ticket)

if any(result["answers"][q]["abstain"] for q in result["answers"]):
    escalate(text, result)
else:
    act(ticket)              # a real object, with real types

Pydantic AI#

The agent framework, as opposed to the models. The natural seam is an output validator: the agent generates, the decision model judges the response against the task, and a weak response goes back for another pass instead of out the door.

python

from pydantic_ai import Agent, ModelRetry

agent = Agent("openai:gpt-5")

JUDGE = {
    "verdict": {"type": "choice",
                "instructions": "The record is a task and a model's response. "
                                "What should happen to the response?",
                "criteria": {"approve": "does what the task asked, usable as written",
                             "revise": "right direction, needs a fix before it goes out",
                             "reject": "wrong or off the task; start again"}}}

@agent.output_validator
def judged(ctx, output: str) -> str:
    record = "Task: " + str(ctx.prompt) + " Response: " + output
    verdict = v.decide(record, JUDGE)["answers"]["verdict"]
    if verdict["abstain"] or verdict["choice"] != "approve":
        raise ModelRetry("the judge said " + verdict["choice"] + "; fix and resend")
    return output

The judge is a forward pass, so putting it on every response costs milliseconds, not a second LLM call. One honest caveat from a live run of this sheet, cold: it caught an explicit rule break (a link in a no-links reply) at 0.80, but approved a factually wrong answer at 0.84. Unfitted, it is a rule-break detector, not a fact checker; pair it with a grounding check when the response cites a source, and fit it on your own judged responses before trusting the verdict. The same sheet ships hosted as the ai-judge template in Studio, where a few hundred of your own judged responses fit it to your bar.

Note

Per-option definitions are worth more than a tidy schema.

A Literal gives the model bare option names with no definitions, and definitions are where most of the accuracy lives. If a choice is doing real work, write it as a plain question sheet with one-line criteria per option (see Questions and answers) and keep pydantic for the fields whose names already say what they mean.