Fit and calibrate, not fine-tune

Every decision model that learns from your labels does it the same way. Jev fine-tunes. Prism, the Sarvam one, has no fitting surface at all, so you take the zero-shot answer or you leave. Our own laya checkpoints fine-tune too, with reinforcement learning against a proper scoring rule, and that is the better version of the idea. But it is still the same idea: run a gradient through the language model until it agrees with you.

vega does not do that, and as of this week the whole of it sits behind one argument in laya: laya.Agent(vegaml=True). The reader, a 0.8B Qwen3.5, is frozen and never sees a gradient. What learns from your labels is a head over its frozen features, a temperature, a quantile and one line. Each of those is either a closed form or a one-dimensional search. Fitting takes seconds on a laptop, and every piece has a statement you can write down about when it is right. This post is the mathematics of that, in the order the code runs it, with the measured record where we have one.

What you point it at

The same three question types, choice, score and noul, and the same answer shape with a set, an abstain flag and a route, whatever the input is. These are the cases it was built for, three per input, and the ones to try first.

And the whole of it in one block: every input through the same call, the same questions, and the fields that come back on every answer.

pip install -U laya                          # 0.4.24 or newer; vegaml comes with it

import laya
from laya.vega import Router

agent = laya.Agent(vegaml=True)              # the 0.8B universal reader; vegaml="4b" for the larger one

questions = {
    "team":    {"type": "choice", "instructions": "Which team should handle this?",
                "criteria": {"billing": "invoices, payments, refunds", "hr": "jobs, payroll, leave",
                             "pr": "press and media", "sales": "pricing, demos, quotes",
                             "technical": "bugs, outages, API questions"}},
    "urgency": {"type": "score", "instructions": "How soon does this need a reply?",
                "criteria": ["can wait", "this week", "today"]},
    "churn":   {"type": "noul", "instructions": "Does the writer threaten to leave?"},
}

# text
out = agent.predict("We were billed twice for March. Refund it today or we cancel.", questions)
# image, speech, video: the same questions, the same answer shape
out = agent.predict(image="invoice.png", questions=questions)               # PIL image, path or array; ~1 megapixel
out = agent.predict(audio="call.wav", questions=questions)                  # any length; transcribed at 16 kHz, transcript rides along
out = agent.predict(video="meeting.mp4", questions=questions)               # speech if there is any, else 4 frames as one image
# long documents: up to 73,728 tokens in one pass, no windows
out = agent.predict_long(seventy_page_contract, questions)

# what every answer carries
a = out["answers"]["team"]
a["choice"]            # "billing"                     the answer (score -> a["score"], noul -> a["noul"])
a["probabilities"]     # {"billing": 0.91, "hr": 0.04, ...}
a["prediction_set"]    # ["billing"]                   the conformal set at alpha 0.1: what it could still be
a["abstain"]           # False                         the set came back empty: the engine refuses
a["unbound"]           # False                         far from every well: likely off-distribution
a["route"]             # "auto" | "review"             decided for you; a["route_reason"] says why

# learn your labels: a CSV with a state column and one column per question id
agent.fit("routing_labels.csv", questions=questions)   # heads, then temperature, conformal threshold, review line
agent.save_calibration("fit/")                         # agent.load_calibration("fit/") later

# model switching: laya's 33 ms encoder for short English text, vega for the rest
router = Router(fast=laya.Agent(), vega=agent)
out = router.predict(state, questions)
out["routing"]         # {"model": "fast" | "vega", "reason": "..."}; model="vega" forces one

What a frozen reader gives you

The prompt is plain text with recorded span positions: the state, then the typed question, then one line per option, then an outcome marker. The reader runs a prefix of its layers and stops; on the 0.8B we tap layers 13 and 19 of 24 and never execute the rest, so the LM head that would generate text is dead weight we skip. Features are mean-pooled per span at each tapped layer, standardised per layer and dimension with statistics stored inside the engine, and mixed by a learned softmax over the two depths. Call the result for one prompt $x\in {\mathbb{R}}^d$. Because the state precedes the question under causal attention, the state part of $x$ does not depend on what you ask, which is what lets one long record be read once and queried many times.

The engine on top of $x$ is small, 14M parameters, and it is a physical system rather than a classifier. Each option $k$ becomes a Gaussian well in a 64-dimensional space with a centre $c_k$, a depth $a_k$ and a width ${\sigma }_k$, all emitted by a small head from the option features and the question. The question drops a particle at $z_0$ with momentum $p_0$, and the particle moves in

under a damped symplectic step whose step size and friction come from a small state-space model reading the particle's own state, at most twelve steps, exiting early once kinetic energy and the gradient norm are both small. The answer is Boltzmann occupancy over the wells at wherever it stopped:

Two things fall out of the geometry for free, and both of them matter later. The distance from the settled point to the nearest well is a number, so a particle that stops far from every well can be flagged: that is the unbound field, an out-of-distribution signal that costs nothing extra. And the occupancy is recorded at every step and supervised at every step during training, so an early exit returns a quantity that was trained, not an approximation of one.

That is the zero-shot path, and it is honest about its limits: on a taxonomy it has never seen it is bad. On our own typed-decisions suite the cold engine scores 0.3945. The rest of this post is about what happens to that number without touching the reader or the engine.

Fit: a convex problem over frozen features

Give the agent labelled rows for a question and it fits one head for that question. The head is the least interesting model in machine learning, deliberately. Standardise the features, project to $k=128$ principal directions, standardise again, multinomial logistic regression with an ${\ell }_2$ penalty of $10^{-2}$:

Equation 4 is strictly convex, so it has one minimiser and full-batch gradient descent reaches it. There is no seed to pick, no learning rate schedule, no early stopping and no checkpoint to choose between. Four hundred steps on a $128\times K$ matrix is a few milliseconds; the whole cost of fitting is reading the rows through the frozen reader once, which is why fitting 1,200 cases took 24 minutes on a T4 and the heads themselves took none of it. At serve time the head is one matmul, measured at 169 microseconds on the same T4, so a fitted answer costs the same as an engine answer.

The part I think is actually novel is not the head, it is the rule for whether the head is allowed to answer. A head is cheap enough to fit that you will fit it on too little data, and a head fitted on noise returns a sharper wrong answer than the engine would have. So every head has to clear two bars, and both are measured on rows it did not see. Stratified 5-fold cross-validation gives ${\mathrm{acc}}_{\mathrm{cv}}$, the accuracy of a head on each example scored by a head that never saw that example. The engine is then scored on exactly the same rows, which gives ${\mathrm{acc}}_{\mathrm{engine}}$. The head is

• kept only if ${\mathrm{acc}}_{\mathrm{cv}}>1/K$, chance for a question with $K$ options; below that it has learned nothing and is discarded;
• served only if, in addition, ${\mathrm{acc}}_{\mathrm{cv}}>{\mathrm{acc}}_{\mathrm{engine}}$ on those same rows.

Otherwise the engine keeps the question, and the report says so in a sentence. A fine-tuned model has no equivalent of this switch. Once the gradient has run, the old model is gone, and whether the new one is better than it was is a separate experiment you have to remember to run. Here the comparison is part of the fit, per question, and the fitted model is allowed to lose.

It does lose, and that is the measurement I trust most. On the typed-decisions suite with 1,200 fitting cases, all twenty namespaced heads cleared both bars, and held-out accuracy went from 0.3945 to 0.7125, with margins like invoice_processing/duplicate at cv 0.943 against the engine's 0.093. On boolq with 60 rows the head scored cv 0.567 against the engine's 0.717, was refused, and held-out accuracy moved by exactly 0.0. Same refusal on sst5 and emotion. The one place the rule bit us was 30 rows on boolq, where a head was promoted on a margin of 0.07 that turned out to be two rows of noise, and held-out accuracy fell from 0.750 to 0.525. The bar is sound; the risk is small $n$. The notebooks now print $({\mathrm{acc}}_{\mathrm{cv}}-{\mathrm{acc}}_{\mathrm{engine}})\cdot n$ and flag it below 6.

The banking number from the vega-U post is the same mechanism: zero-shot 0.319 on the hard banking77 subset, 0.853 after a self-supervised attach and a head fitted on 2,000 labelled rows, with the engine bytes unchanged.

Calibrate: three scalars per question, each with a theorem

A head gives you a sharper argmax. It says nothing about whether 0.91 means 91 percent, and a head fitted on a few hundred rows is usually overconfident. Calibration refits that on your rows, and it is three numbers per question.

Temperature. One scalar $T$ rescales the log-probabilities,

found by golden-section search in $\log T$. The negative log-likelihood is a strictly proper score, so its minimiser is the honest temperature and not merely a flattering one. And because dividing every logit by the same positive $T$ is monotone, the argmax is invariant: calibration moves confidence, sets and routes, never which option wins. That invariance is the reason the two steps can be separated at all. Fit decides the answer; calibrate decides how much to trust it.

The conformal quantile. With the temperature applied, each calibration row gets a nonconformity score $s_i=1-p_T(y_i)$, and

the $\lceil (n+1)(1-\alpha )\rceil$-th smallest score. Split conformal prediction (Vovk, and the Angelopoulos and Bates tutorial) gives $\mathrm{Pr}[y\in \mathcal{C}(x)]\ge 1-\alpha$ for a new row exchangeable with the calibration rows, with no assumption about the model. That is the prediction_set field, and abstain is the set coming back empty. Measured coverage of the 90 percent sets on held-out items is 89.0 to 90.3 percent.

Here is a fact about equation 6 that decides the next design choice. An empty set needs every option below $1-\hat{q}$, and with $K$ options at least one of them has probability at least $1/K$, so abstention is geometrically impossible unless $\hat{q}<1-1/K$. On a two-option question that means $\hat{q}<0.5$, which a well-calibrated model on a clean corpus will not produce. Across the six corpora we published, the conformal abstain never fired once, and on two of them it could not have. A refusal nobody can trigger is not an escalation policy.

The line. So the third number is a confidence threshold $t$ per question, below which the answer goes to a person. The rows with the temperature applied give pairs $(c_i,{\mathrm{ok}}_i)$, confidence and whether the argmax was right. For a candidate $t$, the automatic band is $\{i:c_i\ge t\}$ with $m$ rows and $h$ hits, and instead of its observed precision $h/m$ we use Wilson's one-sided lower bound,

the widest band whose precision holds at the target $\pi$ (0.95 by default) with 95 percent confidence to spare. Wilson rather than the textbook normal interval because the band that matters is the one near precision 1.0, where the normal interval runs past 1 and at $h=m$ collapses to zero width: twelve for twelve would certify any target at all. The lower bound makes a narrow band pay for being narrow. And because $t^{\star }$ is chosen on the rows that then score it, the report also carries a cross-fitted number: the same rule refit on four fifths and scored on the fifth, which is the figure to quote. When no line can hold the target, the fit says so, sends everything to review, and names the best target this much data can support. It does not quietly ship a 91 percent pilot under a 97 percent label.

This is what automatic thresholding means in laya now. Nobody passes a confidence number. route is decided per answer: the fitted line if one exists, and otherwise the engine's own machinery, which is review when it abstains, when the particle is unbound, or when the conformal set holds more than one option. Both $\alpha$ and $\pi$ are targets with stated meanings and defaults, not thresholds somebody guessed. The house rule behind it is a measurement, not a preference: a fitted head once reported confidence 1.000 on every one of sixteen held-out records, including both it got wrong. route on those records was still right, because it comes from the set and the line rather than from the top probability. Gate on route. Never on confidence.

Where abstain comes from, and where unbound comes from

Both are thresholds, but neither is one you set, and they measure different things.

Abstain comes from the conformal set, not from a confidence cut. After calibration each question has a quantile $\hat{q}$, fitted on the calibration rows so that the set covers the true answer 90 percent of the time. The set is every option whose probability is at least $1-\hat{q}$. Abstain is simply that set coming back empty: no option cleared the line. So the number behind it is $1-\hat{q}$, derived from the coverage target, and it moves when you recalibrate on your own rows. Two things follow. With $K$ options at least one of them always has probability $1/K$ or more, so abstain is geometrically impossible unless $\hat{q}<1-1/K$; on a two-option question that means $\hat{q}$ below 0.5, which is why it rarely fires on clean corpora and why the route also reads the set size and the review line rather than waiting for it. And on the cold engine the same rule runs with the published calibration, where the risk setting picks which fitted level is used: the same mechanism, with the engine's own rows instead of yours.

Unbound has nothing to do with probabilities. It is geometric. The engine drops a particle into a landscape where each option is a well and reads the answer from where the particle settles. Unbound is set when the particle stops farther from every well than the 99th percentile of the distances seen on calibration items. So it is a fitted distance threshold, and it flags an input that looks unlike anything the engine was calibrated on. An answer can be confident and unbound at the same time: the probabilities say billing firmly, the geometry says this message does not look like the messages the wells were shaped on. That is why it sits beside abstain rather than replacing it.

One honest limit. Unbound is only produced by the engine's physics. A head-served answer, where your fitted head decided over the frozen features, carries unbound: null, because no particle ran. Abstain survives on that path, because the calibrated set is computed from the head's probabilities like any other.

How an answer becomes auto or review, step by step

Every answer from Agent(vegaml=True) carries route, either auto or review, and a route_reason saying why. The decision runs in this order, per answer:

1. If a band has been fitted, because fit or calibrate ran on labelled rows, the band's line for that question decides. Each row gave a pair, the temperature-scaled confidence and whether the argmax was right; the fit swept every candidate threshold, took Wilson's lower bound on the precision of the answers above it, and kept the lowest threshold whose bound still held the target with at least ten answers in the band. Confidence at or above that line is auto, below it is review. If no line could hold the target, every answer is review and the report names the best precision the data can support.
2. Otherwise the engine decides with no labels at all. The answer is review if abstain is true (the conformal set came back empty), if unbound is true (the particle settled far from every well, so the input looks unlike anything the engine was calibrated on), or if the conformal set at $\alpha$ holds more than one option. It is auto only when none of the three fires.
3. On top of either path, laya's old min_confidence is still honoured if you pass it, and nothing requires it.

The reason this is trusted over the probability is the measurement above: a fitted head is sharp by construction and will report 1.000 on an answer it gets wrong, and route never reads that number. agent.thresholds() prints which of the two paths is deciding for each question right now.

Model switching is a different decision, taken earlier, and the two are easy to confuse. The Router described below picks which model answers before any decision is made: an image, a recording, a clip, text past the encoder's context, non-English text or a question with a fitted head go to vega, everything else to the fast encoder, reported in routing. The auto or review call happens after that, on whatever answer the chosen model gave.

Why not fine-tune, in three equations' worth of reasons

Calibration does not survive a gradient. $T^{\star }$, $\hat{q}$ and $t^{\star }$ are fitted for one function $x\mapsto p$. Fine-tune the reader and that function changes everywhere, so all three are arithmetic without meaning and nothing in the numbers tells you. With the reader frozen, calibration is a property of the question and your rows, and the file that holds it records the SHA-256 of the engine and the name of the reader it belongs to, and refuses to load elsewhere.

Conformal validity needs exchangeability. The guarantee under equation 6 holds between the calibration rows and the rows you serve. Rows the model was trained on are not exchangeable with fresh ones, so a fine-tuned model needs a third split just to calibrate. A head fitted on frozen features and scored by cross-validation keeps the arithmetic small enough that the honest version is cheap: agent.fit(rows) does both on one set for convenience and the report says in words that its ECE is measured on the fitting rows; pass a disjoint set to agent.calibrate and equation 6 holds as written.

Nothing else can move. A head is per question. It reads frozen features and writes one answer, so every other question the model answers is unchanged by construction, with KL exactly zero. We measured the alternative on the engine itself: fine-tuning it end to end on a new task family scored 0.783 on that family and dropped JevBench from 0.623 to 0.602 while breaking two behavioural probes. A gated adapter on the same engine scored 0.784 with JevBench rising to 0.645 and nothing else moving. Learning that cannot forget is a structural property here, not a regularisation strength.

The port: switching the LLM is a ridge regression

The engine lives in a shared 1024-dimensional space that no reader owns. The reference reader, Qwen3.5-0.8B, is fixed as the identity. Every other reader $b$ with hidden width $d_b$ gets one linear map per tapped layer, $P_{b,l}:{\mathbb{R}}^{d_b}\to {\mathbb{R}}^{1024}$, and joint training over the readers minimises

with $F$ and $O$ the pooled slot and option features and ${\lambda }_{\mathrm{inv}}=1$. The invariance term is what makes the space one space: matched items seen through different tokenisers are pulled to the same point, so there is nothing reader-shaped left for the engine to learn from.

Attaching a reader the engine has never met needs no labels and no engine training. Run the anchor and the new reader over the same few thousand prompts of your own text, under six generic probe questions, and solve per layer

with $\lambda$ chosen from $\{10^{-3},\dots ,10\}$ by the cosine between predicted and anchor features on a held-out tenth. Then 300 refinement steps that distil the engine's own answers through the anchor port, with the ridge target kept as a regulariser, then a calibration refit for that reader. The engine checksum is asserted before and after. A port is linear on purpose: it cannot smuggle in reasoning, so all the intelligence provably stays in the reader and all the deciding stays in the engine, and it can be fit in closed form, which is what makes switching a model routine rather than a project.

The measured ladder is 0.765, 0.791 and 0.811 held-out for the 0.8B, the 4B and the 27B, with the engine file byte-identical in every row and the two best readers the ones attached cold. The holdout cosine is the health number: about 0.8 is the guideline, and the banking attach measured 0.70 and 0.86 on its two layers in 25.9 minutes end to end. A VLM is the same procedure, because a VLM is a reader with a vision tower in front of it, and the port only ever sees hidden states.

Multimodality, under laya's names

The 0.8B Qwen3.5 already has a vision tower. For agent.predict(image=...) the image tokens take the place of the state span in the same prompt template, the same two layers are pooled over the same four kinds of span, and the engine runs on the result unchanged. It was trained on text features only, so this is zero-shot, and we say so: heads read text features and do not apply to pixels, and a calibration fitted on text is not applied to an image answer. Speech is the opposite case. For agent.predict(audio=...) a small local transcriber writes the state, and because the engine then sees text, everything above applies unchanged: heads, temperature, set, line. The transcript rides along in the answer, because a decision made on words you never saw cannot be audited, and the blame has to split cleanly between a wrong transcript and a wrong answer about a right one.

Video, since 0.4.24, is the two cases combined and chosen by content. agent.predict(video=...) pulls the sound track out through ffmpeg at 16 kHz and transcribes it; if the transcript holds real words, the clip is decided on those words through the text path, with heads, calibration, set and route all applying. If the clip is silent, or its sound is music or noise, four evenly spaced frames are tiled into one contact sheet and decided through the image path, zero-shot. The answer says which path decided and why. A pure tone, for instance, comes back from the transcriber as a row of dots, fails the word gate, and falls through to the frames.

Long documents are the third case and the one the engine was built around. A state up to 73,728 tokens is read whole: up to sixteen chunk means feed a memory reader whose output projection starts at zero, so it is an exact no-op until the chunks help. laya's predict_long used to window past 1,024 tokens and aggregate; with vegaml=True the same method reads once and reports one window.

The one block near the top of this post is the whole surface; nothing below adds a method to it.

The router is the model switching a laya user already had, extended by one branch: an image, a recording or a clip goes to vega, text past the encoder's context goes to vega, non-English text goes to vega, a question with a fitted head goes to vega, and short English text with no head stays on the 33 millisecond encoder. Every method keeps its laya name, and leaving the argument out leaves laya exactly as it was. The cost is honest too: roughly five times the latency on plain text, 72 milliseconds a decision against 16 batched on a T4, two to four gigabytes of memory, and Python 3.10 or newer.

What is new here, said carefully

None of the pieces is ours. Logistic regression, temperature scaling, split conformal prediction, Wilson's 1927 interval and ridge regression all have their own literatures, and the post cites the ones that matter. What I have not found elsewhere is a shipped decision model where learning from labels, calibration, refusal and changing the underlying language model are all done without a gradient ever passing through that language model, and where each step carries a statement about when it is wrong: a head that can lose to the zero-shot engine and say so, a set with a coverage guarantee, a line placed by a lower bound rather than a point estimate, and a port with a closed form and a health number. Fine- tuning gives you one of those at most, the accuracy, and takes the other three away in the same gradient step.

Everything above is open. The integration is laya 0.4.24, where vegaml is a core dependency, so a plain pip install laya carries all of it; the engine, ports, attach tooling and the fit and calibration code are in vegaml; the reader-switching results are in the vega-U post and the refusal machinery in conformal and abstain, built in. The numbers in this post come from runs whose notebooks and logs are in those repositories, and the ones I would check first are the two refusals, because a method that cannot lose is a method nobody measured.