Most model comparisons on the internet share a flaw: the numbers for the two systems come from different runs, different samples, different prompts. When we put vega against Jev, the hosted decision model in this category, we wanted a comparison that survives hostile reading. So: 3,841 decisions per system, the same rows, one run, vega on local GPUs and Jev over its live endpoint, with the harness and the raw JSON published next to the result.
What the run found
Jev wins raw accuracy on the public menu, 0.836 against 0.717, and we print that number as large as the ones we win. vega answers every row where Jev abstains on a few, and on the probability side the gap runs the other way: expected calibration error of 0.098 against 0.122, and KL divergence to the soft gold labels of 0.854 against 6.099. A seven times smaller KL means that when the right answer deserves 60% confidence, vega says something near 60%, while the hosted model says something near certain.
Calibration is not a consolation prize. A decision model's output is consumed by an if statement. The branch that matters is the one where you trust the probability enough to act without a human, and that branch is exactly where miscalibration costs money.
Latency, measured in the same breath
The hosted number includes the network, because that is what a hosted model costs you in production. The local numbers include everything after the request reaches the process.
The rules we benchmark by
- Numbers from the published result files only, never retyped from memory of a different run.
- The harness pins its library versions and reruns on two T4s in an afternoon, so the result is checkable by anyone with a free GPU quota.
- Losses get printed at the same size as wins. The accuracy gap above is real, and closing it is what the next series is for.
Full tables and raw output: vega results and laya benchmarks.