Jev vs. LLM-as-a-Judge: What Changes When the Model Returns a Type
TypeSafe's Jev 1.13 is on OpenRouter: a decision model that returns a typed answer with calibrated probabilities instead of prose. We put the published benchmarks next to the LLM-as-a-judge setup they are meant to replace - batching, re-ranking, verifier cascades - and mark where the evidence stops.
TechChase Team
12 min read
Share
Since 18 September 2026, typesafe/jev-1.13 is listed on OpenRouter. The numbers on that page look wrong at first glance: $0.042 per million input tokens, $0.00 per million output tokens, 32K context, 0.26 s P50 latency, served by exactly one provider.
Free output tokens are not a promotion. They are a statement about what the model does. Jev does not write output. It returns a typed decision plus a probability distribution — the thing you were parsing out of an LLM's prose all along.
TypeSafe calls this class of model System One (after Kahneman: fast, intuitive judgment, as opposed to slow, deliberate System Two). The question worth asking is not whether that framing is clever, but whether it beats the pattern it targets: LLM-as-a-judge.
The Pattern Under Attack
LLM-as-a-judge is everywhere in production AI systems, usually unnamed:
Which team gets this ticket? → prompt a model, parse the label.
Is this retrieved passage relevant? → prompt a model, ask for 1-10.
Did the extractor hallucinate this field? → prompt a model, ask "is this good?"
Which of these 30 candidates answers the query? → prompt a model, ask for a ranking.
It works, and it has four structural costs that no prompt fixes:
You invent the scale. "Rate relevance 1-10" has no ground truth behind it. The model's 7 today and its 7 tomorrow are not the same 7, and they are definitely not the same 7 as another candidate's.
You pay for prose. Every judgment burns output tokens — priced at $4.50/M on gpt-5.4-mini and $30/M on gpt-5.5, to take the two models TypeSafe benchmarks against.
You parse. JSON mode reduces the failure rate; it does not remove the parse step or the malformed-reply branch.
You get no honest "I don't know." A judge asked for a label produces a label. Uncertainty has to be reconstructed by sampling the same call several times.
What Jev Returns Instead
Keep reading
More articles on AI, automation and software engineering from the TechChase team.
Three differences to a judge, and they are not cosmetic:
The answer space is yours, defined per question. Options and level descriptions are data in the request, not a scale the model improvises.
The probabilities are trained to be calibrated — optimized against outcomes, measured across groups of predictions. Calibration does not promise any single answer is right; it promises the numbers mean something in aggregate. That is exactly what a judge's "8/10" never promises.
48 output tokens cost $0.00. The bill is the state you sent, once.
The Numbers TypeSafe Publishes
All three results below come from TypeSafe's own cookbooks, run on jev-1.12 — the predecessor of the version now on OpenRouter, at the same price. Read them as vendor-published, reproducible-with-your-own-key evidence, not as third-party benchmarks. We flag what they do not show further down.
1. Batching is nearly free — and changes nothing
A regulatory briefing: the GDPR Wikipedia article (~54,000 characters) plus 13 questions (8 Nouls, 2 Choices, 3 Scores). Asked as one call vs. 13 calls, five times each:
Strategy
Calls
Cost
Total time
One call, all 13 questions
1
$0.000497
0.27 s
13 calls, one question each
13
$0.006090
2.71 s
12.2x cheaper, 10.0x faster. The interesting column is the one not shown: across 5 repeats, 11 of 13 answers came back with a standard deviation of exactly 0.0 under both strategies, and the two that moved moved equally under both. Jev ingests the state once and evaluates every question against it independently — no question sees the other twelve.
That is the property an LLM-as-a-judge does not have. Pack 13 rating questions into one judge prompt and the answers start correlating with each other, with their order, and with whatever the model said two lines earlier.
2. Re-ranking: a yes/no question as a ranking score
3,565 court-opinion passages from the CLERC legal dataset. BM25 builds a 30-candidate shortlist per query, then one Noul per query-candidate pair — "could this candidate be from the cited precedent?" — and the shortlist is sorted by the returned probability.
Fast search (BM25)
+ Jev re-rank
Correct passage at rank 1
5%
18%
In top 5
15%
35%
In top 10
38%
62%
1,200 calls, 1,536,002 input tokens, $0.0645 total. That is the whole re-ranking bill for 40 queries.
No scoring rubric had to be invented here: the noul is the score, comparable across pairs because the same question was applied to each. With a judge, the equivalent step means writing a 1-10 relevance rubric and hoping it is applied consistently 1,200 times.
3. Verifier cascade: Jev as the gate, not the worker
The most honest use of the model in the docs, because it keeps the generative model in the loop. Structured data extraction in three rungs:
Extract with a cheap model (gpt-5.4-mini, $0.75/$4.50 per M).
Verify with Jev: one Noul per field, phrased so that bad is true — "is this value absent from the source?", "was it lifted from unrelated text?"
Escalate to gpt-5.5 ($5.00/$30.00 per M) only if any field's P(wrong) crosses the threshold.
The worked example is a scraped NYU events page with no registration date on it. The mini model invents a plausible, schema-valid description. Jev's per-field flags fire at P=0.95 (hallucinated) and P=0.85 (off_target) — while the field that was legitimately empty stays low. The gate escalates, the reasoning model returns an honest empty string.
Note the shape of the win: a blunt "is this extraction good?" judge scored 0.56 on the same record — a shrug. The per-field questions localize the error. Across 100 prompts, sweeping that gate threshold puts the cascade's cost/quality frontier above and left of every single model run alone, including the $0.10-per-extraction reasoning model at ~0.81 quality.
Confidence Is the Part a Judge Doesn't Give You
confidence collapses the probability distribution's shape into one number from 0 to 1. A concentrated distribution means a clear read; a flat one means the model genuinely cannot separate the options. It exists on Choice and Score answers (not on Noul — there the probability is the answer).
That single number is what makes the model usable in code that has consequences:
python
action = response.answers["action"]if action.confidence < 0.5: route_to_human(user_message) # genuinely unsure - don't guesselif action.choice == "check_balance": show_balance(account_id) # low stakes, recoverableelif action.choice == "approve_transfer": if action.confidence > 0.9: confirm_then_execute(account_id) # high stakes, high confidence else: ask_user_to_confirm(account_id) # high stakes, hedge
The threshold is not one number for the whole system — it scales with what the wrong answer costs. Showing the wrong screen and approving the wrong transfer are not the same risk, and your code, not the model, encodes that difference.
Where Jev Loses
TypeSafe publishes a jaggedness page listing its own model's failure modes, which is more than most vendors do. Taken together it draws a hard boundary around where this model belongs:
Failure mode
Reality
Math, counting, numbers
Not a calculator. Counting errors grow with list length. Keep arithmetic in code.
Dates
Read as text, not as ordered quantities. Extract the parts as Choices, compare in code.
Indirection
A property of a property costs accuracy. Point at the relevant state by name.
Literal reading
It answers the question you wrote, not the one you meant. Boundary cases belong in criteria.
Large, noisy state
Context rot is real here: irrelevant material in the state costs accuracy. Filter first.
Adversarial content
State is not treated as hostile by default. Prompt injection in the state can move the answer.
Generation
Not trained for it. Bounded answer space → Choice. Free text → a different model.
And one that deserves its own paragraph, because it will bite anyone migrating a judge threshold across question types: structural invariants do not hold. The same question asked as a Noul and as a yes/no Choice returns numbers that are not comparable — the docs show noul: 0.22 next to Choice P(yes): 0.01 at confidence 0.97 on the same ticket. And a question plus its negation, as two Nouls, summed to 1.19. A Choice is relative (which option wins); a Noul is absolute (can be low for all of them). Tuning a threshold on one and reusing it on the other is a bug.
Add to that: English first (other languages, including CJK, are handled but weaker), text only (no image, audio, video), and rate limits the vendor explicitly says are moving while demand settles.
What the Benchmarks Don't Prove
Being precise about the evidence, because the numbers above are good enough that it matters:
They are vendor-published. Every figure comes from TypeSafe's own cookbooks, on datasets TypeSafe chose. The notebooks ship their API caches, so they are reproducible — but reproducible is not independent.
They ran on jev-1.12, not the jev-1.13 now listed on OpenRouter. Same price, same shape, unverified deltas.
Sample sizes are small. 40 queries in the re-ranking run. 100 prompts in the cascade sweep.
There is no head-to-head against a well-built judge. The cascade compares Jev-as-verifier against no verifier and against whole-output judging — not against gpt-5.5 asked the same narrow per-field yes/no questions. The cost gap would survive that comparison; the accuracy gap is untested.
The SDE cost chart is a historical snapshot that the docs themselves note was not recalculated at the current Jev rate.
Calibration also deserves less magic than the marketing gives it: it is a property of predictions in aggregate. A calibrated 0.8 means roughly 80% of such answers are right. It says nothing about the one in front of you.
Where We Would Actually Put It
Our take after reading the whole doc set: this is not an LLM replacement, it is a cheap, typed decision layer that lets you stop asking a generative model questions it is overqualified and badly shaped for.
Four concrete slots:
Intent routing in front of an agent. One Choice over your handlers, confidence-gated. Cheaper and more consistent than a router prompt, and it fails loudly instead of picking the first plausible tool.
Guardrails on both directions of an LLM app. One request scoring jailbreak probability and potential harm, thresholds in your code deciding pass / review / block.
RAG passage screening. A Noul per retrieved passage before the expensive model reads them — drop the irrelevant, flag the ones carrying injected instructions. Directly attacks the context rot that makes long-context RAG worse, not better.
Verifier in a cascade. Per-field, bad-is-true, max gate. This is where the published cost curve is strongest and where the failure mode is safe: a false alarm costs one reasoning-model call, not a wrong answer.
What we would not do: hand it arithmetic, dates, counting, or anything where the answer space isn't bounded before the call. The docs tell you that themselves — which, in a launch week, is the most trustworthy thing on the page.
The TechChase Take: Jev Is a Context Manager
An agent is easiest to explain as a digital employee — one word, and everything else about the architecture follows from it. Jev deserves the same treatment, and the word is not "judge" and not "classifier". It is a context manager: a cheap layer between raw, unstructured input and whatever consumes it, handing each consumer the shape it actually needs.
Our personal comment on this model: "We think that Jev is the context killer we needed so much!" Killer as in killer of context rot — it throws out the material that drags a long context down, so what reaches the expensive model is only the part that produces a precise answer. The tokens you never send are the tokens you never pay for.
Three consumers, three different needs:
For humans — structure, immediately. One email, eight categorisations in a single call: department, urgency, sentiment, intent, product area, refund requested, VIP, escalation risk. No rubric to invent, no prose to skim, no parse step. Batching is effectively free and the answers do not contaminate each other (the 12.2x run above), so "add another question" becomes a design decision instead of a budget decision.
For agents — the right context, not more context. This is where the killer earns its name. RAG: one Noul per passage, drop everything under the line before the reasoning model ever sees it. Memory: decide what is worth writing down and what is worth recalling, with a calibrated probability instead of a cosine score you cannot threshold honestly. Model routing: one Choice over your handlers or model tiers, confidence-gated — the hard 20% goes to the expensive model, the rest does not.
For data and analytics — a schema on top of the unstructured pile. Ten thousand tickets, reviews or incident reports: run the three primitives across them and you have columns. Or add a field to a table that already exists — a sentiment, a risk level, a category nobody captured when the rows were written — without starting a labelling project.
One model, three primitives, three completely different jobs. That is the part we find genuinely interesting: context management was always one problem, and everybody was solving it with a text generator at a hundred times the price.
Takeaway
LLM-as-a-judge was always a workaround. We used a text generator for classification because a text generator was the only thing that understood the text. Jev's bet is that this specific job — bounded answer space, calibrated probability, no prose — deserves its own model class, with free output tokens as the natural consequence.
The published evidence supports the cost and consistency claims strongly and the accuracy claims narrowly. At $0.042 per million input tokens, the correct response is not to believe the benchmarks. It is to re-run them on your own data, where a real evaluation costs less than a coffee.
