Jev vs. LLM-as-a-Judge: What Changes When the Model Returns a Type
TypeSafe's Jev 1.13 is on OpenRouter: a decision model that returns a typed answer with calibrated probabilities instead of prose. We put the published benchmarks next to the LLM-as-a-judge setup they are meant to replace - batching, re-ranking, verifier cascades - and mark where the evidence stops.
TechChase Team
10 min read
Share
Since 18 September 2026, typesafe/jev-1.13 is listed on OpenRouter. The numbers on that page look wrong at first glance: $0.042 per million input tokens, $0.00 per million output tokens, 32K context, 0.26 s P50 latency, served by exactly one provider.
Free output tokens are not a promotion. They are a statement about what the model does. Jev does not write output. It returns a typed decision plus a probability distribution — the thing you were parsing out of an LLM's prose all along.
TypeSafe calls this class of model System One (after Kahneman: fast, intuitive judgment, as opposed to slow, deliberate System Two). The question worth asking is not whether that framing is clever, but whether it beats the pattern it targets: LLM-as-a-judge.
The Pattern Under Attack
LLM-as-a-judge is everywhere in production AI systems, usually unnamed:
Which team gets this ticket? → prompt a model, parse the label.
Is this retrieved passage relevant? → prompt a model, ask for 1-10.
Did the extractor hallucinate this field? → prompt a model, ask "is this good?"
Which of these 30 candidates answers the query? → prompt a model, ask for a ranking.
It works, and it has four structural costs that no prompt fixes:
You invent the scale. "Rate relevance 1-10" has no ground truth behind it. The model's 7 today and its 7 tomorrow are not the same 7, and they are definitely not the same 7 as another candidate's.
You pay for prose. Every judgment burns output tokens — priced at $4.50/M on gpt-5.4-mini and $30/M on gpt-5.5, to take the two models TypeSafe benchmarks against.
You parse. JSON mode reduces the failure rate; it does not remove the parse step or the malformed-reply branch.
You get no honest "I don't know." A judge asked for a label produces a label. Uncertainty has to be reconstructed by sampling the same call several times.
What Jev Returns Instead
You send a state (a string, a JSON object, or an array of text) and a set of questions. Three primitives exist, and that is the whole vocabulary:
Primitive
Question shape
Answer
Choice
Which team should handle this ticket?
choice: "billing" + probabilities per option + confidence
Score
How frustrated is this customer?
score: 1.035 over ordered levels + legend + confidence
Noul
Does this message request a refund?
noul: 0.95 — the probability that the answer is yes
One call, three questions, from TypeSafe's own quickstart:
Three differences to a judge, and they are not cosmetic:
The answer space is yours, defined per question. Options and level descriptions are data in the request, not a scale the model improvises.
The probabilities are trained to be calibrated — optimized against outcomes, measured across groups of predictions. Calibration does not promise any single answer is right; it promises the numbers mean something in aggregate. That is exactly what a judge's "8/10" never promises.
48 output tokens cost $0.00. The bill is the state you sent, once.
The Numbers TypeSafe Publishes
All three results below come from TypeSafe's own cookbooks, run on jev-1.12 — the predecessor of the version now on OpenRouter, at the same price. Read them as vendor-published, reproducible-with-your-own-key evidence, not as third-party benchmarks. We flag what they do not show further down.
1. Batching is nearly free — and changes nothing
A regulatory briefing: the GDPR Wikipedia article (~54,000 characters) plus 13 questions (8 Nouls, 2 Choices, 3 Scores). Asked as one call vs. 13 calls, five times each:
Strategy
Calls
Cost
Total time
One call, all 13 questions
1
$0.000497
0.27 s
13 calls, one question each
13
$0.006090
2.71 s
12.2x cheaper, 10.0x faster. The interesting column is the one not shown: across 5 repeats, 11 of 13 answers came back with a standard deviation of exactly 0.0 under both strategies, and the two that moved moved equally under both. Jev ingests the state once and evaluates every question against it independently — no question sees the other twelve.
That is the property an LLM-as-a-judge does not have. Pack 13 rating questions into one judge prompt and the answers start correlating with each other, with their order, and with whatever the model said two lines earlier.
2. Re-ranking: a yes/no question as a ranking score
3,565 court-opinion passages from the CLERC legal dataset. BM25 builds a 30-candidate shortlist per query, then one Noul per query-candidate pair — "could this candidate be from the cited precedent?" — and the shortlist is sorted by the returned probability.
Fast search (BM25)
+ Jev re-rank
Correct passage at rank 1
5%
18%
In top 5
15%
35%
In top 10
38%
62%
1,200 calls, 1,536,002 input tokens, $0.0645 total. That is the whole re-ranking bill for 40 queries.
No scoring rubric had to be invented here: the noul is the score, comparable across pairs because the same question was applied to each. With a judge, the equivalent step means writing a 1-10 relevance rubric and hoping it is applied consistently 1,200 times.
3. Verifier cascade: Jev as the gate, not the worker
The most honest use of the model in the docs, because it keeps the generative model in the loop. Structured data extraction in three rungs:
Extract with a cheap model (gpt-5.4-mini, $0.75/$4.50 per M).
Verify with Jev: one Noul per field, phrased so that bad is true — "is this value absent from the source?", "was it lifted from unrelated text?"
Escalate to gpt-5.5 ($5.00/$30.00 per M) only if any field's P(wrong) crosses the threshold.
The worked example is a scraped NYU events page with no registration date on it. The mini model invents a plausible, schema-valid description. Jev's per-field flags fire at P=0.95 (hallucinated) and P=0.85 (off_target) — while the field that was legitimately empty stays low. The gate escalates, the reasoning model returns an honest empty string.
Note the shape of the win: a blunt "is this extraction good?" judge scored 0.56 on the same record — a shrug. The per-field questions localize the error. Across 100 prompts, sweeping that gate threshold puts the cascade's cost/quality frontier above and left of every single model run alone, including the $0.10-per-extraction reasoning model at ~0.81 quality.
Confidence Is the Part a Judge Doesn't Give You
confidence collapses the probability distribution's shape into one number from 0 to 1. A concentrated distribution means a clear read; a flat one means the model genuinely cannot separate the options. It exists on Choice and Score answers (not on Noul — there the probability is the answer).
That single number is what makes the model usable in code that has consequences:
python
action = response.answers["action"]if action.confidence < 0.5: route_to_human(user_message) # genuinely unsure - don't guesselif action.choice == "check_balance": show_balance(account_id) # low stakes, recoverableelif action.choice == "approve_transfer": if action.confidence > 0.9: confirm_then_execute(account_id) # high stakes, high confidence else: ask_user_to_confirm(account_id) # high stakes, hedge
The threshold is not one number for the whole system — it scales with what the wrong answer costs. Showing the wrong screen and approving the wrong transfer are not the same risk, and your code, not the model, encodes that difference.
Where Jev Loses
TypeSafe publishes a jaggedness page listing its own model's failure modes, which is more than most vendors do. Taken together it draws a hard boundary around where this model belongs:
Failure mode
Reality
Math, counting, numbers
Not a calculator. Counting errors grow with list length. Keep arithmetic in code.
Dates
Read as text, not as ordered quantities. Extract the parts as Choices, compare in code.
Indirection
A property of a property costs accuracy. Point at the relevant state by name.
Literal reading
It answers the question you wrote, not the one you meant. Boundary cases belong in criteria.
Large, noisy state
Context rot is real here: irrelevant material in the state costs accuracy. Filter first.
Adversarial content
State is not treated as hostile by default. Prompt injection in the state can move the answer.
Generation
Not trained for it. Bounded answer space → Choice. Free text → a different model.
And one that deserves its own paragraph, because it will bite anyone migrating a judge threshold across question types: structural invariants do not hold. The same question asked as a Noul and as a yes/no Choice returns numbers that are not comparable — the docs show noul: 0.22 next to Choice P(yes): 0.01 at confidence 0.97 on the same ticket. And a question plus its negation, as two Nouls, summed to 1.19. A Choice is relative (which option wins); a Noul is absolute (can be low for all of them). Tuning a threshold on one and reusing it on the other is a bug.
Add to that: English first (other languages, including CJK, are handled but weaker), text only (no image, audio, video), and rate limits the vendor explicitly says are moving while demand settles.
What the Benchmarks Don't Prove
Being precise about the evidence, because the numbers above are good enough that it matters:
They are vendor-published. Every figure comes from TypeSafe's own cookbooks, on datasets TypeSafe chose. The notebooks ship their API caches, so they are reproducible — but reproducible is not independent.
They ran on jev-1.12, not the jev-1.13 now listed on OpenRouter. Same price, same shape, unverified deltas.
Sample sizes are small. 40 queries in the re-ranking run. 100 prompts in the cascade sweep.
There is no head-to-head against a well-built judge. The cascade compares Jev-as-verifier against no verifier and against whole-output judging — not against gpt-5.5 asked the same narrow per-field yes/no questions. The cost gap would survive that comparison; the accuracy gap is untested.
The SDE cost chart is a historical snapshot that the docs themselves note was not recalculated at the current Jev rate.
Calibration also deserves less magic than the marketing gives it: it is a property of predictions in aggregate. A calibrated 0.8 means roughly 80% of such answers are right. It says nothing about the one in front of you.
Where We Would Actually Put It
Our take after reading the whole doc set: this is not an LLM replacement, it is a cheap, typed decision layer that lets you stop asking a generative model questions it is overqualified and badly shaped for.
Four concrete slots:
Intent routing in front of an agent. One Choice over your handlers, confidence-gated. Cheaper and more consistent than a router prompt, and it fails loudly instead of picking the first plausible tool.
Guardrails on both directions of an LLM app. One request scoring jailbreak probability and potential harm, thresholds in your code deciding pass / review / block.
RAG passage screening. A Noul per retrieved passage before the expensive model reads them — drop the irrelevant, flag the ones carrying injected instructions. Directly attacks the context rot that makes long-context RAG worse, not better.
Verifier in a cascade. Per-field, bad-is-true, max gate. This is where the published cost curve is strongest and where the failure mode is safe: a false alarm costs one reasoning-model call, not a wrong answer.
What we would not do: hand it arithmetic, dates, counting, or anything where the answer space isn't bounded before the call. The docs tell you that themselves — which, in a launch week, is the most trustworthy thing on the page.
Takeaway
LLM-as-a-judge was always a workaround. We used a text generator for classification because a text generator was the only thing that understood the text. Jev's bet is that this specific job — bounded answer space, calibrated probability, no prose — deserves its own model class, with free output tokens as the natural consequence.
The published evidence supports the cost and consistency claims strongly and the accuracy claims narrowly. At $0.042 per million input tokens, the correct response is not to believe the benchmarks. It is to re-run them on your own data, where a real evaluation costs less than a coffee.
