Jev calibration, abstention and confidence routing

Compare probability quality, missing-answer rejection, threshold transfer and cascade cost using six versioned studies and explicit evaluation controls.

Short answer: Better-calibrated probabilities do not guarantee a reliable abstention gate or a cheaper cascade. Evaluate the whole policy on held-out target data, separate missing-answer populations, and compare confidence routing with random escalation at the same budget.

Six fixed-version abstracts, selected manuscript sections and release documentation were inspected. Findings remain author-reported; no model experiment, prediction replay or proof verification was performed.

Which quantity should improve?

Probability accuracy, interface coherence, accepted-case error and decision cost answer different questions. Probability Contracts uses worlds with known posteriors to separate them: averaging two reports can improve expected Brier score without improving a decision at every deferral cost. Its guarantee compares averaging with random interface selection, not with each interface individually. Report validity and coverage too; an interface that produces no usable probabilities has undefined probability accuracy. Probability Contracts, v2 · Study limits

Why is the correct answer missing?

An artificially removed answer, a retrieval miss and an out-of-scope query are different populations. The candidate-coverage benchmark reports them separately; at ten retrieval candidates, its natural-miss result has only five supporting queries. The arithmetic rejection study adds another distinction: selecting None in a menu differs from verifying each candidate with a Boolean question. Thresholds tuned on separate development problems improve its tested rejection tradeoff, but do not establish a universal threshold. Coverage sources and denominators, v2 · Arithmetic rejection and threshold rules, v1

Will a threshold survive a new task?

A source-task calibration constraint is not a target-task risk guarantee. The coverage study holds calibration budgets equal and finds that transferred thresholds can over-reject covered inputs or stop detecting missing answers. The Laya reproduction separately finds that a better probability metric does not ensure the frozen gate meets its held-out error target. Its labels come from a synthetic teacher, and its initial reproduction is retrospective. Compare source-frozen and target-recalibrated policies on disjoint target data, with label budgets stated. Fixed-budget transfer, v2 · Laya study limitations, v1

Does confidence improve routing?

A cascade should be compared with the cheap model alone, the fallback alone, and random escalation at a matched budget. In the 495-claim RAG study, similar aggregate scores conceal different false-verification rates. One threshold is cross-fitted, while its matched-random control uses in-sample operating points; neither nonsignificance nor that control establishes equivalence. Our proposed stronger comparison freezes the entire routing procedure and random-control budget before testing on new claims. Cross-fitted threshold, v1 · Matched random escalation · Design limitations

What belongs in total cost?

Count the first-stage decision for every item, then all escalated calls, retries and routing overhead. Report cost per attempted input, coverage and end-to-end quality together. Fast Models, Slow Evidence corrected omitted pre-screen cost, confusion between gate accuracy and final quality, and in-sample threshold optimism. The RAG cascade is an offline replay whose price calculation excludes second-hop overhead, retries and queueing; it has no controlled latency result. These studies do not establish current prices or universal production savings. Self-audited agent evaluation, v1 · Replay and cost limits

What do the artifacts support?

The candidate-coverage repository separates recorded-output scoring from source-text reconstruction and fresh inference. sys1-eval documents raw-response reanalysis; the RAG repository requires upstream access to the gold labels. Laya’s ancillary inventory lists predictions, metric code and manifests, but also withheld collection/training source. A listed file or publisher checksum is not an independently reproduced result. We inspected the documentation and inventories, without running these procedures or auditing their prediction records. Candidate-coverage code · Agent evaluation code · RAG verification code · Laya ancillary inventory