Jev guide: typed decisions, probabilities, and evaluation

What is Jev?

Jev is TypeSafe’s System One model. A request supplies state and typed questions; the response contains bounded answers and probabilities that application code can use for classification, routing, or review. TypeSafe introduction

TypeSafe describes its training as reinforcement learning for calibrated decisions, or RLCD. That description is a provider claim, not enough information to reconstruct Jev’s architecture or training procedure. TypeSafe launch article

How does a typed decision differ from an LLM’s structured output?

A typed decision interface specifies the permitted alternatives before inference. Generative LLMs can also use constrained decoding to produce structured outputs. An interface alone does not establish which model is more accurate, calibrated, or efficient. XGrammar paper

A valid answer can still select the wrong option. Keep schema validity separate from semantic correctness, and test ambiguous instructions, adversarial inputs, and option-order changes on your own workload. Jev 1.13 limitations

What do Choice, Score, and Noul return?

Choice selects from caller-defined options and returns the selected option, a probability for every option, and confidence. Use it for a bounded category or action. Choice documentation

Score evaluates ordered, descriptive levels. Its score is the probability-weighted mean of level indices, accompanied by the distribution and confidence; it is not an exact measurement extracted from the input. Score documentation

Noul returns the probability that a yes/no proposition is true. It has no separate confidence field. A value near the middle signals uncertainty, not necessarily a medium degree of the property. Noul documentation

Keep ordinal label usage, a Score mean, returned confidence, and probability calibration separate. In the tested models and datasets, the v2 ordinal-scale study reports concentrated final choices despite broad average candidate probabilities; finer scales increased compression, and accuracy gains did not necessarily restore utilization. Where a final label is selected, compare its frequency with reference-label frequencies; do not assume every level should occur equally often. Treat this as an additional diagnostic, not a calibration test. The findings are author-reported in the abstract, not our reproduction or evidence that Jev’s Score API has the same failure. Ordinal-scale utilization study, v2

Is confidence the probability that an answer is correct?

Not automatically. TypeSafe derives confidence from the returned distribution. For Choice, four options with a highest probability of 0.70 produce confidence 0.60. Score uses a different formula. Calibration concerns observed frequencies across predictions, not a guarantee about one answer. Confidence documentation

How should I evaluate decisions and calibration?

Separate prediction quality from probability quality. Alongside accuracy or a task-appropriate error measure, inspect reliability plots and proper scoring rules such as Brier score and log loss. Evaluate on held-out data; do not tune thresholds on the final test set. Calibration research · Proper scoring rules

For abstention or escalation, report coverage and error among accepted cases, and test whether the fallback actually corrects mistakes. Freeze model versions and prompts; measure end-to-end latency, failures, and costs separately. This is an evaluation plan, not a result reported for Jev. SelectiveNet · Model versions

How should I compare decision models with trained classifiers and LLMs?

The v2 decision-gate benchmark reports that a fine-tuned DeBERTa-v3-large had the highest observed accuracy on all but one labeled benchmark. That result concerns access to task labels. For your comparison, separate settings with and without task-specific training labels; record data exposure, development/test splits, output formats and failures before comparing accuracy, selective coverage or cost. Decision-gate benchmark, v2

LLM-as-Jev v3 studies frozen Qwen readouts, optional KL-anchored fine-tuning and budgeted reasoning. To distinguish these effects, we recommend a frozen comparator with the same backbone, explicit option sets and splits, and separately recorded reasoning budgets. The reported comparisons concern two Qwen backbones and an independent framework; they do not reveal hosted Jev’s training or architecture. LLM-as-Jev, v3

HakemBench v1 states that most Turkish reference labels are AI-derived, not human-verified, and that earlier test results influenced the lab’s own model on flagged tracks. Report label provenance, known prior test exposure and per-track metric components. Translation probes should remain separate from independent language validation. These are proposed comparison checks; this collection has not reproduced the studies. HakemBench, v1

Can I reuse a confidence gate across languages or domains?

Before reusing a confidence gate in a new language or domain, build a target-specific test set with explicit label definitions and annotation provenance; translated or model-generated labels are not independent human validation. Compare the unchanged gate with a threshold tuned only on separate target-domain validation data. Report accepted-case error and coverage per group, including high-confidence errors under independently checked, answer-preserving context additions.

The social-science audit reports task-dependent confidence failures using benchmark labels; JevOut’s human checks cover only sampled successful perturbations. Neither abstract establishes multilingual transfer. Language-specific label checks and threshold-transfer comparisons are our proposed protocol, not reproduced findings or population-wide guarantees. Social-science annotation audit, v2 · JevOut, v2

How should I test decisions over tables and cited evidence?

Separate schema validity, stability under content-preserving changes, and semantic correctness against reference labels. For tables, pair original inputs with alternative layouts or row/column reorderings only when meaning is preserved. Keep values, relationships and arithmetic fixed, then compare choices and probability distributions. A stable answer can still be wrong; low confidence alone does not identify an error. Tabular decision preprint, v1

For citations, keep the claim and arithmetic fixed while swapping same-number passages that support a different financial role. Include equivalent valid evidence as positive controls, and measure false acceptance of mismatches and false rejection of valid support separately. Inspect paired disagreements rather than counting every changed answer as wrong. These constructed cases are diagnostic probes, not estimates of error rates in real traffic. Same-number citation preprint, v1

How should I test tool arguments and option-channel risks?

Separate schema validity, whether cited evidence supports a proposed argument, and whether the argument follows from the request, history and schema defaults. Test correct defaults or inferred values without explicit references, alongside incorrect values paired with visible references. Reference availability is not derivation correctness. If a model writes references from the full history, hide the candidate from it and verify its output; generated references are not ground truth. Tool-argument verification preprint, v1

Define who can alter request text, tool outputs, option names and policy definitions. Test each channel your threat model lets an attacker control, including confidence on incorrect decisions. The cited guardrail experiments concern open-weight typed-decision models, not a demonstrated compromise of hosted Jev. This is a proposed evaluation protocol, not a reproduced result. Option-channel guardrail preprint, v1

What does an attack success rate measure?

Define the outcome and denominator before calling an attack successful. Probability shifts, action changes, attacker-target selections and downstream harm are different outcomes. Same Attack measures pass-after-attack only among malicious cases each judge flagged when clean. Its separate Jev gate-bypass measure requires a less restrictive action with returned confidence for the severity Score ≥0.7; review can qualify, while low-confidence pass need not. Same Attack metric definitions, v1

Compare attacks with matched unchanged requests using the same decision rule. JevAdvBench distinguishes a changed most-likely Score level from a changed expected score. For adaptive search, separate optimization calls from validation: Decision Hijacking freezes candidates, then uses five fresh calls and requires at least three target selections for case-level success. This finite-repeat rule does not guarantee later success. JevAdvBench typed metrics, v1 · Decision Hijacking validation, v1

Decision Hijacking reports 51 target selections in 2,550 calls for one condition, but 11 majority-success cases out of 510: the denominators answer different questions. Same Attack reports 75% raw severity changes on repeated Jev moderation calls but only 1% routed-action changes. These are author-reported examples, not reproduced results or a ranking. Check answer preservation and delivery separately; equal token counts alone do not prove that an edit was removed. Call-level and case-level counts, v1 · Raw versus routed instability, v1 · Delivery inference limits, v1

Are the SDK and independent models released Jev weights?

The official SDK is a client for TypeSafe’s hosted API, not a Jev checkpoint. Independent projects such as Laya publish their own model artifacts. Similar question types or API compatibility do not establish identical architecture, behavior, or accuracy. Official Python SDK · Laya model card

Check each release’s model card and license separately. For example, Bespoke Nimble 9B v3 declares CC BY-NC 4.0, which restricts commercial use. Publicly downloadable artifacts should not all be labeled unrestricted open-source models. Bespoke Nimble v3 model card

Where should I look for research and benchmark evidence?

Start with versioned empirical papers and their available code or data. The broad Jev benchmark study is one entry point; its findings remain author-reported, rather than independently reproduced by this collection. Jev benchmark paper, v1 · Accompanying benchmark code

TypeSafe WorkflowEvals publishes workflows and model-generated reference labels; agreement with those references differs from human-label accuracy. The community Decision Index also contains private tests, so public rankings do not make the entire evaluation independently reproducible. WorkflowEvals · Decision Index methodology snapshot

What can I reproduce from a public research artifact?

Name what you want to reproduce: tables or plots from scored summaries, metrics from released per-example answers, fresh predictions, or a trained checkpoint. Map each output to its required inputs, versions and hashes. Report whether materials were only inspected, commands were executed, or outputs were matched; these are different claims.

Visual Jev’s pinned reproduction guide describes regenerating specified tables and two plots from scored JSON without a GPU. Other figures need omitted per-example predictions and intervention analyses. Fresh predictions require upstream corpora, backbone weights and a GPU; source images are not redistributed. Visual Jev reproduction guide, pinned commit

The observed ordinal-scale artifact documents metric recomputation from released answers. Fresh inference is separate: source text, weights and post-trained checkpoints are omitted, and missing historical data revisions can prevent a reconstruction matching recorded hashes. Its opaque file revision is not an immutable Git commit. Preserve mismatches rather than silently substituting current data. Ordinal-scale study, v2 · Observed anonymous README, opaque file revision

The radiology paper’s v1 availability statement says analysis scripts and aggregate results are being prepared for release. Third-party dataset links do not supply those missing author outputs. Our checks concern documentation and selected file inventories; we have not executed these reproduction routes or independently reproduced their results. Radiology availability statement, v1