Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

arXiv:2609.32160 · cs.CL, cs.AI, cs.LG · Submitted 2026-09-26 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-09-26

Updated: 2026-09-26

Comments: 25 pages, 8 tables. Corpus: 28 arXiv papers (19-24 September 2026); arXiv versions listed in Appendix A. Ancillary file: per-paper corpus sheet (versions and headline results)

Code: https://github.com/youzizzz1028/Awesome-Jev

License: http://creativecommons.org/licenses/by/4.0/

The gist: Typed decision models (TDMs) return probability distributions over caller-defined options without generating text.

Terminology

Abstract

Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.

Sources

Related papers