Seven decision models in ten days, and not one independent score

Seven decision models in ten days, and not one independent score

Between 15 and 24 September 2026 our launch board listed seven new models of a kind that barely existed a month ago. None of them writes text. Each takes a piece of input, such as a support ticket, an email or a JSON record, together with questions whose possible answers are fixed in advance, and returns a choice, a score or a probability for each one. Their makers call them "System One" or decision models. We counted what our own records hold about all seven, and the count is short on the one thing a buyer needs most.

Five of the seven can be downloaded and run on your own hardware; two are hosted services only. Four are released under Apache 2.0, one has MIT-licensed code but a weights licence that is still being finalised, one is proprietary, and one states no licence at all. Only four give any latency figure, and each of the four measured it differently. Only two publish a price, and both charge for input alone, at four cents or a little over per million tokens, with output free. Five of our seven product listings carry the red "not yet independently verified" note we add when the only evidence is the maker's own. Every accuracy figure in all seven records comes from the maker's own evaluation, and only one model has been tested hands-on by an outside publication. Our benchmark scores table holds exactly zero scores for any of them.

That last number is not an oversight. We store a benchmark score only when a named independent source gives it. Vendor self-evaluations go into the listing description, marked as the vendor's figures. Ten days after the first of these models appeared, no score for any of the seven has met that bar.

The seven, as our records hold them

In order of the launch dates on our board:

• Jev, from TypeSafe AI (15 September). The model that started the run, released in early access the day TypeSafe came out of stealth. It is hosted only, reached through a waitlist, and our record notes that no context limit, licence or weights release is stated. TypeSafe gives end-to-end response times of 70 to 500 milliseconds and prices input at $0.042 per million tokens, with output tokens free. Its launch post says its published evaluations are generally run from the team's laptops on the US West Coast.
• Laya, from Convai Innovations (18 September). A family of three open-weight checkpoints under Apache 2.0. The English checkpoint is a ModernBERT-large encoder with a small decision head, 421 million parameters in all, with a 512-token budget per question. The multilingual checkpoint uses an mmBERT-base encoder at 322 million parameters. The authors give 39.5 ms for one question on the English checkpoint and 32.8 ms on the multilingual one, both on a Tesla T4 GPU. There is no hosted API and no price.
• decision-machine-1, from CloudRaker (19 September). The model behind the Milliseconds.ai API. It is proprietary and hosted only. CloudRaker says the name covers a product surface rather than one newly trained model: its runner uses Qwen2.5-1.5B-Instruct for constrained choices and GLiNER2.5 for span extraction, and no overall parameter count or context window is published. The pricing page gives $0.04 per million input tokens, with output free and 125 million free input tokens a month on test keys. Our record quotes a vendor figure of about 90 ms of model time for an image decision at medium detail.
• Kev, from Jared Palmer (20 September). A personal open-source project: LoRA adapters with a small pointer head on Qwen3.5 base models at 0.8, 4 and 9 billion parameters, with code, adapters and evaluation data under Apache 2.0. It copies Jev's API, so TypeSafe's own SDK can be pointed at a local Kev server. When we re-read the repository on 27 September it also listed a 27-billion-parameter variant that needs an 80 GB GPU.
• Tev1-4B-experimental, from Together AI (23 September). A low-rank fine-tune of Qwen3.5-4B, published with a post titled "How to train your own Jev for $17" and the full training recipe. It picks one answer from between two and 24 lettered options. Its model card says the licence for the fine-tuned weights is still being finalised.
• GLiNER2.5-Decide, from Fastino (24 September). A small classifier that takes its list of labels at call time and runs on an ordinary CPU. The model card lists it as Apache 2.0, with 340 million parameters on a DeBERTa-v3-large encoder. Fastino's own models page says 355 million. The model card says plainly that it is not a general-purpose model and does not reason.
• Lev, from Interfaze (24 September). A LoRA adapter on Qwen3.5-4B under Apache 2.0 that accepts the same request format as Jev, so code written for TypeSafe's SDK can be pointed at a self-hosted Lev by changing the address. It runs on one GPU, or on a CPU at seconds per call rather than milliseconds.

Three observations fall straight out of that list. First, four of the seven are built on one family of base models from Alibaba's Qwen line. Kev, Tev1 and Lev all sit on Qwen3.5, and decision-machine-1 uses Qwen2.5 for its constrained choices. The other two open models, Laya and GLiNER2.5-Decide, are encoder models of a few hundred million parameters, closer in build to the text classifiers businesses have run for years. Second, the parameter counts run from about 322 million to 9 billion among the models that publish one, and two of the seven publish none. Third, four of the six that followed Jev measure themselves against it or copy its interface. Kev and Lev accept its request format, Tev1 is Together's open reproduction of the idea, published under a title naming Jev, and Laya's card sets its figures beside Jev's.

Whose evidence it is: counting the red notes

Our listings carry a red "not yet independently verified" note when we could find nothing beyond the maker's own material. Five of the seven carry one. Each says something slightly different, and the differences are worth reading.

• decision-machine-1. The only independent date is a Product Hunt launch listing of 21 September, which is normally submitted by the maker. We found no press coverage, and the description of the underlying models is CloudRaker's own.
• Kev. It is an individual developer's project. All accuracy and calibration figures come from the author's own evaluation harness, and the one publication that reported them, RuntimeWire on 21 September, did so without testing them.
• Tev1-4B-experimental. Together's blog, model card and repository are the only sources, apart from one independent tracker. A price of $0.042 per million input tokens appears only in a social media post by Together, not on its pricing page, so we did not record it. The weights licence is unsettled.
• GLiNER2.5-Decide. Fastino's own pages, its model cards and repositories and one tracker are the only sources. The release date is 24 September on Fastino's page, but the first repository was created on 23 September. And the parameter count differs between the model card, the models page and the repository files.
• Lev. Interfaze's own blog, model card and code repository are the only sources, and every benchmark figure is Interfaze's own run. Its model card also says some of the training datasets carry their own terms, including non-commercial licences, which a business should review before commercial use.

The two without a note are not better evidenced on accuracy. They carry no note because the rest of their record is dated and sourced independently. Jev was tested by Every in a hands-on article published the day it launched. Laya's release is dated by its package registry and model repository, and its card is unusually frank about its own failures, as the next section shows. Every accuracy figure attached to Laya, though, is still the authors' own.

So the plain count is this. Of seven models, six have accuracy evidence that rests entirely on the maker. The seventh, Jev, has one outside hands-on test. In it, Jev read 37 documents and answered 21 questions about each in under 0.7 seconds. In a second test it caught six of seven planted defects, where a frontier text model at high effort caught all seven, at a median of 0.35 seconds per passage against 8.83. That is a useful test, but it is one publication's test of two tasks, and it is not a benchmark score.

The makers' own headline figures also sit on different tests, so they cannot be lined up against each other. Jev's "193.6x faster, 444.6x cheaper" comparison scores agreement with the averaged answers of two other vendors' frontier models on workflows TypeSafe's own team wrote. Its post says that set-up may carry bias. Kev-9B reaches 0.852 accuracy on a test set drawn from sources it was not trained on, and trails Jev by 3.5 points on the development set, which the author says is not a controlled comparison. Tev1 scored 880 of 1,000 on its main development set and 300 of 300 on a policy-transfer set, and its card calls these development results. GLiNER2.5-Decide scores 60.2 per cent exact match on Fastino's own "fast-decisions" set. Lev scores 68.9 per cent across 13 S1Bench subsets against 76.1 per cent for Jev, in Interfaze's own run of both. Five tests, five authors, no shared ground.

Why the latency claims cannot be compared

Speed is the main selling point of all seven. A decision model is pitched as the thing you call when a large text model would take seconds and cost too much. Yet the four figures on our records measure four different things.

• Jev: 70 to 500 ms end to end, hosted. This is the time a caller waits for a hosted API, network included. TypeSafe's post says its evaluations are generally run from laptops on the US West Coast, so the figure includes the trip to its servers from there.
• Laya: 39.5 ms and 32.8 ms per question on a Tesla T4. This is inference on a local, older data-centre GPU, with no network in the path. The card gives batched figures too: ten questions take 158.6 ms on the English checkpoint and 72.3 ms on the multilingual one.
• decision-machine-1: about 90 ms of model time for an image at medium detail. This is a vendor figure for model time on an image, which leaves out the network and says nothing about text calls.
• Lev: 414 to 654 ms end to end, against about 340 ms for Jev's hosted API. This is measured from a laptop to Lev running on one H100 GPU. Interfaze's own post also gives a compute time of 69 ms per batched call on the H100 before any network. The same model therefore appears at 69 ms or at over 400 ms depending on where the stopwatch starts.

The other three give no number for us to record. Fastino says GLiNER2.5-Decide runs on a CPU, but gives no time. Kev's repository says that on a Mac, answers take hundreds of milliseconds rather than tens, and our record notes its server handles one request at a time. Together gives no latency for Tev1.

So a buyer comparing "70 ms" with "33 ms" with "414 ms" is comparing a hosted round trip, a bare GPU forward pass and a laptop-to-GPU round trip. Those differ in hardware, in whether the network counts, in batch size and in input length. None of them is wrong. They simply answer different questions, and the only latency that matters is the one measured in your own stack, from your own servers, on your own inputs.

Price has the same problem in a milder form. The two hosted services charge $0.04 and $0.042 per million input tokens, with output free. TypeSafe's own post notes that its pricing cannot yet be shown not to be subsidised. The five open models have no per-call charge, but they are not free: someone pays for the GPU, or accepts the slower CPU path, and runs the server.

What the makers say about their own failures

The most useful material in these records is not the headline accuracy. It is what several makers wrote about where their models go wrong, and it is more candid than usual.

Laya's card states that the English checkpoint collapses on non-Latin scripts while staying confident. On Khmer it scores 0.000 accuracy at 0.952 confidence. A system that only trusts high-confidence answers would therefore accept every one of those wrong answers. The card also says the multilingual checkpoint ships over-confident and advises refitting a temperature on the buyer's own data before trusting its probabilities. Convai ships a router that picks the checkpoint from the script before inference for exactly that reason.

Kev's repository says changing the order of the options can change the answer, and that knowledge questions remain a large gap: it gives 0.74 on MMLU for Kev-9B against 0.90 for Jev. It also says temperature fitting cuts confident wrong answers, those given a probability of 0.9 or more, from 8.7 per cent to 4.0 per cent of new-source questions, and tells users to test a threshold on their own data before relying on it. Training covered short inputs, while serving accepts much longer ones than training covered.

CloudRaker writes that probabilities and well-formed output do not guarantee a correct answer, and that a model can strongly prefer the wrong category. It advises evaluation on the customer's own data, with review of ambiguous or consequential cases. Lev's makers note that because it writes no text it cannot answer with a label outside the list, though it can still choose the wrong one. Fastino says its model does not reason.

Put together, these statements describe one shared property. Fixing the output to a list removes one class of error, the malformed or off-list answer, and leaves the other class, the confident wrong choice, entirely in place. TypeSafe's claim that Jev "cannot make a type error" is true in that narrow sense. It is not a claim that it cannot be wrong.

For context only: discussion of these releases elsewhere has largely turned on whether decision models are a new category or a familiar one, the zero-shot classifier, with a cleaner interface and calibrated probabilities. Our records speak to that directly. Two of the seven are encoder classifiers by construction, three are fine-tunes of one open text model, and what separates them from older tools is mainly the typed API and the calibration claims. Those are the parts a buyer should test.

Before you rely on one: a method

Everything below can be done by a small team in a few days, without access to anyone's internal data. It is the check we would want to see before one of these models makes a decision that matters.

1. Write down the output contract first. List every decision you want the model to make, its type (yes/no, one of N, a rating on a scale, an extracted span) and every allowed value. Check that the model supports that shape: Tev1 handles two to 24 options, and Jev's post gives a limit of 255 choices. Check too what happens when the right answer is "none of these". If your list has no such option, add one, because a model that must pick from a list will pick something.

2. Ask for calibration evidence, not a calibration claim. "Calibrated" should mean that answers given 0.9 confidence are right about nine times in ten. Ask for a reliability curve or an expected calibration error on held-out data, and for the share of wrong answers given high confidence. Kev's repository publishes exactly that figure, and Laya's card shows before-and-after calibration. If a maker offers only the word, treat the probabilities as a ranking, not a measure, until you have fitted them yourself.

3. Read the licence of the weights and of the data. Apache 2.0 on the adapter does not settle everything. Tev1's weights have no final licence yet, so commercial use should wait. Lev's card warns that some of its training data carries non-commercial terms. Kev-27B, by its own repository's account, starts from a post-trained model whose training data the author does not know. A hosted service with no stated licence, like Jev at present, needs its terms of service read instead.

4. Establish who operates it, and what happens if they stop. A hosted model is a supplier relationship. Find out where the data is processed, how long it is kept, and whether it is used for training. CloudRaker, for instance, says images are not stored. Ask what notice you get before a version changes. An open model moves that risk onto you: you own the server, the upgrades and the security patches. One of the seven is a personal project, one is labelled experimental, and one is in early access behind a waitlist. None of that rules them out, but it belongs in the risk register.

5. Test the inputs you did not design for. Laya's Khmer result is the warning. Feed the model text in every language and script your customers use, very short and very long inputs, inputs longer than its training length, empty fields, and text that belongs in no category. Shuffle the option order and see whether answers change. Watch the confidence as well as the accuracy: the dangerous failure is a wrong answer at high confidence, because that is the one your thresholds will let through.

6. Run a small evaluation of your own. Take a few hundred real, past decisions from your own system, with the answers a person made, and hold them back. Run the model over them and record accuracy, the confident-wrong rate at the threshold you plan to use, and latency measured from your own servers. If you can, run the same set through the model you use today. Together's $17 training run is a reminder of what else is now possible: if you have a few thousand labelled decisions, tuning a small model on them may beat any of these out of the box. Repeat the evaluation whenever the model or your data changes.

7. Decide what the model is allowed to do alone. Use the probability to route work, not to hide it. Let high-confidence, low-stakes answers go through automatically, send the rest to a person, and sample a share of the automatic ones for review. Several makers say the same in their own documentation. Where a decision affects a customer's money, access or rights, keep a person in the loop regardless of the confidence score.

What this count does not show

Our count covers seven launch records created between 15 and 24 September 2026 and the product listings attached to them. It says nothing about models of this kind that our board did not list, and it describes what was published, not how well any of the models performs. Absence of an independent score is not evidence of a poor model. It means only that nobody independent has published a score we could source. Several of these models are days old, and outside evaluations may follow; when they do, we will record them and remove the red notes.

Some pages have changed since our records were written, and more will. When we re-read them on 27 September, Kev's repository had added a larger variant and longer serving limits. Interfaze's post gives the Lev adapter as about 200 MB, where our record says about 170 MB. Fastino's figures still disagree with each other. Where a figure here goes beyond our records, it comes from the maker's page as read that day. Where a page could not be read in full, we relied on our listing records.

Disclosure. TrustList's parent company also makes software, sold as Rutba products. None of the seven models is a Rutba product, and no Rutba business made or distributes any of them.

Sources

• TypeSafe AI, launch post for System One models and Jev, 15 September 2026, read 27 September 2026: https://typesafe.ai/blog/introducing-system-one-models-and-jev
• Convai Innovations, Laya model card, published 18 September 2026, read 27 September 2026: https://huggingface.co/convaiinnovations/laya
• CloudRaker, Milliseconds.ai blog post on why it built decision-machine-1, 19 September 2026, read 27 September 2026: https://milliseconds.ai/blog/why-we-built-decision-machine-1/
• Milliseconds.ai pricing page, read 27 September 2026: https://milliseconds.ai/pricing/
• Jared Palmer, Kev repository, release of 20 September 2026, read 27 September 2026: https://github.com/jaredpalmer/kev
• Together AI, "How to train your own Jev for $17", 23 September 2026, read 27 September 2026: https://www.together.ai/blog/how-to-train-your-own-jev
• Together AI, Tev1-4B-experimental model card, read 27 September 2026: https://huggingface.co/togethercomputer/Tev1-4B-experimental
• Fastino, models page (GLiNER2.5-Decide listed 24 September 2026), read 27 September 2026: https://fastino.ai/models
• Fastino, GLiNER2.5-Decide model card, read 27 September 2026: https://huggingface.co/fastino/gliner2.5-decide
• Interfaze, blog post releasing Lev as an open counterpart to Jev, 24 September 2026, read 27 September 2026: https://interfaze.ai/blog/jev-now-open-source-lev
• Interfaze, Lev model card, read 27 September 2026: https://huggingface.co/interfaze-ai/lev
• TrustList listings, as recorded 15 to 26 September 2026: https://trustlist.uk/listing/typesafe-jev, https://trustlist.uk/listing/convai-laya, https://trustlist.uk/listing/cloudraker-milliseconds-decision-machine-1, https://trustlist.uk/listing/jared-palmer-kev, https://trustlist.uk/listing/together-tev1-4b-experimental, https://trustlist.uk/listing/fastino-gliner2-5-decide, https://trustlist.uk/listing/interfaze-lev