The System One: Story So Far

Oct 5, 2026

TypeSafe shipped Jev on September 15, 2026, with early access the same day, a $40M seed round led by DCVC, and a claim that they had built the first “System One” decision model (InfoQ covered the release). The pitch was that Jev returns typed judgments with calibrated probabilities instead of generating text, at 40 to 200 times the speed of a frontier LLM and a fraction of the cost.

The system we have built is based largely on locally hosted, open-weight models, so finding gains in performance and efficiency is critical to the system. The Jev announcement started our investigation into System One judgment models. Jev is inexpensive, but not free, which forced a decision: use the closed model, find an open-weight alternative, or build the thing ourselves.

First, we looked for open-weight alternatives. In mid-September the open projects were days old, and most of them used one trick: take an ordinary model unchanged, letter the options, run it once, and read the probability it puts on each letter, then rescale those probabilities so they mean what they say. SemIf does exactly that; others, like von, train a small dedicated model instead. On the community’s benchmark at the time, the trained versions were no better than the trick, and our own llama-server and Ollama already gave us those probabilities, so we built on the trick ourselves, using Gemma, with a resident, always-loaded model behind a judge primitive. The first probe ran at 96 ms, quick, but what about accuracy?

Our initial use-cases are a chat lane router and a content safety check. The first test on the chat lane router was to decide whether a chat prompt would need tool calls or could be answered directly by the model. Labelling a week of real prompts showed that 42% of the prompts it sent down the fast, no-tools lane actually needed tools. We tuned the question (prompt) and got this to 3%, but measured on the same prompts we had tuned it on. Running in shadow mode on the next week’s prompts, it got 22% wrong according to a frontier model marking them, or 31% once we (humans) reviewed those labels. The judge had always seen the start of the previous response; switching it to the end of the previous response, where offers like “want me to save this?” sit, brought that down to 17% on our labels. Adding a simple recency rule (only an explicit “the latest on…” goes to tools) got both weeks to 3%, again on prompts it had already seen. Week 3 is running both changes in shadow, and is the first real test.

Our first clear win for a trained System One model came from the content safety check, which we call the Sentinel. It screens tool output such as emails and web pages for prompt injection before our system reads it. It had been running Llama 3.1 8B locally, asking for a JSON verdict, batching short items four at a time, and it had always been slow. It also started blocking legitimate emails we needed, including messages from a school. We built an 893-item test set: 692 real items the Sentinel had actually scanned, each labelled by reading it, including 47 school emails and 279 real hard negatives, plus 60 realistic synthetic injections, 41 synthetic hard negatives, and 100 samples from the public deepset/prompt-injections dataset. Running one item per call on our dev machine, nimble, Bespoke Labs’ 9B decision model run locally through Ollama’s System One API, blocked 0 of 692 real benign items, no school emails, 0 of 320 hard negatives, and caught 58 of 60 realistic injections at about 1.0 second per item. The old batched setup blocked 10.5% of real benign items and 4.8% of school emails, caught 78% of realistic injections, and took 15.6 seconds per item. The same Llama judging one item at a time blocked nothing benign but caught only 72% of realistic injections. Purpose-built injection classifiers did not fit our traffic. ProtectAI’s deberta classifier flagged about three quarters of the school emails, reading “please sign and return” as an attack, and Meta’s Prompt Guard 2 22M was quiet on mail but caught only 4 of 60 realistic injections. The fast-classifier-in-front-of-a-strong-judge design is dead for us. Batching caused the false blocks, and a System One model fixed both accuracy and speed while keeping personal email on our own hardware. The decision on October 4, 2026 was to move the Sentinel to nimble, one item per call, with a cloud fallback, and Ollama on the production box was upgraded the same evening. The caveats are plain: we labelled the data ourselves, most positives are synthetic, and 60 realistic injections is a small n.

We also turned a System One model on this post. Before publishing, we asked nimble whether each linked page actually says what our sentence claims it says, and tested that on 26 links from our own drafts that we had labelled by hand. Asked in one go, it was good at spotting a link to the wrong site, but it passed every claim pinned on the right source that the source doesn’t make, including one the source directly contradicts. What worked was splitting the job: a generative model pulls out the single claim the sentence credits to the link, and nimble answers two yes/no questions in one call, is this the source the link names, and does the page state this claim. That caught 12 of 13 bad links and passed 11 of 13 good ones, at a few seconds per link. It flagged one of our own errors before this went out, a funding figure linked to a page that never mentions it, and missed another, a poker statistic that isn’t on the page we cited, sitting inside an otherwise accurate sentence. Twenty-six links is a small test and we picked the threshold on the same set, so this is a first look, not a result.

The path forward is prompt tuning first, then fine-tuning. We are collecting thumbs-up and thumbs-down feedback on chat answers across voice mode in our iOS app and the web interface, labelling in progress now. A thumbs-down is not always “the judgment was wrong”. Part two will show whether fine-tuning on the labelled thumbs data cuts the lane router’s error rate on prompts it hasn’t seen, which decides whether we let it route for real. That is the open question: does the training loop beat the prompt fix, or does the prompt fix turn out to be most of the model? We will know in the coming weeks.