Theo's Jev take: incredible classifier, not a reasoning replacement
A 30-minute case that Jev's speed and zero-format-error rate make it a genuinely useful classifier, paired with a pointed pushback on the two ways people are already misusing it: as an LLM judge and as a context-compaction engine.
Jev is a fast, cheap, format-reliable classifier model, genuinely useful for structured decisions at scale, but the same narrowness that makes it fast also makes it a poor substitute for judging, reasoning, or compacting context.
Who This Is For
Read if. Skip if.
READ IF YOU ARE…
You build agentic tools or pipelines and want a cheaper, faster way to handle classification, ranking, or routing steps than calling a full reasoning model.
You're evaluating structured-output tooling (BAML, function calling, JSON mode) and want to understand where a purpose-built classifier model fits versus forcing a general LLM into a schema.
You want a grounded read on a hyped model launch instead of the launch-day thread, including where the vendor's own benchmark chart is fair and where it isn't.
SKIP IF…
You're looking for a tutorial on how to call Jev's API; this video covers concepts and demos, not integration code.
You want a purely critical takedown; Theo thinks the model is a real, useful release within its lane.
TL;DR
The full version, fast.
TypeSafe AI's Jev is a 'System 1' model: it doesn't generate text, it takes structured data and returns a guaranteed-shape JSON decision in 70-500ms for a fraction of a cent, with a claimed zero percent structured-output error rate versus double-digit error rates on general models like Claude Haiku. Theo shows it winning benchmarks on cost and speed, demos it on checkers and his own chat history, then draws a hard line: it's excellent for classification, ranking, and routing (a 'smart if statement'), but two trends already forming around it, using it as an LLM judge and using it to compact agent context, misunderstand what it can do, because both jobs require reasoning over information Jev never has access to.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
01 · Cold open: Jev is a classifier, not a chatbot
Theo frames the whole video up front: Jev isn't a better chat model, it's a data-classification model from TypeSafe AI, so judging it like the next LLM misses the point entirely.
03:00 – 04:53
02 · Sponsor break: Depot
A read for CI/build platform Depot, pitched as the kind of infrastructure speedup that pairs naturally with fast, agent-driven workflows.
04:53 – 06:40
03 · Checkers demo: blazing fast, plays badly
Theo plays checkers against a browser demo running on Jev. It responds to every move almost instantly but loses badly, since it's reacting to board state rather than reasoning ahead.
06:40 – 08:34
04 · Origin story: Diogo, RLHF, and the System 1/2 framing
Founder Diogo Almeida, previously at OpenAI on RLHF and ChatGPT, frames Jev as a 'System 1' model, borrowing Kahneman's fast/slow thinking split to position it as a different class from reasoning models like Astra and Fable.
08:34 – 10:39
05 · BAML and the pre-Jev structured-output landscape
Theo walks through BAML's text-sentiment example (a typed Verdict function) to show how developers were already forcing LLMs into typed shapes, and where that patch-after-the-fact approach still breaks.
10:39 – 12:16
06 · Bookmarked tabs: the announcement post and his own T3-chats experiment
A cutaway to Theo's own open browser tabs, including the TypeSafe announcement post and his in-progress project classifying his own T3 Chat history with Jev.
12:16 – 15:03
07 · Does Jev understand color? A Twitter demo
A shared post shows Jev classifying a 'tomato on a vine' description into a color split almost instantly, offered as evidence the model is fast enough to drive a live UI.
15:03 – 17:20
08 · The Pareto frontier chart: cheaper than everything, beaten only by the top reasoning models
TypeSafe's own benchmark plots accuracy against cost across four workflows. Jev sits far to the cheap side of the frontier, out-accuracied only by Sonnet- and Opus-class models that cost far more per call.
17:20 – 19:28
09 · Structured output error rates: Jev near zero, Haiku at 45.5%
A bar chart comparing structured-output error rates shows Jev near zero across categories, while several general models, worst of all Claude Haiku 4.5 at 45.5%, regularly fail to honor the requested JSON shape.
19:28 – 22:40
10 · Wikipedia race benchmark: seconds and cents
A head-to-head benchmark times Jev against Claude Sonnet, Claude Haiku, and a GPT-class model on a link-racing task, clocking Jev at a fraction of a second against multi-second, higher-cost responses from the others.
22:40 – 25:43
11 · The Braintrust misfire: using Jev as an LLM judge
Theo reads a Braintrust post suggesting Jev replace an LLM-as-judge for scoring agent responses, and argues this is exactly backwards: judging other models' outputs requires reasoning Jev doesn't have.
25:43 – 28:23
12 · The compaction critique: why Jev shouldn't summarize your context
A tweet proposing Jev for agent context compaction draws Theo's sharpest pushback: compaction is synthesis, not deletion, needs tool-call history Jev never sees, and would break prompt caching by rewriting early context.
28:23 – 30:19
13 · Classifying 1,118 of his own T3 Chat threads
Theo shows a table of classifier categories from his own experiment tagging his coding chat history (repeated instructions, corrections, workflow family), positioning high-volume self-data tagging as the model's actual sweet spot.
30:19 – 30:29
14 · Closing take: use it like an if statement
Theo restates the thesis one more time: if a decision takes under 10 seconds to make once you have the information, Jev is a good fit; if it needs real thought, it isn't.
Atomic Insights
Lines worth screenshotting.
Jev doesn't generate text or write code; it takes structured data in and returns a guaranteed-format JSON decision out, which is why comparing it to a chatbot misses the point.
TypeSafe AI's founder Diogo Almeida previously helped build RLHF and the methods behind ChatGPT at OpenAI, before spending two years building a model focused on classification instead of generation.
Jev is named after Jevons paradox, the idea that making steam power cheaper increased total coal consumption rather than reducing it, since cheaper capability gets used more, not just substituted.
Traditional LLMs take 3 to over 300 seconds on classification tasks Jev completes in 70 to 500 milliseconds, a gap TypeSafe measured at roughly 4200x on one benchmark.
Jev's output tokens are effectively free (about 4 cents per million input tokens versus roughly $10 per million on a model like Fable), because the model is built for cheap decisions, not long generation.
In a structured-output error rate comparison, Jev scored close to zero across categories while Claude Haiku 4.5 had a 45.5% error rate producing the wrong JSON shape.
Jev's context window is only 32k tokens, and it never sees tool-call outputs, reasoning traces, or full message history, which is the core reason Theo argues it can't safely judge or compact anything.
Classifying 32,311 messages across 1,118 of Theo's own coding chat threads with Jev cost 37 cents total.
A confidence-threshold slider on a classification result changes the answer meaningfully: at an 80% confidence bar, 22% of Theo's threads counted as 'expanding scope', but at 90% confidence that dropped to 6.8%.
Cache invalidation punishes editing early context: change or delete something near the start of a long history and everything after that point has to be reprocessed, which undercuts using a classifier to actively rewrite an agent's context.
Reasoning traces behind a model's decisions are frequently not exposed by provider APIs at all, so no tool, fast or slow, can compact reasoning it was never given access to.
A rough self-test for whether a decision fits a 'System 1' model: if you could answer it in under 10 seconds once you have the information, it's a good fit; if it takes longer to think through, it isn't.
Takeaway
Judge Jev by what class of task it's for
System 1 Thinking
Jev is fast, cheap, and structurally reliable for classification, but the same properties that make it great at deciding don't make it good at judging, reasoning, or compacting context.
01Cold open: Jev is a classifier, not a chatbot
A model can be worth building around even if it never generates text: Jev's whole value comes from returning a guaranteed JSON shape, not from writing anything.
Before judging a new model release, check what class of task it targets. A classifier competing on classification benchmarks isn't weaker than a chat model, it's built for a different job.
03Checkers demo: blazing fast, plays badly
Response speed and decision quality are separate axes. A model can answer in milliseconds and still lose badly, because speed only measures reaction time, not judgment.
Testing a model against its own demo, like a live checkers game, shows the gap between fast and smart faster than reading a benchmark chart does.
04Origin story: Diogo, RLHF, and the System 1/2 framing
The System 1 / System 2 split is a useful filter for any AI tool claim: ask whether the task needs instant reaction or real reasoning before picking a model for it.
Naming a product after an economic paradox (Jevons: cheaper power increases total consumption) is a bet that lower cost per task expands how often you use it, not just what it replaces.
05BAML and the pre-Jev structured-output landscape
Forcing an LLM into a typed output format and using a model natively built for typed output solve the same problem at different layers, know which one you actually need.
A tool that repairs malformed JSON after the fact is a patch. A model that can't produce malformed JSON in the first place removes the failure mode instead of catching it.
06Bookmarked tabs: the announcement post and his own T3-chats experiment
Running your own experiment on a new model, even something as small as classifying your own chat history, tells you more about fit than reading the announcement post.
Keeping the announcement post open next to your own test is a simple habit: claims from a vendor's benchmark chart and results from your own data rarely match exactly.
07Does Jev understand color? A Twitter demo
A model with no vision can still answer a visual-sounding question if the input is restructured as text data, worth remembering before assuming a task needs a multimodal model.
Near-instant response time changes what's possible in a live UI: a feature that used to need a loading spinner while an LLM streamed text can update immediately instead.
08The Pareto frontier chart: cheaper than everything, beaten only by the top reasoning models
A cost-vs-accuracy chart only tells you which model wins on that chart's chosen workflows. Confirm the workflows resemble your own before trusting the frontier position.
Being unbeatable on price while trailing the top two models on accuracy is still a strong result, if your task can tolerate that accuracy gap for the cost drop.
09Structured output error rates: Jev near zero, Haiku at 45.5%
A high structured-output error rate on a task you've delegated to automation is a silent failure mode: nothing crashes, the wrong shape just ships downstream.
When comparing models for a structured-output pipeline, error rate on the format itself is a separate metric from how smart the model seems in conversation, and worth benchmarking on its own.
10Wikipedia race benchmark: seconds and cents
Benchmarks measured in fractions of a second and fractions of a cent only matter if your workload runs often enough for either number to add up; check call volume before optimizing for either.
A model's raw speed on a narrow task doesn't transfer to tasks needing multi-step reasoning, so winning a speed benchmark doesn't predict how it does on a harder decision.
11The Braintrust misfire: using Jev as an LLM judge
Using a fast classifier as a judge for other models' outputs saves money in the moment but reintroduces the exact unreliability you were trying to score away, because judging needs reasoning the classifier doesn't have.
A vendor suggesting you replace a reasoning tool with their own product, in a use case it's poorly suited for, is a signal to slow down adoption, not speed it up.
12The compaction critique: why Jev shouldn't summarize your context
Context compaction isn't deletion, it's synthesis of a summary from everything that happened, so a tool that only sees partial history can't do the job correctly no matter how fast it is.
Reasoning traces behind a model's decisions are often not exposed by the API at all, so nothing downstream, fast or slow, can compact what it was never given.
Cache invalidation punishes edits to early context: changing or removing anything near the start of a long history forces everything after it to be reprocessed and repaid for.
13Classifying 1,118 of his own T3 Chat threads
High-volume tagging of your own historical data, chat logs, tickets, emails, is a legitimate use for a cheap fast classifier, and the cost can be trivial at scale.
A classifier can score confidence per category and let you set your own threshold after the fact, more useful than a single fixed label when you're still exploring what the data shows.
14Closing take: use it like an if statement
A simple gut-check for whether a task fits a fast classifier: if you could answer it yourself in under 10 seconds once you have the information, it's a system 1 task.
The most durable way to evaluate a hyped new model is to name exactly which class of task it was built for, then judge it only against alternatives in that same class.
Glossary
Terms worth knowing.
System 1 / System 2
A framing from Daniel Kahneman's Thinking, Fast and Slow. System 1 is fast, intuitive, low-effort judgment; System 2 is slow, deliberate reasoning. TypeSafe uses it to argue Jev handles System 1 tasks while reasoning models handle System 2 tasks.
RLHF
Reinforcement Learning from Human Feedback, a training method used to make language models follow instructions and behave well in conversation; one of the techniques Jev's founder helped develop at OpenAI.
Jevons paradox
The economic observation that making a resource more efficient to use (like coal-powered steam engines) can increase total consumption of it rather than decrease it, because falling cost expands demand.
BAML
A schema-definition language and runtime that sits between application code and an LLM, defining typed input/output shapes and repairing malformed model output so calling an LLM behaves more like calling a typed function.
Structured output
A model response constrained to a specific, predefined format (usually JSON matching a schema) rather than freeform text, so downstream code can rely on its shape.
Context compaction
The process of synthesizing a shorter summary of an AI agent's accumulated conversation and tool-call history so it fits back within a model's context window, without simply deleting lines from it.
Cache invalidation (prompt caching)
The way cached token prefixes are invalidated once any earlier part of that prefix changes, forcing everything after the edit point to be reprocessed and re-billed at full cost.
01:40productOpenRouter / Vercel AI Gateway (early access points for Jev)
Quotables
Lines you could clip.
00:30
“This isn't our usual new LLM that's slightly better at code... This model is for data processing.”
sets up the entire video's thesis in one line→ TikTok hook↗ Tweet quote
07:30
“Think of Jev as a frontier intelligence function call. Unstructured state in, typed probabilistic decisions out.”
tight, quotable definition of the whole product category→ newsletter pull-quote↗ Tweet quote
17:30
“It's a model that works like a switch statement. It is roughly as intelligent as a switch statement.”
blunt, memorable framing of the model's limits→ IG reel cold open↗ Tweet quote
23:00
“It makes me not trust them... they suggest you should use Jev as a judge scorer... to decide between outputs from LLMs.”
the sharpest criticism moment in the video, names a real company→ TikTok hook↗ Tweet quote
30:25
“If the task requires thinking, this model is not right. If the task requires classifying, organizing, ranking, real quick decision making, this model is incredible.”
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
17px
metaphoranalogy
Oh boy, it's new model time. This one's very different though. This isn't our usual new LLM that's slightly better at code, and it's not going to be the type of thing that the model counter guy is going to show up and say, look, best new model.
If anything, this might take the counter guy out of his job because this model is for data processing. It's by a company called TypeSafe AI, and it's called Jev. The point of this model isn't to generate text, write code, or do all the things that we expect models to do today.
It is classification. This model is the best model ever made to take data and organize it and classify it. It returns JSON perfectly in a type safe format.
When you give it a format, it does it and it does it well. That means this model isn't going to replace things like Fable or Astra. Well, hopefully not.
If you are using Fable and Astra for the stuff you can do with this, I have a lot of questions for you. All that said, this model is incredibly capable, and that's why it's blown up on Twitter. I've seen demos of everything from lightning fast computer use, even in like the iOS simulator, to the model playing Minecraft at lightning speed, to compaction that takes under a second instead of multiple minutes.
I can't wait to show you all the best things you can do with this model. But first, we got to classify something. The next section, which is the sponsor break.
If you're not a developer, you can skip this ad. But if you are, you should listen close. Have you ever sent a prompt to Cloud Code, Codex, Cursor, or some other tool and been surprised that it took hours when you expected it to take minutes?
Did you check to see why? Because I would bet, good money, there's a very good chance that a handful of things happened. Maybe it took too long to download the Docker image.
Maybe it spun up and had some error when it did. Maybe the CI that it was trying to run took forever. Maybe it pushed it up to GitHub and the CI run on there was going to take hours long.
I cannot tell you how many times I've run into this. Wouldn't it be nice if your agents could do all of those things way faster and more reliably? And when those things failed, you got insight into it.
Imagine taking all of these eight minute builds and knocking them down to 20 seconds or building and testing three React apps and seven Go binaries in 30 seconds or making your cross platform builds 10 to 20 times faster. Hopefully you get the idea now. because today's sponsor is Depot, and they are trying their damnedest to make AI more capable, not by giving it things it can't do already, but by making all of the things it does do more reliable and faster.
Whether you're looking for a faster and cheaper alternative to GitHub Actions, or just a nice place to let your agents run, or a cache for all the Docker images you and your team are using, Depot provides all of this and more. Their CI platform is fully compatible with GitHub Actions, which means you can swap over trivially, but if you don't want to deal with GitHub's downtime, you can move fully over to Depot's platform, still compatible.
with actions, but without having to worry about GitHub's reliability. You can also transform those workflows to let them run in parallel, which ends up being comically faster than the alternatives on GitHub.
Their CLI is incredible too. You and your agents will understand it immediately. It makes it trivial to migrate your CI, run it without pushing up code, diagnose things when they go wrong, find secrets and clone them across different places, and more.
You can even SSH into a session while it is running to figure out what's going wrong in real time. And when I say you, obviously, I mean your agents. GitHub actions were poorly assembled for humans over 10 years ago.
Depot was crafted carefully for agents today. Figure out what that means at soydev .link slash depot. quick breakdown on how this all happened because it's actually been pretty crazy to watch it feels like it just poof appeared and took over my entire twitter feed like half of what i've been seeing is all of the stuff about jev it was created by diogo who used to work at openai he apparently helped co -invent chat gpt as well as rlhf And as great as RLHF is for model behaviors, it is not helping as much with classification, which is the thing he cares about.
If you focus more on the classifying side and less on the general usage of text generation side, what you can get is way, way faster and way, way cheaper. In particular, all output tokens being free. The cost characteristics of this model are almost as insane as the speed.
Both are crazy. So let's read through the announcement. The core concept here is System 1 models.
This is an idea that comes from Thinking Fast and Slow by Daniel Kahneman. System 1 is the fast and intuitive judgment of your brain, the part of your brain that can do things without effort, that looks at somebody and is like, yeah, that shirt's blue. System 2 is slow and deliberate reasoning.
In TypeSafe's framing, Jeff handles the first kind of task while reasoning models still handle the second. They named it Jev after Jevon's paradox. I actually think it's pretty cute.
The idea is that when the steam engine gets more efficient, it actually increased coal consumption, not decreased, because now that power is cheaper, we can use it for more things. That is really the right way to think of this model. It's not unlocking new capabilities that weren't possible before.
It's making them fast and cheap enough that they're way more viable than they've been. But what are those cases? What is this valuable for?
Pretty much anything that needs to be classified. Think organizing videos by their topic in my YouTube channel, figuring out if an email is important or not, and what category it should be put under. Things like safety and moderation, identifying what messages are safe and which ones aren't, and what makes them unsafe.
There's so many things you can use it for, and when it's as insanely fast as this model is, the results are kind of crazy. We'll go to the official announcement in a sec, but first I just want to show you how insanely fast the model is with an example of something it can do. I'm not saying you can do it well.
You'll actually see the type 1, type 2 distinction here. But this is a game of checkers that I built to use Jev. Jev takes the whole board as an input, as a state, not an image because it doesn't have vision.
It takes the board state as data, and then it chooses what to move based on the output it gives. So here it might say it wants to move B6 to C5. I go first, so I'll make a move.
Watch how fast it responds. I'm clicking now. Yeah.
It's practically instant.
There is a catch though. It's not very smart. I was able to barely pay attention while playing and crush this model.
Yeah, it's just, it's not good. I was barely paying attention and I'm going to crush it here. The point I'm trying to make is that it's super fast, but it's not using the reasoning part of its brain, so to speak.
It is just processing the data. Let's take a look at what Diogo had to say. Models have been superhuman at chat for years.
So where's all the automation? This has been my driving question for the last four years. At OpenAI, I helped build the methods that make language models useful at following instructions and talking with people.
The work ended up as the research behind ChatsGPT. At the time, I thought maybe chat models would lead to AGI. But despite the hype, it became obvious to me that there was something really big missing.
After two years in stealth, countless technical challenges, and research breakthroughs, he's beyond excited to announce that today, TypeSafe AI is releasing its first System 1 model, a new class of frontier model built to make fast, structured decisions that software can use directly. This is one of the biggest, important pieces to understand.
The point of this model is to integrate it into your code. This model isn't even useful if a human triggers it. The model is useful if the human triggers some code to run, or something else triggers some code to run, and when the code runs, it passes certain data to this model, and then it comes back with JSON.
That's when it's useful, when it's integrated almost like a function in your code base. In order to do this, they had to build a new stack entirely focused on automation, new model architectures, parallel samplers for maximum efficiency, and training methods that they call reinforcement learning for calibrated decisions.
The first public version of this is Jeb, which is now available in early access. It is invite only, but you can get access to it on things like OpenRouter, the Vercel AI Gateway, and a couple other places. While Jev gives up string generation, it's optimized for structural outputs and it can't hallucinate.
This is a key piece. Since it has to give the output in a certain format, it's not going to change or make up the format, which happens a hilarious amount. Think of Jev as a frontier intelligence function call.
Unstructured state in, typed probabilistic decisions out. That's the key piece. You hand it some bullshit text data and you get back a type safe shape.
There have been other attempts to do this in various different ways, from attempts to train models to act like this, to tools to force other models to behave similarly to. One that I really liked once it clicked for me is called BAML. The point of BAML was to make a language that interfaces between agents and real code so that you can define something in a syntax that, of course, agents understand, but also to give a specific format and instruction set to the model to get it to output a certain shape.
They frame it kind of like TypeScript, but as the interface from the TypeScript to the LLM. For example, here is a text sentiment classifier that they have on their site. You give it a label type, which positive, negative, or neutral are valid for.
Verdict, which has label and confidence, which is a float. And you define the function classify. It takes in text.
It outputs a verdict. You use OpenAI. GPT -5 here is the client.
You hand it the prompt. You use their special formatting things. And then you can call this in your TypeScript code in order to get a formatted correctly output.
BAML is basically a made -up language, but what makes it cool is how well it interfaces with other languages and how it fixes all the things that can go wrong. For example, I was using this a bunch with GPT -OSS120B to find comments that mentioned sponsors and things. And when I did that, I had a really good experience with it.
It all just worked. And when I tried to move off BAML for another thing I was doing, I learned that apparently GPT -OSS120B doesn't format JSON correctly half the time. One of the many things they do with BAML is in their runtime, they fix the malformatting in the JSON outputs to make sure it is guaranteed to match the format it puts on the tin.
When you say this is the response format and you call it through BAML, that's the format you get. Sometimes that takes time, though, because it has to reformat it, fix it, change it all with code that is running. But and even some high end models can sometimes screw these things up.
Jev literally can't. If you give it a JSON format, you get back that JSON format. It doesn't handle things like a message history while it's input.
It's really looking for structured program state, like data it can use to make decisions. I already see people getting confused, so I want to be really, really clear here. The point of things like BAML, like Jev, and like all structured outputs is because LLMs are not deterministic.
If I have code that formats someone's name, it will always do the same thing when I call it with a given name. If I have an LLM that I ask to format the name, it might do something else.
If I have a function that returns a user object that has a name colon string and an age colon number, and I write code to get that, it will always be that shape. If I call an LLM, it might screw up the formatting of the name. It might use a float for age instead of an int.
There's a lot of different things that can go wrong. The point of Jev is that you can call it as reliably as you call code. And it's really fast, kind of like code.
It also can do this for a bunch of inputs at once in parallel, which is super, super cool. And the token cost is insanely low. It's about four cents per million tokens in compared to $10 per million on a model like Fable.
I see more confusion in chat already. So is it deterministic? The format of the output is yes.
There are two ways things can be deterministic, the shape and the content. The content isn't deterministic, but it should be quite reliable. The actual format is what is deterministic.
It will always give you the format that you define. And on the topic of cost, the tokens out are free because it's so cheap. They frame it as too cheap to meter.
Meanwhile, with LLMs, the output tokens are so expensive. It's often the majority cost when you're doing stuff like this. Traditional LLMs can take three to over 300 seconds to do the type of classification work that this model can do in 70 to 500 milliseconds.
That is actually 4200x faster. There's no doubt on that claim. That is real and true and based.
Even if prompted for a confidence estimate, models tend to be overconfident and inconsistent. The model can do a task 95 % of the time, but doesn't say when it's in the 5%. You can't automate the task.
Meanwhile, Jev will always communicate confidence and uncertainty with every output. So you can calibrate around the accuracy numbers you get, and the answers also tend to be more consistent. I like the framing here of one of the good use cases being a smart if statement.
I know a lot of y 'all don't code anymore, sadly, I get it. But if statements were a great way of thinking of logic, and that's what this model is for as a thing you put between states to decide what path you go down, or to map reduce over a giant pile of data in order to get insights out of it. and this is one of the coolest things you can do with it real -time applications not that like it'll build the app but as a tool in the app because it responds in just under 500 milliseconds worst case you can do something silly like a search with it or the demo i just showed with chess or checkers This demo for Matt is really cool as well, where you can define a term and ask it to generate a color palette effectively for it.
And it shifts that bar at the bottom. And it's practically real time because the model is so quick to respond. The final use case they have in their example list here is verifying everything.
Score, judge, verify, guardrail, and detect jailbreaks of LLM prompts, reasoning traces, and outputs. Stuff like that. I want to make a quick point, though, because one of the questions I've seen the most by far is something along the lines of, wait, so if I use four different LLMs to generate an output, I could use this to judge it.
You do understand how expensive it would be to generate all four of those and how silly it would be to have a model that's only using one side of its brain judge that. The benefit of other LLMs is that they can reason. They can think through the decision.
They can. walk through your code base using agentic tools in order to figure out what changes to make. They can grow their own context over time and prompt themselves with sub -agents.
They can do all of these different things. It makes no sense at all to give a model that takes a bit of text input and immediately responds with JSON three different implementations of something by three different LLMs. It just doesn't know enough to make a good decision there.
I want to beat into your guys' heads how this works, and I'm sorry for those who get it because it's going to be tedious, but I'm increasingly tired of comment sections that fundamentally don't understand what's being talked about. This is a model that works like a switch statement. It is roughly as intelligent as a switch statement.
It's for classifying things. It doesn't know how to look through a code base to make a good decision. It doesn't have the intelligence to distinguish between the outputs of language models.
And most importantly, the context window is tiny. It's only 32k tokens. That doesn't mean it's not cool.
And it is cool as hell. The speed and the cost is insane. They do a comparison here against an LLM for responding to a query.
Typesafe race, ask LLM versus Typesafe. He got back the response immediately. It was given 27 questions about a bunch of data, and it responded to...
all of them with the right format with numbers ranking them and the type as well as confidence on its decisions for these things it took 0 .114 seconds and it cost an amount that rounds to zero meanwhile terra once it finally finished took nine seconds and cost 1 .3 cents that is a comical gap here 170x cheaper and 70 something times faster respect to them for keeping the cost per workflow in their chart in log they could have made this linear and it would have looked Hilarious.
The only models they measured that had better classification in their demo workflows than Jev were Sol and Opus 5. They didn't test Fable or Astra, but those are way too expensive. You shouldn't even look at them for this.
But it is outranking DSV4 Flash in various tests for ranking things. And it owns the Pareto Frontier with almost two orders of magnitude. Again, because it's very specifically focused on this one thing.
They didn't publish the bench they used here, but they gave some examples. There's an alert. The next step is a question.
Is this unauthorized activity with options for what it can choose? Once that happens, we determine different flows it goes through. And if we have decided that this is an incident, then we ask what state is the incident in?
And it has 11 readings for it. It processes all the data. We then ask what action it thinks we should take.
And then we rank the results after. They are surprisingly transparent with a lot of things. Like they love to say all the things the model isn't good at.
This is where the numbers on their homepage come from, the 193 .6x faster and 444 .6x cheaper. But they say that they expect this is the higher end of real world gains. The context of these workflows were not deliberately chosen nor constructed to make our model look good.
Cool. They're using the average of GPT -6 Astra and Fable 5 .1 as the reference answer. That's why they don't appear in the list and also why the data might not be perfect since those models might be wrong as well.
The LLMs use their System 1 LLM wrapper, which constrains LLMs to output structured decisions compatible with their API. We found it to be the most accurate way to get decisions from LLMs, but this tends to be slower and more expensive than giving decisions without probabilities. I find that they love to tie the idea of hallucination and type safety, type safety being you have a contract for what it should output and it follows the contract.
That's why they're type safe AI, because the shape of the data will always be honored. Hallucination goes a lot further than hallucinating fields in a return type, but In this particular case, it can be pretty brutal if your LM hallucinates that it can change the fields when it can't.
That does suck for type safety. The way they frame it here is that having a hallucinated tool call can be inconvenient for an agent, but it's an absolute deal breaker if it's part of a system with latency guarantees or if it's buried several layers deep in a dependency chain. Yep, that part I absolutely agree with.
No matter how smart an existing model is, it can still hallucinate and have type errors. They show that here. where, crazy enough, Astra actually has more errors with structured tool outputs than Sol, Terra, and Luna.
It actually has more than Terra and Luna combined, which is kind of crazy. But then you start to look at models from a certain Anthropic, where Haiku had a 45 .5 % error rate. That model is so bad, and it needs to stop being used for anything.
It's about a year old now. It might be over that. Anthropic is treating Haiku as dead.
We should do them the favor of doing the same. but of course the number that matters here is jev with a zero percent this is for structured output errors when you give it a json format and tell it to honor it haiku cannot do it and we switch over to tool call error rates funny enough and throwing models start doing a lot better and openai models start to have more problems but of course jev still is at zero it honors the format it's given They made a demo of it playing Doom and the engineer who made it was concerned about cost, but they figured out even though it's running 10 times a second, the cost would still be under $7 an hour because of how efficient the model is and how cheap it is to run.
Remember, it doesn't have vision. So this is all from game state data that it's being given. The application state is handed to the model and it's given the JSON format on what it should decide to do next and it decides.
You will see some quirks with that though. Watch how often it like... just like rotates left and right wildly that's because every single frame it's told make a decision so it doesn't necessarily know what decision it made before it doesn't know it turned left so keep turning left so it's like okay looking here i'll turn left okay new state turn right on every single frame effectively its brain is being wiped and it's making a new decision you get the idea though and i saw this with the chess as well it's not meant for long running jobs It's meant for making a quick decision on the fly or being part of some other long running jobs work.
They call out that this demo is again on structured state as a data structure with text, not on images yet. This is particularly exciting. This model having image support will be super, super useful.
The model is also insane at wiki racing because it makes decisions so quick. So when it has the contents of an HTML page and it knows where the links are on it, it can choose which one to click and continue going much faster.
Yeah. It is very, very fast. Terra even had a hallucination during its run.
That's funny. Let's take a look at what people are using the model for. They claim you can't use it for generating text.
But if you give it the ability to respond by choosing which character it thinks makes the most sense, you can kind of get it to do things. Here's one that I think actually makes way more sense from Chris over at Vercel. It's the idea of using JSON render with Jev.
The point of JSON render is to make it easy to create a component and use it for arbitrary JSON. So you can link the outputs of an LLM into your UI in a more... visible and useful way.
But you have to wait for the LM to generate the JSON before you can update the UI. What if the model could do that comically faster?
That's a pretty big difference. Yeah, it's insane. It responds with the whole thing at once because it's not streaming in text.
Everybody's been telling me Ryan's been cooking with it. Let's see what he's done. 1 ,500 of my own emails that I've exported from my email.
And we're going to run this classification model because I got access to it and see how well it runs the classification on it. We're going to do a batch of 100 emails just to start out with eight workers and we'll see how long it takes to do it. So let's start.
Boom, 100 emails done. It doesn't tell me how long it took. Average was 200 milliseconds, 200 milliseconds per.
email p95 was 240 milliseconds and it did 38 per second which is amazing so this is a much better example i already use lms to go through my email it's expensive but it's fine this is a first pass to like get the quick low effort spam things out of your way super useful it's the things that are too cheap to justify running an lm for or the things that are run too often to wait that much time for Once this has image support, an example of something we can do with this is take a bunch of frames from my video and ask it, does this frame have sensitive data or PII on it?
Like, does it have an email address on it? And to tell me where it is some amount so I can go find these and clean up our videos before we release them. That type of thing is so nice.
I think computer use will also be way more compelling with it. once it has the ability to see what's going on but it's already really good at navigating websites because it can just take the html and then decide what to do based on the current page content there's an example of it booking flights in under 7 .1 seconds right yeah that's crazy lm is doing this take so much longer i am scared to even make this video from being real Because if I increase excitement too much around this model, we're going to end up with some really, really dumb things.
For example, this tweet from Braintrust. Their goal is to increase observability for agents. This is a really bad thing for them to tweet.
It makes me not trust them. Because they suggest that you should use Jev as a judge scorer in a tool like Braintrust to decide between outputs from LLMs. They were genuinely suggesting that you replace an LLM that you use for judging for scoring agent responses with Jev.
And the reason is so that you don't need to spend time and resources prompting a general purpose model into an LLM judge. What? Those prompts take fucking 30 seconds to write, not even.
And like, it's, if you just give it the examples from something like Jev, it's going to follow them. This is hilarious. On that note, there was another demo that I actually think is really cool conceptually.
Like it looks. crazy and it helps show what capabilities this model has. But if you actually think it's a good idea to use a non -reasoning classifier model for compaction for your context, then I would plead that you never, ever, ever stray from the defaults in those tools because you just fundamentally don't understand yet.
And don't worry, you're not the only one. I don't think anyone has to. Context compaction is a complex topic and it's gotten more complex over the years.
I will do my best to TLDR why this is a bad idea, but you should just read my longer post if you're curious. First thing, compaction isn't a filter. We're not just going through your history and selectively deleting lines from it.
We are synthesizing a summary based on everything that's happened so far. Jev also doesn't have enough context to even know what it's deciding on. Then we just saw it doesn't have the tool call outputs and results.
It only has the inputs and a little bit of the context from the thread and not that much of it because it's 32k tokens of context. So it doesn't have enough data to make a decision. even separate from the fact that it doesn't have access to the reasoning data at all because the reasoning data is never shared by the labs anymore when you call the clod or codex apis you don't get back reasoning you might get back an encrypted payload that they can then map to the reasoning on their end or you might get a summary if you're lucky but you don't know what the model was thinking when it made a decision so any attempt to compact that is not going to include those decisions this gets even worse when you remember that anthropic is making changes to how history preservation works such that if you your history you lose all of the reasoning traces for that context there's also the fact that models are tuned on the way that they compact they do this in training now models learn how to handle compaction well and they make adjustments to the weights and how this works through the process of training through rl the compaction that the models do is already pretty damn good and changing what history they have before they do it makes even less sense
Another important thing to recognize is that cache writes are often, if not always, for agentic use cases, much more expensive than cache reads end up being. And when you remember how cache invalidation works, you realize that this will probably break the cache quite a bit if you run it actively enough. Because if you have a history like 1, 2, 3, 4, 5, 6, and then you delete number two, everything three onwards has to be rewritten because the history has to be prefixed.
Any changes early mean everything past that point is invalidated. The model's given weird instructions in this implementation too. Things like whatever's not kept is permanently deleted, but the assistant can rerun a tool if needed.
Some tools are destructive. It's not that simple. And you'll also get to a point if it's classifying in such a shallow way where it gets stuck in a loop where it's already removed everything it thinks doesn't matter and it only has left what does.
No, and people have actually tried this and benched it. It doesn't perform well at all. There is one benefit to this style of compaction.
It's so fast that it fits in the attention span of the average Twitter user, so the video is guaranteed to go viral. But you're not like the average Twitter user. You've been watching this video for much longer than the attention span of the average Twitter user, and for that, I appreciate you.
If you haven't hit the sub button, I would appreciate that as well, because a lot of y 'all haven't, and it seems like you want this type of long -form content. You should consider subscribing to signify that. And as a Twitter user, trust me, you don't want to be like us.
Avoid it to the best of your ability. I do think there are ways that a tool like Jev can be useful to things like agentic dev work. Not having it write code or compact by context or change anything that the harness is doing.
The harnesses are pretty dang good now. You should lean into the fact that billions upon billions of dollars are being spent on that and not reinvent the wheel constantly. Don't get me started on the people who think they can compact their history by taking their entire context and then shoving it into a small image.
Insanity. Anyways. A thing you can use this for is going through large amounts of data.
For example, all of your history using models. Here are my 1 ,118 T3 code chats that I did on this particular machine categorized and classified using Jeff. It classified 32 ,311 messages across 1 ,118 threads and doing all of that cost 37 cents.
and it found that nearly half of what i did was bug fixing and pr work specifically reviewing and managing prs was 20 of my threads remember that the results here are classifications so they're usually scored it's not responding with a list of strings on what to tag it's taking all of the values inside of the body there and giving you a threshold on a scale of zero to one for all of them and you can change what you want to count and not count For example, here with the simple web app I made, I am changing what bar we have set for each of the classifications.
So if we have it at an 80 % confidence interval, then 22 % of my threads are expanding scope. But if I bump that up to 90%, it's down to 6 .8 because it was not sure in the rest of those cases, or as sure, so to speak. funny enough when i had astra build this it did note some of the things that it wasn't great at particularly trying to identify which threads could be used for content for me so one of the ideas it had was what if we can figure out which threads theo should save to use in a video and it tried multiple times to revise the prompt to get better threads for that and it still concluded about half of my threads were worth using for content so it's not good at this it's not smart enough to do those types of complex reasoning things, whereas to make a decision that requires thought, it's like one last simple way to think of this.
How many seconds would it take for you to answer the question once you've perceived all of the information? If I show you a picture of a person wearing a shirt, how many seconds does it take for you to say what color it is? If the answer is under 10 seconds, this model is probably good for it.
If it's over 10 seconds, this model is likely less good for it. This is the point I'll end on, and it's similar to the one I started on. I really want to emphasize the system one point because I'm tired of people not getting it.
If the task requires thinking, this model is not right. If the task requires classifying, organizing, ranking, real quick decision making, this model is incredible. It's a system one model.
System two is when you think about a thing. System one is when you react because your brain thought for you. It's almost like the reflex.
It's like when you tap your knee and your leg kicks out, that kind of thing. This model is designed to work like that first part, to be really fast and quick. If you're thinking of this model in terms of how it replaces the other ones you use, you're probably not thinking about it correctly unless you're doing a lot of structured output work.
The value of this model is that it made so many things that weren't really realistic before way cheaper and faster in a way that is actually kind of cool as a programmer. But you shouldn't be viewing this as a tool you use in Codex or Claude or even in your terminal. You should see this like a new library you install or a function that you call.
It is meant to be integrated in the tools that we build, not used as an inference farm to generate code or text or all these other things. Somebody in chat just said the only way this would replace an LM is if you're using them badly. mostly agree, there are a lot of use case for structured output type stuff.
And this is where it is strongest. I think this model is really cool. And I've been enjoying it a lot.
I have a feeling you will too, as long as you go in with the right mindset. Don't use this to replace fable or judge between complex topics. Use this like an if statement, and you'll have a lot of fun with it.
Let me know how y 'all feel. And until next time, peace.
The Hook
The bait, then the rug-pull.
Theo opens by warning this isn't the model-counter's next 'best new model' brag, because Jev was never built to write code or chat back. It's a classifier, and that one distinction is the entire argument of the video.
Frameworks
Named ideas worth stealing.
06:40concept
System 1 / System 2 fit test
System 1: fast, intuitive, reactive decisions
System 2: slow, deliberate reasoning
Kahneman's fast/slow thinking split, used throughout the video as the filter for deciding whether a task suits a classifier model like Jev or needs a full reasoning model.
Steal forquickly deciding whether a task needs a full reasoning LLM or can run on a cheap, fast classifier
30:19concept
The 10-second test
Theo's closing heuristic: if you could answer a question yourself in under 10 seconds once you've perceived all the information, it's a System 1 task and a good fit for a model like Jev. Longer than that, it needs real reasoning.
Steal fora quick gut-check before routing a step in a pipeline to a fast classifier versus a reasoning model
08:34model
BAML's typed AI function pattern
Define a typed input/output shape (like a Label type and a classify function) and let the runtime repair malformed model output against that schema, so calling an LLM behaves like calling a typed function.
Steal forevaluating structured-output tooling options before committing to a model-specific approach
CTA Breakdown
How they asked for the click.
VERBAL ASK
28:10subscribe
“You should consider subscribing to signify that [you want long-form content like this].”
Casual, low-pressure ask placed after 25+ minutes of substantive content, framed as a signal of demand rather than a hard sell.
Add Modern Creator as a preferred source and Google shows you more of our breakdowns in Search, Top Stories, and AI Overviews. It only changes what you see, and you can undo it in your Google settings anytime.
Add to Preferred SourcesOpens your Google source preferences with us pre-loaded. Tick the box and you're done.
Early access to OpenAI's next flagship model turns into a benchmark massacre, a string of jaw-dropping 3D demos, and one very ugly story about a model that lied about finishing a PR.
Theo spends a week testing two rival "skills" repos for AI coding agents, Matt Pocock's 215,000-star collection and Cursor engineer Lauren's PStack, and finds the real value in a handful of specific files, not the whole install.
Theo puts Meta's Claude Code clone, Muse Code powered by Muse Spark 1.2, through benchmarks, a codebase audit, a game rewrite, and a live integration test to see if the price is the whole story.
Theo runs OpenAI's GPT-5.6-Sol through Claude Code instead of Codex and gets visibly better designs and cheaper orchestration — then reads Codex's system prompt on camera to find out why.
Theo spends a day stress-testing Moonshot's 2.8-trillion-parameter open-weight release — and comes away convinced it's frontier-class, cheap enough to matter, and genuinely dangerous once the weights go public on July 27.
Theo spends 36 minutes putting real numbers behind the GPT-5.6 hype — Sol, Terra, and Luna, benchmarked against Claude Fable, one blog chart at a time.
00:34
00:57
01:20
01:42
02:05
02:28
03:14
03:41
04:00
04:22
05:08
05:31
05:58
06:17
07:02
07:25
07:47
08:11
08:57
09:26
09:42
10:05
10:55
11:14
11:37
12:00
12:45
13:08
13:31
14:05
14:17
14:39
15:17
15:42
16:11
16:27
16:57
17:48
17:59
18:25
18:49
19:11
19:49
20:25
20:46
21:08
21:29
21:49
22:12
23:04
23:26
23:48
24:11
24:31
24:57
25:20
26:06
26:28
26:51
27:14
27:37
28:00
28:53
29:08
29:31
29:54
44:12
38:21
44:42
30:48
41:35
36:00