Theo's rebuttal to Sentry's David Cramer and XState's David Khourshid: model choice barely matters for a five-minute task, but it decides whether an agent can run for four hours without falling apart.
Posted
2 days ago
Duration
Format
—
Views
176K
2.9K likes
57 · 43
Big Idea
The argument in one line.
Frontier coding models don't win by being smarter at their best, they win by being dumb less often at their worst, and that gap compounds exponentially the longer an agent runs unsupervised.
Who This Is For
Read if. Skip if.
READ IF YOU ARE…
You run coding agents for 30+ minutes at a stretch and want to understand why some sessions fall apart while others don't.
You're deciding whether to downgrade from a frontier model to save money and want to know what you're actually trading away.
You write narrow, closely-supervised prompts today and want a framework for judging when it's safe to widen them.
You want a mental model for comparing AI coding models beyond benchmark scores and cherry-picked demos.
SKIP IF…
Your prompts are already short, narrow, and closely supervised end to end — the argument here mostly won't change your results.
You're looking for a benchmark comparison or numbers on any specific model's accuracy — this is an argument about methodology, not a scorecard.
TL;DR
The full version, fast.
David Cramer, Sentry's founder, tweeted that people using Fable and Astra should switch back to Opus and Sol and would see no difference. Theo calls this wrong for anyone running agents on wide, long-horizon tasks. His argument: model comparisons fixate on the ceiling (the best output a model can produce) when what actually matters for autonomous work is the floor (how rarely it does something stupid). A model that fails 5% of the time in a 10-minute window already has a 95% success rate, but stretch that window to four hours and the failure rate rockets to roughly 70%. Cut the failure rate by just 2 points and that same four-hour run swings from a 70% fail rate to 50%. Small floor gains compound exponentially into much longer safe run times, which is why Theo tracks two things: how long a model runs unsupervised, and how likely the result is correct by the time a human checks back in. His conclusion: Cramer and Khourshid are still prompting for short, narrow, closely-watched tasks from an earlier era of models, so they can't see the gap frontier models have opened up on long, wide, unsupervised runs.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
Theo reacts to David Cramer's (@zeeg) tweet claiming Fable/Astra users would see no difference switching back to Opus/Sol, and previews that he can prove it wrong.
01:58 – 03:33
02 · Sponsor: Greptile's T-Rex
Ad break for Greptile's T-Rex, a code-review tool that spins up sandboxes to actually run a PR's changes instead of just reading the diff.
03:33 – 07:17
03 · Separating the Astra complaint from the Fable complaint
Theo concedes that frustration with Astra specifically is fair, since Astra's quality swings from best-ever to embarrassingly bad. He draws a response-quality curve for Fable vs. Astra.
Theo redraws the same wild-swing curve and relabels it Gemini 3.8 Flash instead of Astra, arguing nobody should be using Gemini for this, and that Astra performing that badly even sometimes is the real problem.
10:17 – 11:22
05 · The real comparison: Fable vs. Opus
Theo draws the Fable vs. Opus response-quality curve. For narrow, low-difficulty prompts both clear the bar easily, which is why people think the models are interchangeable.
11:22 – 14:45
06 · Width, not depth, is what separates the models
Theo introduces prompt width (scope, duration, autonomy) as the real variable: as a task gets wider, weaker models hit their floor and frontier models don't. Includes a Twitch poll on whether viewers watch their agent work.
14:45 – 15:26
07 · Learning from failures instead of avoiding them
Theo argues long unsupervised runs surface the failure modes you need to fix in your codebase; adjust the codebase more than the prompt.
15:26 – 17:26
08 · Maria's PR and the two questions that matter
A real PR from teammate Maria that merged without her writing any of it. Theo distills the whole argument into two questions: how long can the model run unsupervised, and how likely is the result to be right by the time a human checks in.
17:26 – 19:29
09 · Token cost vs. engineer cost
Theo argues token spend is cheap next to engineer salaries: a $200/month Claude sub plus a $200/month Codex sub buys thousands of dollars of tokens, and 3x output from one strong engineer beats hiring two more.
19:29 – 22:42
10 · The exponential failure-rate math
Theo works the math live: a 5%-per-10-minutes failure rate compounds to roughly 70% failure at four hours; a 2-point floor improvement (5% to 3%) drops that to 50%, showing how small reliability gains compound into much longer safe run times.
22:42 – 25:22
11 · The prompt-length data: median and P95 over time
Theo pulls his own agent logs: median prompt runtime went from 53 seconds to 2:20, P95 from under 7 minutes to over 16 minutes, with the biggest jump coinciding with Fable and Sol's arrival.
25:22 – 27:09
12 · Closing: still prompting like it's February
Theo argues Cramer and Khourshid are judging today's frontier models with yesterday's short, narrow prompting habits, and signs off.
Atomic Insights
Lines worth screenshotting.
A model that fails 5% of the time in a 10-minute window has roughly a 70% failure rate once that same task runs for an hour, because the failure risk compounds across every window.
Cutting a model's failure rate from 5% to 3% per 10-minute window turns a 70% failure rate at four hours into a 50% failure rate, a 2-point floor improvement producing a 20-point outcome swing.
Judge a model by its floor (how rarely it does something stupid), not its ceiling (its best possible output) — the floor is what determines whether a long autonomous run survives.
Widening a prompt (more scope, more time, more autonomy) exposes floor problems that a narrow, closely-watched prompt never triggers, which is why two people can disagree about the same model.
The real capability upgrade isn't a model solving novel problems, it's a model going from a vague bug screenshot to a merged, tested pull request without losing track of the task for an hour.
If a coding agent reliably hits the same failure in your codebase, that's a signal to fix the codebase, not just the prompt — a new human hire would trip on the same spot.
Watching an agent step by step isn't just error-catching, it's the fastest way to learn why a model goes wrong so you can fix your code or your prompting.
Token cost stops mattering once you compare it to headcount: paying for 3x the output from one engineer's token budget is cheaper than hiring two more engineers who might slow that engineer down.
A model with a wildly inconsistent quality curve (huge highs, ugly lows) is harder to trust than a model with a lower ceiling but a steadier floor, because you can't calibrate how much to rely on it.
Comparing models only on jira-ticket-to-code tasks understates the gap, because that task is narrow enough that almost any current frontier model clears the bar.
Takeaway
Judge a model by its worst moment, not its best one.
WHAT TO LEARN
For long, unsupervised agent work, a model's failure rate compounds exponentially over time, so a small improvement in consistency matters more than any single impressive demo.
A model that fails 5% of the time in a 10-minute window looks fine for a quick task but compounds to roughly a 70% failure rate over four hours, because each window's risk stacks on the last.
A 2-point improvement in per-window reliability (5% to 3% failure) can cut a four-hour failure rate from around 70% to 50%, showing how small consistency gains produce outsized results at scale.
Two models can look identical on a short, narrow, closely-watched task and wildly different once the task widens into more scope and less supervision, so test comparisons at the length you actually intend to run.
Evaluate any coding agent by two questions: how long it can run without your input, and how likely the result is correct by the time you check back in, rather than by its most impressive single output.
If an agent reliably fails at the same point in your codebase, treat that as a codebase problem to fix, not just a prompting problem, since a new human hire would likely trip on the same spot.
Watching an agent work step by step is worth doing occasionally even if you mostly let it run, because it teaches you why failures happen so you can fix the underlying cause.
Weigh token spend against headcount, not against a monthly bill in isolation: paying more for a model that lets one strong engineer ship 3x more can be cheaper than hiring additional engineers.
Glossary
Terms worth knowing.
Floor vs. ceiling (model quality)
The ceiling is a model's best possible output; the floor is its worst. Theo argues autonomous agent work is bottlenecked by the floor, since a single bad decision can derail an unsupervised run, while the ceiling only matters if you're chasing a one-off best result.
Wide vs. deep prompting
A 'wide' prompt covers more scope and runs longer with less supervision (e.g. 'fix this bug, test it, open a PR'). A 'deep' or narrow prompt is short and specific (e.g. 'edit this one file'). The same model can look identical on narrow prompts and wildly different on wide ones.
Fable
The model Theo prefers for long, autonomous coding-agent runs in this video, praised for a high, consistent floor rather than a spectacular ceiling.
Astra
A model Theo describes as capable of things no other model can do at its best, but with unpredictable, sometimes very poor low points — a high ceiling paired with a low, unpredictable floor.
T3 Code
Theo's own coding agent product, referenced throughout as the tool he runs these long unsupervised agent sessions in.
Theo's direct reply to Cramer's tweet, the inciting line of the whole video.→ TikTok hook↗ Tweet quote
14:18
“I don't like Fable because it's way smarter. I like Fable because it's less dumb. And those are different things.”
The one-sentence thesis of the entire argument.→ IG reel cold open↗ Tweet quote
11:40
“I don't care if the model solves novel math problems if it doesn't know what I mean when I say revert.”
Punchy floor-over-ceiling example, easy standalone quote.→ newsletter pull-quote↗ Tweet quote
21:40
“When you make these small cuts to fail rates in given time windows, you exponentially increase the distance that it can run for.”
The math payoff line, works as a caption over the chart.→ TikTok hook↗ Tweet quote
The Script
Word for word.
Read-along
Don't just watch it. Burn it in.
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
17px
analogystory
If you enjoy the videos where everyone calls me a paid shill, you're going to love this one because I kind of have to glaze for a second because I see some comments that are just so dumb that it makes me realize the majority of engineers just aren't using agents right at all. And I mean that when I say it. I'm inspired to make this video because of a post from another engineer that I generally quite respect, but is so...
dumb that I need to talk to you guys about it because I know a lot of y 'all feel the same way and you feel it so deeply that you accuse me of being a shill because of how deep my feelings here go. The take is as follows. From David Kramer, aka Zeeg, the founder of Sentry, everyone using Fable or Astra, which is most of us on subs because no one can afford it, should try switching back to the other high reasoning models like Opus and Sol.
You'll likely realize your tasks don't perform any differently. To which I responded, this is your worst take of all time, which is kind of crazy. Kramer's had some wild ones in the past.
And I genuinely believe you suck at prompting if you believe this. So if what Kramer just said resonates with you, if you actually think or have experienced going back a model gen or down a model tier and not notice a difference in the way you do work, you also suck at prompting. And this isn't like a...
disprovable thing the other way where you can't prove i'm wrong because there is literally no way to do that but i can very easily prove you're wrong which is what i'm really excited to do right after a quick break for today's sponsor i have a challenge for you next time you're reviewing a big pr and you're not sure if it's ready to go or not ask your agent on your machine to pull it down play with it test it and make sure everything works as expected There's a good chance it's going to find things that both you and your agents wouldn't have otherwise, because if you're just looking at the code, you're not going to be able to find too much.
This is a mistake that a lot of the AI code review bots make. They think they can know everything and how it works just by reading the code. And reality is not that simple.
Code breaks in ways that are not clearly visible just from reading code. This is why I think T -Rex by Graptile is so damn cool. These guys realize that agents know a lot more about code when they can actually run it than they can possibly guess by just staring.
at it. And that's why T -Rex uses sandboxes to actually test your changes before leaving review comments. Greptile already has a best -in -class system for understanding the context of changes because they know your whole code base, they monitor it closely, and they've indexed the hell out of it in order to make good insights happen.
But now they can also test the code, which makes it so much more powerful. And this goes so much further than just letting the reviewer run in a sandbox. It's actually kind of the opposite.
It's more that the reviewer orchestrator can spin up and run sandboxes with sub -agents for whatever theories it has about what might be wrong. Some reviews might need no sandboxes.
Some reviews might need 10. The orchestrator will figure out what is needed to verify your changes. This helps you ship with way more confidence and not just because it gives you a thumbs up or thumbs down, but because it will respond with images and videos of the things it tests.
So if you want to make sure that your dropdown disappears correctly or that the signup flow works end to end. A simple approval message should not be enough. A video proving the changes worked is so much more valuable if you're trying to ship with confidence.
We all know coding agents work way better if they have a real computer. Turns out review agents do as well. Get your review agents the support that they need at soydiv .link slash reptile.
Let's just break this down piece by piece because if I do the whole thing at once, people are going to read into the parts I don't focus on and say that's why I'm wrong. So let's start with a thing that I just want to get out there right now. The Astra and Sol portion is very different from the Fable versus Opus and Sol portion here.
Astra can do things no other model in history could. It is incredible, but it can also screw up in ways I haven't seen since I last was using a Gemini model seriously, which was like 2025. So if your feeling here is purely around Astra, because yes, sometimes it does incredible things, but sometimes it does weird or annoying things.
then I see why going back to Sol would feel not even not bad, but in some real cases, somewhat good. And I understand why people will be moving from Astra to Sol. So I'm jumping in that one immediately because I know people will be feeling that and I get it.
I even have a diagram. Let me find it. Here it is.
This diagram was meant to show roughly how I feel about the quality of responses over a large set of responses with Fable and Astra. Astra at its best can do things Fable never could. Astra at its worst makes me question why I'm using AI to code at all, because it can make some real fucking dumb decisions and assumptions.
Generally speaking, though, one of the benefits you get from these frontier models is that the gap between the worst and the best gets closed and the floor goes up. Actually, I think this diagram is a really good starting point for the things I want to try and communicate here. Generally speaking, y 'all focus too much on the high points with models and not enough on the lows.
And I'll be real, both of these assumptions have gotten me in trouble. There were times where I focused too much on what the best looked like for a model, things like Opus 5, and I thought it was really good when it wasn't, and it misbehaved far more often than I had known at the time. And then there's times where I focus on how pathetic the model behaves sometimes and just ignore it outright after that, even though there are real strengths.
I would argue to an extent that my disdain towards Gemini comes from the floor being so low and so consistent, where it's just... It'll read the same file 26 times before making a change because it's a shit model. But sometimes it can name skateboard tricks really well.
So its ceiling is in interesting places, but you got to think about the ceiling and the floor. And we all have a bad habit of thinking too much about the ceiling. I bring this up because the first huge benefit from this new era of models and from frontier stuff, the thing that you don't get as much from open weight models and from other labs other than Anthropic and OpenAI generally, is a higher floor.
And I personally find that raising the floor is way more beneficial than raising the ceiling. I don't care if the model solves novel math problems if it doesn't know what I mean when I say revert. Yes, Astra has actually been confused about what the word revert meant for me before.
I don't care how good an agent is at building a 3D environment in Blender if it can't center an icon in a div, which yes, I have also seen Astra fail to do. Astra has... kind of rotted the conversation that I want to have here, which is why I'm choosing to do it now.
Because Astra has the traits of the best models and some of the traits of the worst, which makes it hard to recommend in the way I want to. So I'm going to do a thing I don't want to do. I'm going to clone this diagram because I'm going to delete Astra from it.
And I'm going to relabel it to what it is. Gemini 3 .8 Flash. This is how I actually feel.
And the fact that Astra can perform as poorly as Flash ever is pathetic. And the engineers involved should feel bad and fix it. And thankfully they do.
I know for a fact they're going to fix it. And this little spike here is why Flash had those couple of good benchmarks that made it look really good when it wasn't. I'll do a more realistic comparison here to get my point across because no one should be using a Gemini thing.
So we're going to talk about Fable versus Opus because it seems like people believe Opus can do the work that they are doing. And that is true a lot of the time if your line for quality is here. If this is the prompts you're sending, if your prompts are things like, here's the ticket, I want you to find the file and make this change and tell me when it's done so that I can test it, then it is unlikely Opus or Fable or even a Gemini model is going to struggle too, too much to do it.
And if you have low tolerance for failure, which I'll admit even I do, when I ask the model to do a thing and it fails to do the thing, it pisses me off. Like it actually angers me. Which means if one of these dips is worse than the others, a bad thing happens.
My bar gets lowered. This is where I'm comfortable prompting. Because if I tried a harder prompt or a prompt that involved more work and the model failed to do it, I now think models can't do that.
So I lower my expectations. The things I prompt for, the ways I prompt, and the most important detail, we'll be talking about this a lot, the width of my prompt, not the depth, not the difficulty of the thing, but the amount of things and the distance from the start to the end. Wider prompting requires higher floors.
And to be clear, I'm not trying to say Zeke's bar is set this low. I am guessing Zeke's bar is set here -ish. where sometimes it slightly disappoints sometimes much more rarely but sometimes it really disappoints but generally things are solid this is also what makes astra so annoying is the random spikes into dumb are so random and so spiky that you don't really know how to gauge what level to operate at this is why when a new frontier model comes out i immediately try to reset instead of just sending the same prompts i sent before i always start with a set that i've tried on other things that failed to see if it can succeed usually when i do those tests i'll see a few things that did better than expected and from there i can start to figure out what new things i can do that i couldn't and again i want to be clear here if your goal is to go from jira ticket to code zeeg is right If I have a well -formatted Jira ticket that lists what files the code is in and what exact behavior exists that shouldn't, I can throw that at Claude and get an answer relatively reliably.
That's not what I'm talking about here. What I'm talking about is I get a DM on my phone from a bug a user had in T3 code. So I screenshot it.
I paste it to my agent in T3 code on my phone and say, fix this, test it, record a video showing it works now. and link me the PR when you're done. Babysit it until all the issues that come up in review are addressed.
That isn't harder than what I said before. That isn't harder than going and editing the code from the Jira ticket. But having the whole end -to -end where the model can go from a vague screenshot of what's wrong to a real functioning solution with a pull request that has a video proving it worked without my intervention at all.
That is the capability that I'm excited about. Is it cool I can demo it making a crazy 3D game? Yeah.
But it's way cooler that I can send it a screenshot of a problem and then go do something else. And in an hour when I check in, it hasn't lost track of what it's doing. Has it burned more tokens?
Yeah. I don't care though, because despite the tokens being expensive, so are my engineers. So is our time.
If Julius can be three times more productive by spending his salary in tokens, That's a no -brainer. Hell, he could do 2x the salary in tokens because people like Julius are rare.
And I would rather Julius ship three times more than risk it hiring two more engineers that will be more likely to get in his way than help him ship faster. But this is the core point I really want to drive home here. If your tasks aren't super, super narrow, and this is, again, how I want to think about this, let's draw vertical lines like this instead.
I find for most devs, and this is like, I'll ask them to show me their prompts and show me their histories. I find most of their prompts look something like this. They are small and concentrated and only require a little bit of how the model operates.
You only are relying on the model for so long. You might even be watching the thread as it goes. You know what?
I'm going to do a poll quick. I want to know, do you watch the agent while it works? I will admit I'm a little disappointed in these results.
I was hoping for almost never to be a clear win. you guys need to stop paying so much attention to your agents i've been saying this for a while now you guys should push your limits in your trust and see where it goes i like this i only watch if the first output is horribly wrong that is a great mindset because watching isn't just trying to keep it from doing wrong again it's also helping you learn why it's going wrong and even better you can ask you can say i expected you to do this but you did this other thing instead what led you there it won't Get it perfectly.
The models rarely know exactly why they did a thing, but they will usually indicate what signals they found and what tools they called and what files they read that led them in the wrong direction. I actually like the way Jamin framed this here. He likes doing overnight autonomous work, not because it gets a ton of code written, but because it will hit a bunch of those fail cases so that he can architect his code base and his systems so that when he's working more in the loop, doing like one or two hour threads instead of five to 10 hour threads.
If you have a speed bump in your code base that agents hit on average once every two hours, then they'll hit it on average two to three times every five hours. And if you do these super long runs, you can hit those failures faster and fix them. Learn from the bad runs.
Make changes based on the bad runs. You should adjust your prompts a little, but you should adjust your code base a lot. If agents are screwing up in your code base, then a new dev would too.
Like if you took a super experienced dev that's never worked in your code base before and they couldn't contribute by the end of the day, that's on you, not the agent. And if you are giving these very specific, precise instructions, hell, if you're, I have a new poll actually. This one's going to hurt me.
This one's going to really hurt me. do you still mention names of files in your prompts often sometimes basically never actually never this one's gonna be very eye -opening for me actually i'm over correcting due to greenfield t3 code we got 12 kprs and 300 000 users if i do anything wrong we get flamed immediately and have also had phenomenal luck using this to maintain other huge code bases too but i really don't believe this like It doesn't work this way in real companies.
No, it absolutely fucking does. I just, I don't believe it. I hope enough bigger companies get this right.
Let's see the results here. Okay, you guys have redeemed yourself. I feel much better now.
The people in the often section I refer to as Atlassian devs. And the ones at the bottom section I refer to as realistic ones. And I would guess that the ones at the top have a much better time with Opus and are confused about why people like things like, you know, fable and aster so much the reason we like these new models isn't because we are pushing them to their absolute limits to make new sciences up we just like that they're stupid less often we like that they can go longer and make fewer mistakes and need less guidance to do the right thing and to go back to my chart here the thing i'm trying to emphasize is you should be striving to have one prompt Use more of the window of what the model is capable of.
If you know where the rough areas are and you can smooth those out with changes to your code base or just not using the model for those things, and you find ways to go more horizontal, maybe instead of investigating the code base and telling the agent which files to touch, you tell it what the problem is and tell it to find the files and change them itself.
Maybe instead of telling it... to let you know when it's done so you can build it on your phone. Tell it to push the build to your phone when it's made the changes.
Maybe tell it to run it in the simulator first, verify it, and then push it to my phone after so I can do one last check if I'm still concerned. Or just tell it to throw a video in the pull request so you know it worked. That's what we do most of the time now with T3 Code.
Here's a PR from Maria. This was a bunch of fixes to provider history issues when people were using the rewind features and various harnesses. Maria wrote, none of this PR.
And it merged pretty quickly after she filed it because it was a good PR and I could open it. I could look at what it changed because it visualized what it changed because the model did all that. And by the time a human is bothered, it is much more likely the thing works.
So these are the two core things I really want to push you guys to think more about. The first is how long can the model run without your input? And the second, similar, but not exactly the same.
And I want to clarify the differences. How likely is it that the thing works by the time a human gets involved again? These two things are not defined by using the models to rebuild 3D worlds.
They're not defined by how well they can port all of Electron to Rust or anything. They're defined by how likely the model screws up. And your goal is to make it less likely that the model screws up in any given window.
So if in your experience, if you let the model go for 30 minutes, it usually hits bugs. It usually runs into problems, usually does stupid shit. Fix that.
Make it not do that. Because I have had runs go for six hours with no intervention that merged 10 minutes after I filed the PR because they did exactly what they were supposed to. And the model had verified it itself before I got pulled in.
But to go back here with the response quality, if your prompts look like this and the necessary bar for you is here. then yes absolutely the difference between fable and opus isn't very big because both massively clear your bar in this window but watch what happens when i move the end point over more and more oh no now my bar isn't being met and all of a sudden if i make the task wider not harder wider which means longer it does more the likelihood we hit one of those edges in something like opus where it does something stupid goes up as time goes up.
And the thing that makes the frontier model special is that they hit those floors less and the floor is raised meaningfully. I don't like Fable because it's way smarter. I like Fable because it's less dumb.
And those are different things. Less dumb and smart are almost opposites. They are the opposite ends of the spectrum.
And if you're thinking of models as their peak capability and not their worst capabilities, you're not talking about them the right way. And as such, it's really hard for me to take anyone seriously when they say Opus will perform just as well for your tasks, because it means that their tasks are really, really short and simple.
And to be very clear, I have nothing against simple. I love using agents for simple stuff. That's what I do most of the time.
But it's long, simple stuff. At every generation bump, the amount of time until the model is 50 % likely to have done something stupid goes down exponentially. And if you don't feel this way, or maybe you've been prompting this way occasionally and you're not happy with the results, then you have a great opportunity to make real improvements here.
Maria had some good comments on this that I want to bring up. She spent three to four days going over her traces and refining skills and whatnot after seeing what the model does and doesn't do right and made all of these adjustments. And now she can just fire a single prompt and get a PR landed instantly.
Yeah, it's great. She credits Potato, not me. Fair.
I get it. I'm trying to push these same things more. What percent of devs have a spend of many hundreds of dollars a month just to play with it?
Most full -time devs can afford a $200 sub to Claude and a $200 sub to Codex. And most of them probably work somewhere that is willing to pay for those as well. So yeah, that gets you eight grand of Claude tokens and 12 grand of tokens from OpenAI.
You got a lot of wiggle room for not a lot of money. And I know that is a lot for people who aren't in... western countries who aren't full -time devs who are younger who are students etc but those aren't the people i'm talking about here if you are using the dumber models because that's what you can afford you should be very careful which ones you use because sometimes the cheaper model ends up more expensive but that's not who i'm talking about here i'm talking about people like kramer who concluded i think he's wrong here because i should stick to youtube videos and don't know anything about engineering Or people like David Kaye, who I love.
He built Xstate, which is one of the best state management libraries in the whole web dev world, saying you don't need AstroSol or Fable for most things, which is true. But my time is more valuable than Fable's. So if I downgrade to a cheaper model, I have to put more time in before it fires and more time in when it stops.
And the more I let the model chew out both sides there and get involved earlier and pull me back in later, the better things are. I also saw somebody in chat pushing back on me saying exponential here. Let's see.
This will take a bit, so I'll have to record an extra later for it. I want you to go through a set of my prompts and the responses from, I don't know, January versus now, maybe February if I don't have enough history. And I want you to figure out from a reasonably randomized set how long my average prompt ran for.
So from when I sent the prompt to when the response stopped generating, how has the amount of time changed from January or February to now? Great. My vibe proxy is quite broken.
You know what? I got some Opus usage. Let's let Opus do something for once.
I saw somebody in chat say they thought it would be 10 to 15 % longer. And I feel like I am insane. I didn't trust models to run for more than 15 minutes just a few months ago.
Even something like GBD 5 .5, which was generally better with agentic stuff, it stopped so often that I found its runs were actually kind of shorter overall. The amount of time these sessions ran for went down with 5 .5, and then with 5 .6, suddenly I could let it run way longer. And now my agent runs average probably 2x longer, but the top 1 % longest ones are at least 10 times longer.
I've had things run for two days straight with no issues, and I could barely get a thing to run for an hour before. Just got the numbers in and I didn't have as much data as I was hoping. I only have my logs since March because I did a computer move and didn't back up my agent history because I didn't care much yet.
And here are the results. Median prompt went from 53 seconds to two minutes and 20 seconds. So median more than doubled in length.
P95 went from a bit under seven minutes to over 16 minutes and 20 seconds. You understand, right? That's from April to now.
And here you can see over time, it's actually really useful. In March, my 5 % longest requests were nine minutes long. Then it went to 11, then 12, and then May to June, this is when we started to get Fable and Sol.
We went from 12 minutes to 22 minutes, nearly doubled month over month, just from the new models. So yes, it is exponential. The rate at which the length your prompts can go for is massively skyrocketing.
Chat's hopping in to agree here. I can confidently say that mine went from five to 15 minutes to one to four hours. Sorry, it was the floor improves 10 to 15%.
I didn't say the floor improved exponentially. I said the impact of it improved exponentially. Here, let's do the math out here.
Let's say you have something that fails 5 % of the time in a 10 minute window. That means that you have a 95 % chance of success in that same window. What happens if you want to run for 30 minutes?
You all know how this math works, right? 0 .95 to the power of three. Going for 10 minutes to 30 minutes changes your failure rate from 5 % to 15%.
Let's say you want to go for an hour. Oh God, now I'm at 73%. Two hours, now you're at a 50 % fail rate roughly.
Four hours, and now you're at a 30 % success. Let's just slightly bump this. Let's say you improve the floor by 2%.
Instead of failing 5 % of the time in 10 minutes, It's now 3 % of the time in 10 minutes. That bumps us here to 9 .7.
Oh, wow. That's kind of crazy. That's only a 50 % fail rate at four hours, but it's only a 2 % difference.
Wasn't it? Oh, yeah. It was a 70 % fail rate with a 5 % every 10 minutes.
Now it's only 50%. So that 2 % change ends up being 20 % at the time scale of four hours. That's a 2 % difference.
Now imagine it's 10 to 15 % like you said it was. Oh man, that's an exponential change in how long you can run. When you make these small cuts to fail rates in given time windows, you exponentially increase the distance that it can run for.
AI Dev Guy said he could run for 48 hours straight back in February, but he'd have to spend several hours building up specs and not even do things that long with more improvised and shorter prep periods. I don't think you really could do this before. even with really good specs, because the coherency the model has over time wasn't great.
And as crazy cool as Ralph loops were, models weren't good enough at compaction or keeping track of what they've done in the past or leaving reminders of what they've tried. And the results ended up being still very, very high failure rates. Those have dropped exponentially.
You can do things like write these specs to help keep it somewhat more on track, but it only helped so much and it didn't. bump these failure rates often enough no matter how much work you put in the result of speccing out a run and letting it go for eight hours isn't t3 code the result of that is cursor two and cursor three where everything broke as soon as you looked at it too closely and now that models are good enough cursor starting to get stable because they don't want to be in the loop they want to run it for four to eight hours even if it's bad and now that four to eight hour runs are way more likely to come out good suddenly cursor functions again obviously Lauren's to credit there to some extent too but yeah meaningful difference so while I deeply respect Zeke and David Kaye I genuinely think both are still prompting like we're in February and the reason why they're doing that is they're not valuing their time properly they feel good putting that extra effort in at the start in the end because as a great dev before the thing that got you to level up the thing that got you from a good contributor to a good leader
was doing more of the prep before the code started and doing more of the vetting after so it's even more uncomfortable to give that up to the agent so the parties i see falling for this are the ones whose work isn't serious enough to realize the power of the models but even more so it's the incredibly talented leaders who have largely left behind coding in their day -to -day because the thing before and after the code writing matters more they struggled to give up the code in the middle but they did They won't give up the things on the other side yet, which is why they don't see the benefit.
They are testing the models against the thing they already stopped doing, and the models have been able to do the thing they stopped doing for six months. They're correct there. But the moment you let the model go a little further in either direction, you'll suddenly start to see the edges a hell of a lot closer.
So as per your request, Zeke, I will stick to making videos, because otherwise your stupid take's gonna go too far, and I need to make sure the next generation of devs who haven't fallen for this shit don't because what you're saying sounds good and we want to believe it i want to believe it i would love to not have to spend more money on my models but i do because my time is more valuable than my posts and i hope you realize the same soon too i hope you enjoyed this video zieg and if anybody else happens to see it maybe you'll like it too let me know how you feel about this one and how wide your prompts have been and if you think i'm crazy for not including file names in my prompts anymore and until next time peace nerds
Frameworks
Named ideas worth stealing.
20:40model
Exponential failure-rate compounding
5% failure per 10-minute window = 95% success per window
Compounding across windows: 30 min ≈ 15% failure, 1 hr ≈ 70% failure, 4 hr ≈ 70%+ failure
Improve the floor by 2 points (5% → 3% per window) and 4-hour failure drops from ~70% to ~50%
Small, linear floor improvements produce exponential gains in safe unsupervised run length
Theo's live math showing why a model that seems only marginally more reliable per-step can support dramatically longer autonomous runs.
Steal forDeciding whether a model upgrade or a prompt/codebase fix is worth it for long-running agent workflows.
16:10list
The two questions that define a model's usefulness for agents
How long can the model run without your input?
How likely is it that the result works by the time a human gets involved again?
Theo's replacement for judging models by peak capability (3D worlds, novel demos) — these two questions are what actually determine whether an agent workflow saves time.
Steal forEvaluating any coding agent or model for autonomous/overnight runs.
CTA Breakdown
How they asked for the click.
VERBAL ASK
27:09next-video
“let me know how you feel about this one and how wide your prompts have been”
Soft engagement ask, no product pitch outside the mid-roll Greptile sponsor read.
Add Modern Creator as a preferred source and Google shows you more of our breakdowns in Search, Top Stories, and AI Overviews. It only changes what you see, and you can undo it in your Google settings anytime.
Add to Preferred SourcesOpens your Google source preferences with us pre-loaded. Tick the box and you're done.
A developer who used to type 160 words a minute lost the use of one hand, and rebuilt his entire coding workflow around a whisper, a fleet of machines, and agents he no longer reviews before they merge.
Theo reads Dario Amodei's essay "We Must Pace the Frontier" end to end, checking whether Anthropic's three-step plan for slowing AI down is a real commitment or a well-timed announcement.
A four and a half hour Labor Day stream where two entire YouTube videos get filmed live, one-handed, between sub thanks, a ban, and forty agents running in the background.
Jacob Coxon spent three years pretraining models at both companies before quitting Anthropic and calling the industry's safety race a hubristic gamble. Theo reads the thread, then checks it against OpenAI's own system-card admissions about its newest model.
Theo says he barely codes hands-on anymore, then spends 49 minutes proving he still ships more than most full-time engineers by showing exactly how he runs dozens of AI agents at once.
00:21
00:50
01:11
01:32
02:09
02:32
02:53
03:08
03:54
04:34
04:55
05:15
05:35
05:56
06:16
06:37
07:38
07:58
08:18
08:39
08:59
09:19
09:40
10:00
10:21
10:41
11:01
11:42
12:02
12:23
12:43
13:24
13:44
14:05
14:25
14:45
15:06
15:26
15:38
16:27
16:47
17:08
17:28
17:49
18:09
18:50
19:10
19:31
19:51
20:11
20:31
20:52
21:12
21:33
21:53
22:13
22:34
23:14
23:35
23:55
24:15
24:36
25:22
25:37
25:57
26:18
26:56
38:21
34:48
4:35:12
1:16:51
26:33
49:22