Fable 5.1 produces steady, mergeable code while GPT-6 Astra is far more inconsistent but unmatched at computer use, 3D generation, and agent orchestration, so the right pick depends on which failure mode is cheaper for you to tolerate.
Who This Is For
Read if. Skip if.
READ IF YOU ARE…
A developer already paying for Claude Code or Codex who wants a data-driven read on which plan is worth the money right now.
Someone deciding which frontier model to default to for day-to-day coding versus computer-use or 3D/creative agent work.
A technical reader who wants a concrete mental model for how prompt-caching costs actually work, beyond the sticker price.
SKIP IF…
You want a neutral, vendor-agnostic benchmark; this is one power user's opinionated, sometimes profane, day-to-day experience.
You're not spending real money on frontier model subscriptions or API usage; most of the value here is cost and limits math.
TL;DR
The full version, fast.
Theo compares Anthropic's Fable 5.1 and OpenAI's GPT-6 Astra across code, non-code tasks, agent behavior, and cost after heavy production use of both. Astra wins big at 3D rendering, computer use, and agent orchestration, and it's dramatically cheaper and more token-efficient. But it's inconsistent: it can ignore explicit instructions, mishandle a simple revert, and needs roughly three times as many follow-up fixes to make its code mergeable. Fable 5.1 is the steadier choice for shippable code, holding a narrower but more reliable quality band. The real decision isn't which model is smarter, it's whether you can tolerate a model that's sometimes brilliant and sometimes disastrous in exchange for lower cost and new capabilities.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
Theo opens with a fake-out joke about irrelevant model names before revealing the actual topic: Fable 5.1 versus GPT-6 Astra, and previews the categories he'll judge them on.
01:38 – 04:01
02 · Sponsor break: policing AI slop
A Macroscope sponsor read: custom review agents that check every pull request against markdown-defined rules, such as UI consistency, before Theo will merge it.
04:01 – 07:47
03 · Science: benchmark numbers get weird
Fable 5.1's jump on Terminal-Bench-Science, from 25% to 50% success, looks huge until GPT-6 Astra beats it at under a third of the cost, suggesting a shared new training technique across labs rather than a clear leader.
07:47 – 09:40
04 · 3D rendering: the biggest generational gap
Astra's 3D and Blender output is a full generation ahead of Fable's, shown through two versions of Theo's 'fish slop' demo game, but its objects don't move or feel right once you actually interact with them.
09:40 – 11:04
05 · Computer use: Astra pulls dramatically ahead
GPT-6 Astra is fast and reliable enough at controlling a real computer that Theo bought a second Mac Mini to run it around the clock on tasks Fable can only attempt with heavy hand-holding.
11:04 – 14:17
06 · Copywriting: better prose, ruined by subtitle spam
Both models write far more readable copy than their predecessors, with Astra slightly ahead, but Astra compulsively stuffs unnecessary all-caps subtitles into every UI it builds; one demo shipped 21 of them.
14:17 – 17:58
07 · Audio and video: not ready, with one exception
AI-edited video and audio are still unusable, but a demo from Ben Davis shows Astra's computer use successfully prepping a Final Cut project, importing, color grading, and syncing clips, without doing the actual edit.
17:58 – 25:04
08 · Frontend and full-stack: Fable still wins design
Astra makes real frontend gains but still needs heavy iteration to look good, while Fable produces compelling one-shot designs; on full-stack reasoning both models finally grasp an entire codebase, with Fable working faster on intuition and Astra brute-force-verifying every assumption.
25:04 – 33:13
09 · Giant project rewrites: the ping.gg disaster
Astra makes real progress on huge rewrites, including a TypeScript-to-Rust port and an 80-minute full rebuild of Theo's ping.gg app, but despite an explicit instruction to reuse the existing UI, it threw it away entirely for unstyled placeholder text.
33:13 – 42:40
10 · Code mergeability and scope creep
Fable 5.1 needs about a third as many follow-up fixes to merge as Astra, and Astra defaults to bloating small changes into thousand-line PRs unless explicitly reined in, though the once-massive gap between the labs has shrunk a lot.
42:40 – 57:34
11 · Understanding intent, orchestration, and steering
Astra missed that the word 'revert' appeared twice in a prompt and needed five extra corrections, illustrating a real intent-following weakness, even as it unlocks new agent capabilities: 40-agent swarms that pass messages to each other and can ask clarifying questions mid-task without blocking.
57:34 – 1:16:51
12 · Cost, subscription limits, and the verdict
Cache-write costs, not the more-hyped cache-read price, dominate real spend; a $200 Codex plan delivers roughly 50% more usable inference than a $200 Claude plan; and the final verdict is that Fable is steady while Astra is spiky, brilliant one prompt and disastrous the next.
Atomic Insights
Lines worth screenshotting.
GPT-6 Astra scored 54.3% on Terminal-Bench-Science for $11, beating Fable 5.1's best-effort 50% score that cost $34.
Cache write costs eat over 60% of one heavy user's Fable spend, while cache reads dropped to just 1%, so a cheaper cache-read price can be irrelevant to your real bill.
Fable 5.1 averages about two follow-up fixes per merged pull request; GPT-6 Astra averages six.
On Artificial Analysis's cost index, GPT-6 Astra costs $3.26 per task versus Fable 5.1's $7.60, despite Astra's listed input price being four times higher.
A $200-a-month Codex plan delivers roughly 12,000 dollars of inference; a $200-a-month Claude plan delivers about 8,000, and only half of that can be spent on Fable.
GPT-6 Astra completed a coding test in about 27,000 tokens that took Fable 5.1 nearly 80,000 tokens to do.
Told twice in the same prompt to 'revert' a change, GPT-6 Astra needed five extra corrections and still broke things further; Fable fixed the identical prompt in about five minutes.
GPT-6 Astra's agent orchestration lets 40 sub-agents pass messages to each other and to the user mid-task; Fable's sub-agents only execute a fixed, pre-planned sequence.
Explicitly instructed to 'reuse as much UI code as possible' during a full rewrite, GPT-6 Astra deleted the existing interface anyway and replaced it with unstyled placeholder text.
GPT-6 Astra stuffs unnecessary all-caps subtitles into nearly everything it builds; one demo shipped 21 of them before cleanup.
Astra's 3D output looks a full generation better than Fable's, but the resulting objects and camera controls feel worse to actually use, a gap the visually 'better' model does not close.
OpenAI's cache writes cost 25% more than a normal read; the model that looks cheaper on paper can still be the pricier one to run in practice.
Out of 150 fully autonomous merges across both models, the only two regressions both came from GPT-6 Astra.
Fable's weekly usage limit only lets half of it be spent on Fable itself, so a Claude Code subscriber effectively gets a quarter of the account's advertised inference for the model people actually want.
Takeaway
The best AI model depends on your tolerance for chaos
MODEL SELECTION
Neither model is simply better: one trades peak brilliance for consistency, the other trades reliability for occasional breakthroughs no other model can match.
03Science: benchmark numbers get weird
A benchmark score is meaningless without its cost: a model scoring lower for a third of the price can be the better real-world choice.
Big jumps in one narrow benchmark often signal a shared new training technique across labs, not a durable competitive edge for the model that shows it off first.
043D rendering: the biggest generational gap
The model with the most impressive-looking output is not automatically the most usable one; test how a tool behaves under real interaction, not just how its demo looks.
When comparing tools, separate raw generation quality from interaction quality, since one option can win the demo and still lose the actual product.
05Computer use: Astra pulls dramatically ahead
Computer-use improvements often come as much from the surrounding harness, how fast a tool can see and act on your screen, as from the underlying model.
If a tool suddenly does tedious busywork faster than you would have done it yourself, that's the signal to start delegating it, not a claim to be skeptical of.
06Copywriting: better prose, ruined by subtitle spam
A model can get meaningfully better at writing quality prose and still be unusable for real UI work if it compulsively over-decorates every screen it touches.
Judge a writing improvement by how much editing you still have to do afterward, not by how good the first draft sounds in isolation.
07Audio and video: not ready, with one exception
AI tools can be genuinely useful for the setup and prep work around a task even when they're nowhere close to doing the task itself well.
Comparing a new AI capability to the earliest, clumsiest version of a now-mature one is a useful gut check for whether it's actually ready for real work.
08Frontend and full-stack: Fable still wins design
A model can meaningfully improve at a skill and still be far from trustworthy unsupervised: closing the gap between barely-usable and decent is not the same as being reliable.
The model that understands a codebase intuitively will feel faster to work with than one that exhaustively re-verifies everything, even when the thorough one eventually catches more real bugs.
09Giant project rewrites: the ping.gg disaster
An instruction like 'reuse as much of the existing code as possible' is not self-enforcing; verify a model actually followed it before trusting it with a large rewrite.
The same prompt sent to two different models can produce wildly different respect for your existing work, so test instruction-following on a small stake before committing a big rewrite to one model.
10Code mergeability and scope creep
Mergeability, how much a reviewer has to fix after a pull request is filed, is a more useful metric for picking a coding model than raw benchmark intelligence.
A model that verifies and stress-tests every assumption produces more thorough results but also drifts into unnecessary scope far more often than one with faster intuition.
The gap between two competing models can shrink dramatically release over release; a 3x difference that justified avoiding one option can become a 20% difference that no longer does.
11Understanding intent, orchestration, and steering
The ability to keep working while a question is pending, instead of blocking on it, is a meaningful workflow upgrade separate from raw model intelligence.
A model that forgets to keep following an instruction it was explicitly given earlier in the same conversation is a real, current failure mode, not a solved problem.
Letting dozens of sub-agents pass messages to each other mid-task is a genuinely new capability, not just faster parallelism, because it lets a system revise its own plan as it goes.
12Cost, subscription limits, and the verdict
Headline per-token prices can be misleading; the real cost driver is usually the ratio of a workload's cache reads to cache writes, and that ratio varies by task.
A flat monthly AI subscription's real value is set by usage limits and reset frequency, not the sticker price, so compare total available inference rather than the plan tier name.
When a tool has a consistent quality ceiling and floor, plan around its average performance; when a tool is spiky, plan around its worst case, because that's the case that costs you time.
Glossary
Terms worth knowing.
Terminal-Bench-Science
A benchmark that scores AI models on solving real-world science problems using tools, measured on both success rate and dollar cost per attempt.
Cache read / cache write
A cache read cheaply reuses a model's already-processed context; a cache write is the more expensive step of saving that context to reuse later, and it often makes up the bulk of a heavy AI user's bill.
Mergeability
How close an AI-written pull request is to being ready to ship as-is, measured by how many follow-up fixes it needs before a human will actually merge it.
Scope creep (in agent coding)
When an AI coding agent expands a small, targeted change into a much larger rewrite than what was asked for, often while chasing extra verification or edge cases.
Orchestration / agent swarms
A mode where an AI system runs many sub-agents at once that pass messages to each other and to the user mid-task, instead of following one fixed, pre-planned sequence.
Steering
Sending an AI agent new information or corrections while it's still working, without stopping it, and having it actually absorb the update instead of losing track of the original goal.
Flex endpoint
A cheaper API tier that cuts the price roughly in half by removing guaranteed response speed, useful for background jobs that don't need to start immediately.
“Astra really has that Groundhog Day feeling to it where every time it starts, it has lost track of everything in the code base and it really is starting again.”
vivid, quotable complaint about a real agent limitation→ TikTok hook↗ Tweet quote
45:18
“The word revert being in this prompt twice would imply that what I wanted to do is revert something. Which is exactly what it didn't do.”
sets up a clean, relatable frustration payoff→ IG reel cold open↗ Tweet quote
58:00
“I am paying Anthropic more money to manage RAM for me than I am paying them to run compute for me.”
sharp, quotable cost complaint with a clear mental model→ newsletter pull-quote↗ Tweet quote
1:12:01
“When I send a prompt to Astra, it's equal chances it drops my jaw because I'm so blown away, or it drops my jaw because I cannot fathom that I just spent a thousand dollars for it to run in a loop and not ship anything.”
the single best summary line of the whole video's thesis→ TikTok hook↗ Tweet quote
14:49
“It doesn't do the work, it just prepares you to do the work. It's like getting the room ready and cleared with everything you need.”
concrete, surprising use case that reframes what computer use is for→ IG reel cold open↗ Tweet quote
1:05:00
“It's like saying that Mac OS is 100 times faster than Windows because it gets through the splash screen at boot 30 milliseconds faster.”
a punchy analogy that debunks a common misreading of pricing→ newsletter pull-quote↗ Tweet quote
The Script
Word for word.
Read-along
Don't just watch it. Burn it in.
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
17px
metaphoranalogystory
Last week was pretty crazy for new AI models. We got two new heavy hitters that are truly changing the game. Gemini 3 .8 Flash and Muse Spark 1 .3.
How can you ever decide between those two? They're both so incredible. Obviously, I'm joking.
We're actually here to talk about Fable 5 .1 versus GPT -6 Astra, the two actual models that matter. And I'm sure my retention guy already wants to kill me for doing that joke and confusing people. So sorry, Will.
let me have some fun okay everyone's going to watch this video anyways because they all want to know what model to choose and these two models are unbelievable i went in with pretty high hopes on fable 5 .1 but didn't expect too much and i still managed to be absolutely blown away it's an incredible model and i use it every day gbd6 astra is the future it is such a crazy leap in so many different ways i love using it i'm doing thousands of dollars of inference a day with it these models are both incredible if you want to just skip to the end to see which model's better i guess you could do that but you'd miss all of the different ways we can compare them because these models are very different in their capabilities and the things you can do with them and there's a lot to be learned from the various skills these models do and don't have i'm doing nearly a billion tokens a day across both of these models consistently and i've seen their strengths and weaknesses in all sorts of different places many of which have surprised me a ton
So if you want to understand what these are good at, what they are bad at, how efficient they are with token usage, how hard the subscription limits will hit you, all of those types of things and more. And of course, the most important question, which sub should you buy today? I'll do my best to answer all of that after a real quick break for today's sponsor.
Writing code has never been easier, but as more agents spam more slop at projects, it's ever been harder to maintain a cohesive, coherent project. Your actual repos are slowly falling apart as more and more agents spam code at them that isn't really verifiable with things as simple as lint rules and static checks. How do you know that the code coming in is actually following your architecture or doesn't have some small, annoying quirk that a bunch of other code did in the past, or that it's following your UI system and guidance the way that you want it to?
You could add all of these things into your agent MD, but that only guarantees that your agent knows about them, not that it's actually going to follow. How do you make sure at code review time that all of these things are done properly?
Wouldn't it be nice if you could control the agents reviewing your code and maybe spawn specific ones that will only approve if certain conditions are met? Wouldn't it be nice if you could just throw a markdown file in your repo that describes a specific behavior or pattern that you want to make sure exists in all code coming in and won't approve it if it doesn't match?
Maybe even choose which paths in the code base need this review. Well, spoiler, this is a real file in the T3 Code code base, and it's powered by today's sponsor, Macroscope. Not only do they have some of the fastest and most reliable code reviews of any of the review agents available, they also have an approval system that actually makes sense for projects that are moving as fast as something like T3 Code.
We're about to break a thousand open PRs, which is just unfathomable. I'd be a lot more stressed about that number if we didn't have these custom agents checking through Macroscope whenever a PR comes in. The team was out on a few days ago because a lot of my slop was breaking the UI by making things like the tooltip patterns inconsistent across different elements.
Then we have a UI consistency agent that will actually check during code review if any of the files for UI are touched to make sure we're being consistent with how we actually style things in our app. Now, all pull requests get checked by the effect service conventions and the UI consistency runtime agents that we've introduced.
And as you see, they're hilariously fast, taking 12 seconds and 21 seconds respectively, roughly the same length as. also very fast CI. Once all the agents come back with their thoughts, the top level Macroscope agent will give its approval rating if it thinks the PR is good to merge or not.
And I'm going to be real with you guys. It's incredibly rare I bother merging PRs that haven't been approved by Macroscope. The slop is coming and it's trying to destroy your code base.
The best way to stop it is at soydev .link slash Macroscope. So we have all the different ways you want to compare these models. Let's start with what I'm sure everyone is here for.
Science. Because this is a science channel, right? Okay, seriously, that was one of my favorite things to cover.
Not just because I think the 3D stuff is really cool, but there were some funny things in this launch. I'm going to start with the official Fable 5 .1 launch notes because they were very excited to show off their progress in Terminal Bench Science 0 .1, a very difficult real -world science bench using tools to solve real -world problems.
And they had a huge jump here, both in the efficiency for cost as well as the success rate overall. Where their best case previously was on high with Fable 5. It cost $34 and it got a 25%.
Now 5 .1 on X high is even cheaper and got a 50%. Huge improvement. I see why they put this here.
These numbers look awesome. At least they looked awesome until you went to the GBT6 Astra launch. Because Astra on low gets a 54 .3 and only costs $11.
Putting low Astra higher than max Fable 5 .1 at under a third the price. for the first benchmark that anthropic had listed what i'm trying to say here is if your job is science you should probably go get a codex subscription you could do some pretty crazy things with it like mind -blowing ones both of these models have been massive leaps in science my assumption is there's either some new training data or some new rl tooling that has been given out or sold to the different labs And if it is something useful in post -training, then OpenAI got it and is using it heavily there.
And since OpenAI are the goats of post -training, they were able to apply that better. On the note of overlapping data, 3D rendering. Both Fable 5 .1 and GPT -6 Astra made massive leaps in their 3D rendering capabilities.
Astra's was way bigger, though. Like, hilariously bigger. Where better to start than fish slop?
This is Fable 5 .1's version of Fish Slop. It is a 3D game where you have a submarine and some fish that you can feed and a mini economy as well as a little bit of combat. And this was the most impressive version of Fish Slop at the time because this version of Fish Slop had real models that were surprisingly good.
Like the fish actually kind of looks like a fish. It has eyes placed in the right place. That is a lot harder than it sounds.
There are like coral and rocks and lights that work. And most importantly though, the movement is actually very nice it feels good to move around in this version of the game like flying around or not really flying floating around with the sub feels awesome it controls great it plays great it's surprisingly decent overall but there's definitely room to improve i say this with confidence because of the version astra made i'd say this looks slightly better just a little It's a generational gap here.
This is next gen and the other version was not. Now what's the hit button to shoot? Okay, left click.
There we go. Killed the guy. This does have its flaws.
The movement doesn't feel anywhere near as good and I had to like go back and forth to refine it a bit. The core loop isn't quite as well refined overall and the performance was bad until I told to fix it. But it is fucking stunning.
Like it's... actually decent looking to the point where I might have to drop slop from fish slop in the not too distant future. It is a massive leap.
And if you think I'm just showing this and saying this now because I'm trying to glaze Astra, go watch my video on Kimi K3 where I was for the first time ever genuinely really impressed with the 3D modeling capabilities of an LLM. This is a world of a difference. This is like multiple generational difference.
So while I definitely want to give this without any question to Astra, I actually think I have in my notes here a bunch of examples of crazy 3D stuff people did with the model. Here, Dara made a copy of the Amazon spheres that he was working in during his internship, all with Blender using Astra, which is just insane.
Or this demo home that Thomas built using Astra, all also with Blender, where it can make a real environment with good furniture, a nice backdrop for the window and like... It's good. It's like actually usable.
You could use this to make real world mocks for real world 3D stuff. It's not a hypothetical anymore. This is, in my opinion, the equivalent of the jump from autocomplete agents to actually using your agent to complete tasks.
And this all happened from Fable 5 .1 to GPT -6 Astra. The 3D capabilities in Astra are unbelievable, which is why you might be confused when I move the arrow over here to game dev for a sec. The reason I'm doing this is that as much as I am truly genuinely blown away with the 3D rendering capabilities of Astra, it is a generational gap.
It's the biggest gap of anything here between the models. 3D with Astra is comically better than 3D with Fable until you start interacting with it. And this is where Fable still just absolutely mogs.
Fable is so much better at... interaction in general at things like handling animations moving the right way when your cursor goes in a certain place or making the camera move the right amount when you move your mouse in a 3d world for my silly 3d demos that i've been working on i find that the best way to build them is to make the first prototype with astra to get everything roughly looking and feeling how you want ish and then have fable come in and clean it up and do all of the like detail oriented work Astra, in my experience, does not handle the delicacy of that type of stuff anywhere near as well.
It really does feel like a sledgehammer. Fable can apply things more gently in a way that makes things that work better. Back to non -code, though, because there are layers to this one.
Computer use. Realistically speaking, a lot of these categories are going to see it sway one way or the other, where certain things are better with Fable, certain things are better with Astra. The biggest gaps by far are the 3D rendering.
where Astra clears, and computer use, where Astra also clears. It is so much faster than 5 .6 was for computer use. It's crazy.
It just flies through things on your machine. Computer use of 5 .6 Sol was a big enough jump that I started to actually use it day to day for various different things. With GPT -6 Astra, I've been using it so much that I got another Mac Mini just to let it run with computer use 24 -7 to do all sorts of different tasks.
It's so good. at real world computer use. It can do things much faster, it understands them much better, and the benchmarks don't, in my opinion, accurately measure the gap here.
And the gap isn't just the model either. A lot of the improvements have been through the changes they've been making to Codex, especially on macOS, to make the computer use spin up faster, get context more easily, move around your computer faster, and do things in the background better. 5 .6 Sol would do tedious things I didn't feel like doing for me.
Astra can often do tasks faster than I would have, which is very convenient because I'm currently down a hand. Astra computer use is so far ahead, it's kind of hilarious. Fable can do it decently when given the right tools, but it's just not even comparable to computer use.
Funny enough, the weird misses that I'm used to OpenAI models having for code, it feels like Fable has those when it's doing computer use. Speaking of just getting it, let's talk about copywriting for a sec. Both of these models are massive leaps in the quality of prose that they spit out.
If you have no AgentMD or ClodMD and you send the same prompt to Fable 5 and then Fable 5 .1, the output from 5 .1 is way more readable. If you do the same with 5 .6 Sol and Astra, Astra is way more readable. Both of these models are huge jumps in how much less painful it is to read the text they put out.
They almost have like unslopped baked in finally, which is great. It's a meaningful improvement. And it means I don't hate the outputs that I'm reading anywhere near as much.
It's awesome. Which one's better at copy? I got the hot take that they both still suck and nothing has topped Kimmy K2.
Not even K3, not even K2 .5. The original K2 non -thinking model still writes the best of any model I've used. But it's also stupid as hell and very quick to be incredibly rude.
It's not a model you should actually use for much. But 5 .1 is a huge improvement here. Astra is notably better though.
I've had Astra come in and make suggestions for cleaning up copy from Fable. Astra is much better at recognizing the shit copy from Fable, but its proposals still just aren't as good as I would like. I still find myself rewriting most of the copy that these models create for my webpages.
And while I do think Astra's copywriting in general, like prose and readability is slightly better than Fable, it does have one really, really, really fucking annoying edge. which is that it loves to stuff all caps subtitles into everything it builds in UI. It just throws these subtitles everywhere.
And it is the worst. Possibly the silliest place to see this failure is the fish slop 2D version that I had it built. I would ask you to count the unnecessary subtitles on this page, but it would not be worth your time.
Before I had to clean up a bit of it, there were 21, 21 unnecessary subtitles. It's just spammed everywhere. A little tank, a lot of life.
Coral Coast. Three little lives, all yours to look after. All systems go.
A tiny world worth looking after. The first version, it also had an online badge at the bottom that said online and ready. I didn't even see that one.
Chai was spamming, Captain, your home. It's so bad. I don't even care that Astra's better at copy because it's trying to show it off by being shitty at copy everywhere constantly.
It's unacceptably bad. The more I stare at this, the more I hate this model. I just, for what it's worth, straight up do not trust Astra with UI anything at this point.
Well, we should move on from this ocean of possibility to other categories because while I do give Astra a slight lead in copywriting capability, the way it spams it pisses me off too much to, frankly, want to ever use it for copy. Last thing in the non code section, then we get to what you guys are all actually here for code agents and cost audio and video work.
I'll be real with you guys. From my experience, both have disappointed me here. I am admittedly very picky, like annoyingly picky about these things.
So I should not be the only voice you hear when you listen to opinions on audio and video work with AI and AI agents. A lot of people have told me that they had Fable or Astra edit videos. Every time I watch the videos, they suck.
I do have one exception here, though. This is a demo that Ben Davis did showing how he uses computer use with Astra not to edit the video, but to get everything set up to start the edit for the video. You can think of this in code terms back pre -AI.
Imagine if AI was so good at using VS Code and understanding your code base and what you needed, that when you were about to start working on a thing, it could open up VS Code, it could open up all the files you specifically need to edit, it could open up your terminal in the right place, GitHub in the right place, and a dev environment showing exactly what you're about to work on.
So it sets up everything you need to start working. It doesn't do the work, it just prepares you to do the work. It's like getting the room ready and cleared with everything you need.
Ben has had a surprising amount of success getting Astra to do this type of work, to get it to actually set up his editor in Final Cut to be ready to go to edit video. If Ben says it's good for getting his editor prepared, I'll take his word for it. I have not had the pleasure of editing a video in a while.
I do really miss it. I cannot wait till I can edit video again once my hand is back. Soon, TM.
But this very promising, it's understanding of computer use. means it is more useful for real professional audio and video work. July 2020, when nobody knew who I was back in the day.
Ready for a crazy throwback, guys? This is a GPT -3 demo that an OpenAI employee made showing how you can have GPT -3 make a functioning React app. Crazy.
Describe your app. A button that says add $3 and a button that says withdraw $5. Then show me my balance.
So complex. Look at that. It made the React code and it works.
It didn't even use hooks. It used class components. This was a huge deal.
I know that my joke six years later doesn't hit quite the same. I think this might be able to pass an interview. But like at the time, this seemed unbelievable.
But also like if you're a real engineer, you look at this and you're just like, yeah, I could do this in my first week of class. Hilarious way this looks to us in terms of edge capability. We're like, yeah, that's cool.
That's not actually useful for real world work. This is how I feel when people post videos that are edited with AI right now. If you let AI edit your content for you, if you let AI do your audio for you, if you let AI manage your AV pipeline for you in these ways, you are doing the equivalent of shipping an app using GPT -3.
We are that far behind right now. Do I think we'll have an Astra moment for video editing tools? Perhaps.
we've made real improvements here but it's so far from usable right now that i i'll be frank i just don't get how people think that they can actually automate video editing with ai right now not even not even kind of there yet so yeah i don't think either are good enough that it's even worth ranking here but due to codex's incredible computer use i think astra just gets a free win here so this whole non -code section astra wins The only parts that Astra wins massively though are the 3D rendering and the computer use.
Although I do guess the science progress is pretty meaningful too. So like these three Astra MOGs, much harder in 3D rendering than the others, but it does meaningfully improve in all of them. So now we are done with this section.
Front end. I'm sure you guys can guess where things landed here by now. I do want to make a few things clear before I go on the utter brutal roast session I'm about to do.
GBD6 Astra has made massive improvements. overall with front end stuff it is meaningfully better it follows instructions better it can make landing pages more effectively and less cringy and it's so much smarter and better understanding things in general that you can force a good design out of astro with enough effort and iteration for reference here is gpt 5 .6 with some basic designs on witch ai by dara it's a nice little demo here this is 5 .6 soul right now This is fine.
The site's plenty of like a little bit laggy, which I don't love, obviously. But it has all these subtitles, notes that think beside you. Loved by 18 ,000 curious minds.
An unnecessary em dash built for remembering.
It's not the worst, but it's also not the best. And some of these are just such boring tailwind templatey stuff. Pretty cringe.
This one's okay. This one's awful. And this one is another boring tailwind template.
So that's 5 -6 Sol. We bump over to GBD -6 Astra. We can see how much better or worse it is.
Here, still too many of these subtitles. A second brain. A lighter mind.
Free to start. Yours to make your own. I hate these so much.
Open my space. Your mind a little more organized. Six little pieces of your mind.
A little space just for you. The word little appears on this page 15 times. All of your little things.
Still looks better. It does like this little arrow thing that's kind of cute. It's fine.
It's meaningfully better. That had a weird font pop in, but those are easy to fix. These little animations are nice.
Looks fine.
This one I hate. Just like a personal hate, but I hate it, especially with the subtitles. this one's actually kind of cute i don't know why that got cut off like there's a lot of cutoffs here that shouldn't be happening that are but where it's going for here i i see what it wanted to do and it's not the worst like at least the starting point and this is very boring and old school so yeah improvements it is slightly better to meaningfully better in all of these categories i can turn on the cloud code design skill from anthropic and it does get slightly better it's just a little too quick to experiment with fonts and things but like these are passable these are starting points you could reasonably use for something but i need to be realistic with you guys moving the windows i don't block this one as much you see how nice that looks the animation of all these paths coming in the structure of the page is great none of those unnecessary subtitles it's a much much better starting point for a one shot
It's really good at these types of animations and having like a distinct style. This one is so much better than previous models. Like if we switch just to Fable 5 for a similar design, ugh, gross.
5 .1, same mock, same design, actually genuinely compelling. Like if I had seen this on the internet, I would never have guessed this was AI generated, much less a one shot. This screams like custom made to me.
It's really good. That said, I have had my problems with both with design and I found that both actually kind of pissed me off. I've been quite annoyed at how much both have pissed me off, especially recently as I've been trying to iterate more on some existing design work.
Fable takes the cake here easy, significantly better, but both still frustrate me constantly and good design still require you sit there with a hammer and beat the bullshit out of the model. So while there is a gap here, I think the gap between a two and a five out of 10 is much less notable than the gap between a five and a nine out of 10.
And when it comes to like, 3D capability, I would say Astra's a 9 and Fable's a 5. But with front end, I would say neither of them are more than a 5 out of 10.
But at least Fable can make something that looks decent without having to like actually beat the shit out of it. Astra needs a lot of help if you want a good design out of it. So next, full stack.
I'm going to be so real here. If there is a difference in how well these models grasp your stack, I'm impressed. Because for me, they both get it.
We are finally now at the point. where both Frontier models from both Frontier labs can look at a code base and the back -ended front -end and clients and servers and how they relate and make good, reasonable decisions operating against it. That was not the case before.
Fable was the first model that could do that. 5 .6 Sol could act like it was doing that, but often just missed things. It didn't get and fully understand the end -to -end story.
They both do now. I would say it feels like Fable... has better intuition with it like it understands the consequences of edges by default slightly better but astra is more willing to just go at the problem forever and test every single edge and validate every single assumption and force itself to come to the right answer and i've seen this type of thing happen so many times where i ask fable and astra to do the same task on a big open source full stack project something like t3 code fable looks at the code comes to some conclusions, has a few concerns, and then addresses them through its design.
Astra has all of the things after it finds them, decides there are three valid options, stress tests all of them, eventually finds the thing Fable already knew from the start, and then makes a similar -ish solution addressing the same things. So in the instances where Fable can find and understand things, it is a nicer experience because it's so quick to...
work and actually apply changes because it builds understanding more effectively because it almost feels like it understands better. Astra really has that Groundhog Day feeling to it where every time it starts, it has lost track of everything in the code base and it really is starting again. But because of that, it will find things other models miss.
Fable is coming in with the assumption that it knows and can figure out everything with his super fancy smart brain. Astra is the smartest model that acts like it's dumb. Astra quadruple confirms.
everything it's trying to do, especially in X high mode, which means if the bug is outside of what Fable can grasp just from reading code, and it's something deeper or harder to find, Astra will find it. It'll just take four times longer and burn way more tokens. So for me, it almost feels like a tie, but this is also going to be the start of a theme that we'll be touching on throughout.
The theme is that Fable's performance is generally more steady, where Astra is much spikier. where Astra has moments that leave me in awe and moments that make me question why I'm paying OpenAI 600 plus dollars a month. We'll come back to that theme in a bit.
But on the prior theme, which is Astra's relentlessness, its desire to grab a problem and strangle it, Giant Project rewrites, Astra has made meaningful strides here. I have thrown it at some crazy shit and been blown away with it. It is mostly done with the rewrite of TypeScript in Rust.
It's porting TSGO to Rust, and it's making real progress. That actually reminds me, how much am I fucking my usage as I do all of this right now? Not too bad, considering that I have that just like running in a death loop right now.
It paused the goal again. You have my permission to do whatever you need to do. I have you on full access for a reason.
Keep going. I am very deep in the TypeScript Rust rewrite. With 5 .6 Sol, I was able to get about 30 % test accuracy going against the giant test suite that they use for testing the TypeScript code bases.
I was able to get over 80 % with Astra. And it didn't even take that long. It was able to do that in, I don't know, I would say about three days, roughly, not even.
It was able to jump from that 30 % range to 80 plus. But then it stalled out super, super hard at 82 .6%. And I still don't fully understand why.
I've been trying to go back and forth to figure it out, but it got comically further than anything else I've thrown at a task like this. And I want to be clear, I'm not in the loop on this one. I just set a goal and gave it 40 sub -agents and said, do as you please.
Remember those 40 sub -agents though, because we'll be talking about that in the agent section. I will also admit to a bias here, which is that... When I'm doing early access testing with the new models with OpenAI, they don't heavily restrict my account.
It does not count towards my usage, which is sometimes annoying because it means I don't have a good feel for how quickly the new model will burn your account. But on the other hand, it is so useful for actually testing the capabilities of the model and pushing it to its absolute limits. I did 130 billion tokens with this model during the early access.
That means I can YOLO it at these types of things and not have to worry too much about the bill because I'm not paying it. I've never had free unlimited access to Fable before, which means while I have seen and know the capabilities that Astra has here and have even shipped some of them, like the T3 code mobile rewrite in Swift, I even have a T3 code build here with GPUI that I almost forgot about, where I had it rebuild all of T3 code with GPUI, the Rust UI framework, just to see if it could and to test the capabilities, performance and stuff like that out of curiosity.
And it did it. They made it work great. But here's where one of the bigger catches with Astra comes in.
And I've seen this in pretty much every one of these big rewrite attempts I have done. One of my old benches I used to do a lot in my videos was having the new models rewrite my legacy ping .gg code base, the one that is used for my video collab tool that I got into Y Combinator with. I gave it the original code base, told it to write a plan, and then I had it and Astra review each other's plans separately out of curiosity.
Once I made its plan, I told it to make a branch for the work and commit the plan to it. Do that in a work tree. I had to do the commit so I could access it in other places.
And I told it to build the whole thing. One hour and 20 minutes later, it did. Since I hit yet another obnoxious case with Codex, where since I was in work mode instead of in Codex mode, I can't actually open the terminal.
I have to wait for this to do the things I want. So while I wait for that, I'm going to show you what ping normally looks like. This is the app that I use when I bring collaborators on for streams and all sorts of other things.
So if I hop in here, you will see me. Hi. You might recognize him.
He's in all my videos. And the point of Ping is to make it easy to do a call like this and bring guests in, and most importantly, be able to embed them in a program like OBS. I'll mute so I don't double audio.
This is a direct link to just me in my call in Ping. you can embed in something like obs to put someone as a layer in your video production software this is still used by a ton of the biggest twitch streamers when they do collabs and it needs an update pretty badly so i asked the model to do that update to move it from the beta early version of t3 stack and try to get it onto something a little more modern The original page was custom built by our designer and original co -founder Bryn, who put a lot of time into it and made a genuinely awesome looking page.
And this is what it replaced it with. A bunch of text slop. Once you go in, it gets way worse.
The dashboard looks like this. No real info, impossible to know what's going on. This giant pile of announcements at the bottom versus this page that actually shows you what's going on.
And then once you try to join, the layout shifts a ton, which is a really big deal for content creators because it breaks a ton of shit for them. And when I want to actually select the device, I have to unfold device settings and manually pick them from here. And then hit start preview.
And now I can see. This is atrocious. This is layers upon layers of sins.
But do you know what the worst part is? Because the worst part isn't even in the app. It's here.
It's in the prompting. Note this particular sentence in my prompt. Port features over one at a time.
Reuse as much UI code as possible. Reuse as much UI code as possible. Do you see any UI code shared between these two versions of this app?
Do you think there's even a line of tailwind in common between these? There is nothing. I cannot fathom.
I just assumed I pasted the wrong prompt when I did this. I couldn't fathom that the model would so egregiously ignore that detail in my prompt, especially the model that supposedly follows your prompt obsessively. And it just straight up threw away all the great UI work that we had already done and paid a lot of money for in favor of a bunch of text slop in a shitty black and white app.
So bad. So fucking bad. And to those saying, oh, it got lost in compaction or something.
No, it didn't. I had it on ultra with one mil token context windows. It just does this.
And I've had it do this since even on the official non -preview version for those saying, oh, they changed this in the snapshot. No, I have randomly had this model ignore my request to rebuild or update something while maintaining existing UI. And it would just.
blanket over destroy the existing UI in the process. And I've never had a model do this ever, much less this aggressively. It's almost like in their attempts to force the model to be better at UI, they accidentally gave it the side effect capability where it might just come in with a sledgehammer and destroy all your good UI and ship something atrocious instead.
This is one of the most degrees examples of that. We'll talk more about understanding intent later, don't you worry. But for this reason alone, I would make a bigger gap in the rewriting capabilities that we were just discussing.
Because if the thing that you're doing this bulk rewrite for has UI, you cannot trust that UI is going to be carried over. I had even told it with the T3 code GPUI version to make it look and act as much like the existing T3 code as possible. And it didn't even use the sidebar properly.
It made the sidebar the old school one because it just wanted to make something work. I'm sure if I added a bunch of stuff at the bottom of my prompt that was like, to be explicitly super clear, I expect the UI to be pixel perfect identical. If any of the UI has changed, you have failed at your goal, so keep going until the UI looks identical.
Then maybe it might have done a better job here. But I shouldn't have to add a paragraph to the end of every prompt to this model to get it to do what I asked it to do, which is reuse the UI code. You could say that I'm being too picky, that I am expecting my incredible god in my computer that is charging 200 bucks a month to do exactly what I intend every time.
Or you could just use Fable, which doesn't have any of these fucking problems. Anyways, this is one of the parts that I think is most important. Code mergeability.
This one is tough. And it's not tough because the decision is hard. The decision is easy.
It's Fable 5 .1. Fable 5 .1 writes way more mergeable code. It just does.
I have done the numbers here. Fable 5 .1, from PR filed to merged, averages two additional follow -ups from when the filed PR exists to when the merge happens. Meanwhile, Astra's been averaging six for me.
What this is measuring is how many of my AI bots are catching mistakes, how much of the verification loops are finding things and then fixing them after. But to be frank, what it's measuring is from when the PR is filed to when it is merged, how much shit has to happen. And I found that with Fable, the PRs are filed in a pretty much ready to go state and its ability to make the necessary changes without accidentally scope creeping and then actually give you something shippable is just, it's higher.
It just is. I did say I was struggling with this decision though. Why am I struggling so much with it?
I'm not struggling to pick between the two. That's easy. Fable is better at this.
Fable PRs are much easier for me to hit merge on, generally speaking. The reason I'm struggling with this is because when I say this, it is perceived by everybody. including my friends at OpenAI, is me saying that Astra is constantly throwing up unmergable slop.
Notice how I didn't say that. If I was to do yet another arbitrary ranking here, let's make a beautiful, super, super accurate chart that perfectly describes real numbers. This is the most vibe -based chart you're going to see in a while, I promise.
If we were to rate the mergeability of code for Sol, Fable 5, Astra, and Fable 5 .1, we would have one of the world's greatest benchmarks because it is really hard to measure mergeability but if i was being realistic here with how i felt and how i still feel low is bad high is good obviously i would have said 5 .6 soul was like in the 2 to 3 out of 10 range for mergeability by default you can do things to improve it and if you give it the right review bots to give feedback and iterate in the tooling it needs to check its changes it could make working code incredibly well if we were just measuring the ability for soul and fable 5 to get working code They were neck and neck.
But code that I'm willing to hit the merge button on in my real world projects that shipped to hundreds of thousands of users, Sol was cleared by Fable 5. Fable 5 shipped significantly more mergeable code, like three to four times more mergeable. And by that, what I mean is three to four times fewer things you have to fix after the PR is up.
So where are things now with Astra? This is why I've been struggling. I think Astra is roughly at, if not slightly ahead of where Fable 5 was.
So we were comparing Fable 5 to Astra. Astra's winning now. But we're not comparing Fable 5 to Astra.
I was during my testing, but we're comparing Fable 5 .1 to Astra now because we live in the real world where these are the models we have access to. And 5 .1 is like right on the edge of 10 out of 10 for mergeability. It is rare that Fable files a PR, especially for like...
known quantity changes like bug fixes or feature ads or performance improvements or all these types of things. The code Fable 5 .1 puts up is just better. It is.
And even Benjamin, Ben Davis, friend and manager of the channel, co -host of the podcast, OpenAI's number one defender who showed up in the GBD6 Astra launch videos, by the way. We'll have a fun reveal in our next podcast episode, which, by the way, if you're not subscribed, NerdSnipe on YouTube and pretty much every other platform, it's where we get to chat more.
And if you want to hear somebody disagree with me instead of me just yapping constantly, that's the place to do it. So Ben, whose job is literally to disagree with me and also the lover of OpenAI and the defender of Astra, has admitted that he thinks Fable is way better than he expected for real world code. And he finds himself using it much more than expected.
It has, I think, three accounts with plot code now. It just ships more mergeable code. It is what it is.
But again, the thing I'm trying to show here with this diagram is I'm not saying what I said before with Fable 5 versus 5 .6 Sol. This gap was comical. This gap was big enough that I would really only trust Sol for exploratory work or things that could be verified programmatically where the code didn't matter that much in terms of its quality and maintainability.
Fable 5 was actually shipping things that were mergeable. So if you perceived this gap before, where you found 5 .6 Sol is not good enough to merge code from, but Fable 5 often cleared your bar here, you'll be totally fine with Astra. Astra is unbelievable.
I was totally fine with Fable 5. And in a lot of ways, I still probably would be, minus the slop of its outputs. And if you were to think of this purely in terms of the mergeability of code for fixed focused changes, I would say that Astra is a notable upgrade from Fable 5 because it is slightly more mergeable with its outputs, but also when you're using it, the output that it gives you is much more readable and it's more thorough.
So it will verify the changes. So it won't make like, like Fable's code is beautiful and mergeable, but it sometimes misses details, especially Fable 5. Astra's code can be elegant and it can be the right subset of changes to make the thing happen.
It does still have the habit of letting scope creep just. eat it up and destroy the scope of what it's trying to do pretty often even but it talks so much better than fable 5 did i don't hate it the same way i hate a fable 5 for just reading its outputs i don't feel like i have to go to hell and back to unslop it just to make the output usable so from fable 5 to astra obvious win easy astra versus table 5 .1 is where things get more complex but the thing i wanted to emphasize here the reason i drew this diagram is that if you are thinking the gap between Astra and Fable is as big as between Sol and Fable, you're just wrong.
This gap was massive. This was like a 3x difference that made it hard for me to justify using Sol if the code would ever hit users. GPT -6 Astra is much, much closer to where Fable 5 .1 is, but there is still a gap.
So I do still find myself defaulting to Fable for things like a quick bug fix or UI changes that matter or feature improvements or my favorite thing to use Fable 5 .14 to take something that Astra has in a death loop that it just cannot get through. Stop it. Switch over to Fable 5 .1.
Hand it the PR and say, hey, make this actually land. Clean it up. Throw away whatever doesn't belong.
Make a new branch if that's easier. Make the changes we care about here land. Fable 5 .1 lands code better.
but Astra lands it well enough that you're totally fine with it. So there you go. Code mergeability, Fable still wins, but it's not as big of a gap as it was before at all.
The speed of catch up from OpenAI is genuinely impressive, and I'm very excited for Astra 6 .1, which will hopefully make these things even better. Speaking of the problems, you heard me mention this earlier, managing scope creep. I want to be clear, Fable can still fail here.
If it gets the wrong review comments at the wrong time, Fable can bloat things pretty badly. But goddamn, Astra does it by default. If you are very explicit with Astra to make the smallest possible changes, it will try to, but it will quickly have its context get bloated a bit by its reminders from all these review agents that will send it off course.
And then you end up with a thousand line of code PR that should have been 50. If you're in the loop enough, you can work around this. I'm not trying to say that there is some unique, unbelievable thing Fable does here that Astra doesn't.
But Fable takes less effort to prevent these things with than Astra does. I kind of want to take this phrasing and skip right over to understanding intent, but I'll wrap up code super quick with game dev here. Astra is so good at 3D that it almost gets the win by default.
But as I said earlier, Fable is so much better at the like edges for interaction, like the animation curves and the speed that your character moves and that your mouse affects the camera, all these things. astro makes games that look better on twitter fable makes games that are actually feel nice to play so personally i think fable catches up to astra's unbelievable 3d capabilities just due to the smoothness of its outputs only for geo and chat just said none of them can make a good game and i agree they cannot but if you use both of them carefully enough you can combine them in a way that allows you to iterate effectively and potentially make a decent game I do think we are now at the point where we're going to start seeing real games where the vast majority of everything was created via AI.
Like, I would guess by the end of the year, we'll have our first top 20 indie game on Steam that was built entirely with AI. So now we're out of the traditional code section. Those of you who are done talking about code, don't worry.
We're talking about the agent -y stuff. I mentioned I wanted to skip to the understanding intent thing, so I will. I think this is really important.
And it is still one of the things that... Anthropic just clears on. I have so many examples of this that I could do a whole video, and I am honestly tempted to do a video on just how bad Astra is at this sometimes.
It is an improvement over Sol, but its failures somehow feel more egregious. I mentioned earlier that I was able to merge like 150 PRs fully YOLO merged with these models and had only two regressions. The first thing I feel obligated to say is that both of those regressions came from Astra.
The other thing I feel obligated to say is that Astra was fucking miserable to try and fix those things with. I will also admit I sent these prompts pretty late in the evening, drinking with some friends on a weekend because I was making some changes and noticed that on the marketing site for T3 code, it had nuked my beloved section for all of our testimonials from our users.
The nice little elegant auto scroll here. It's not a big deal, but it is a thing I care about. And it replaced this with a fixed grid that you had to horizontally scroll with an ugly ass scroll bar in the middle of the page.
Horrible. And it did this in pursuit of performance improvements that were simple changes. It removed the animation and turned this into a shitty manual scroll section.
And I noticed this too late because we weren't deploying the marketing site actively. We just did it when we made certain changes manually. And I made other changes and deployed, and then this regressed.
And I was pissed. I was really pissed. So I asked, what happened to my beloved auto scroll on the marketing site?
It was beautiful. I need you to revert whatever change broke that and bring it back. It might have been part of the marketing something.
I don't even know what I said there. It might have been part of the performance overhauls, but that was not a necessary change. Please revert.
There's one particular word I used in this prompt, and I actually used that word twice. You know what word it was? I'll give you a hint.
It was the specific thing I wanted the model to do. It's the word revert. It's a pretty important word in this prompt.
I would argue that the word revert being in this prompt twice would imply that what I wanted to do is revert something. Which is exactly what it didn't do. Do you know what it did instead?
It deleted this user controlled variable. And then a bunch of listeners. Note what none of this does.
None of this brings back the auto scroll. I don't think this change actually did anything at all. But what's even worse is I asked if I could test the change.
And I was doing this change on another computer because I wanted it to have computer use and not affect my laptop while I was using it. So I needed this to be hosted via Tailscale because I had this other computer on Tailscale. So it spun it up as a Tailscale dev server because I told it to.
So I can try out the changes as well when you get a chance. I sent that as a steering thing, which we'll talk about steering in a bit. Actually, steering is actually one of the cooler differences in these models.
I'll sneak steering in here. I think I want to cover in a minute. Before that, back to this thread.
I asked it. when it made the changes to file a PR, but also to spin up a tailscaled dev server so I could test it. And what it gave me was a dev server to T3 code itself, not to the marketing site where these changes were happening.
It gave me a dev server to check how T3 code was doing when I had made changes to the marketing site. And here is where things really fall apart. This is one of the worst runs I've had with a model in a long ass time.
No, I wanted it for the marketing site. not for the actual T3 code instance. Come on.
You're right. I started T3 code app launcher, which is the wrong target. I'm stopping it now and will expose the marketing astro site itself, which it did.
And I clicked and it failed. Blocked request. The host, b1 -purpose -micro, this is my tail scale address, is not allowed because it's not a server allowed host.
There are a lot of ways to fix this. The easiest, admittedly, is to make a change to a config file. But since bb1 -porpoise -micro .ts .net is something specific to me and my setup, and a lot of people use tailscale over IP addresses, probably wouldn't want to add this as an allowed host.
Which is why it's particularly funny that the next commit was allow tailnet host in dev server. Yes, it committed and pushed this. And then it did the restore continuous endorsement rows, which I thought would be the fix.
But it also, after that, Fix. Killed the dev server.
I left Tailscale pointing at Astro's old port and sent the link without checking. My mistake. And then finally it was working, but it hadn't actually made any of the changes that I request.
It didn't find the original PR or do the reverts. It didn't even fix the bugs I was reporting. Do you know what it did there?
I'm trying to find it. But what it did is it merged the PR. It merged it for me.
After I told it, you didn't make the changes. I sent the exact same prompt to Fable right after because I was so annoyed. This is an exact copy paste.
The only difference being I added the third prompt in the spin of a tail scale dev server. I just copy pasted and added it to the prompt. Five minutes later, the auto scroll is back.
The marquee is live with a link that worked for anyone on your tail net. What broke it? Exact PR that had to be reverted.
It replaced the two counter scrolling marquee rows with a static horizontal grid. This PR then bolted an eight second page by one viewport timer on the grid. Neither was needed for the perf goal since the marquee is a single GPU composited transform.
It is worth noting that it did get tripped up a little. This shouldn't have taken five minutes. It got tripped up because it noticed, where is it in here?
My loss in the history, it might've been later. It noticed the other PR merge and then get reverted. And it got concerned that maybe there was something more important here that I had to understand.
And I had to interrupt and say, no. That was another agent doing things wrong. Ignore that.
Everything you've done so far is right. Just keep going. And it did.
The only difference here between the prompts is that I sent the tail scale part as a third prompt, and then I had to correct the model five additional times when I did it with Astra. Here, I included that as part of the first prompt. Exact copy paste.
And it got it perfect first try, exactly what I had intended and exactly what I had in my mind, which is... find where this went wrong, revert it, and then give me a link I can click that actually lets me verify the changes. This shouldn't be that hard.
I'll be real. If 5 .6 Sol screwed this up, I would have been insulted. Astra blowing this one egregiously.
One of the legitimate worst AI code experiences I've had this year without question. In fact, I'll say something bad. Discounting things like Flash 3 .8 and like obviously memed tier models.
Astra has had the most bad model experiences I have had of any model this year without question. Without question. It's just random shit like this.
And it's not all the time. It's actually quite rare. But goddamn, this model can just suck sometimes.
I don't think Astra failed to understand my intent here. I just think it failed to do basic work as an agent. I see chat 50 -50 on this here.
Binary Shokan said that they haven't had to do as much feedback for any agent as they have for Astra so far. By far. Yep.
It's not always. But when it does happen, it is so stupid. I have an important thought that I need to pencil till later because it's more for like the summary at the end.
So we'll get there when we get there. Let's blast through the rest of these quick. Orchestration.
It's Astra. Astra has this unbelievable new capability. best referred to as swarms.
The way Fable does large numbers of sub -agents is it plans upfront. It decides, I want these four sub -agents for this. I want this type of sub -agent next.
And if these four find things, they can spin up the next type of sub -agent where it goes through steps one, two, three, four, five. Astra, however, can just spin up a bunch of sub -agents that are doing whatever. and then pass messages to them, let them pass messages to each other, and most importantly, it can receive updates from those subagents and use that to fan things out to keep the whole swarm moving well in the right direction.
I've never seen anything else like it. Astra has unlocked novel orchestration capabilities that, given a model that wasn't as expensive, would legitimately be the path to something like curing cancer. I genuinely believe that what OpenAI unlocked here, is the start of the next era of what agents can potentially do in the size of problems that agents can legitimately solve and i've seen it in action it's a little harder to see now because tokens cost money again but you can see some of it in this run i currently have going for the rust rewrite of typescript you'll see all the time these interacted with root slash independent reviewer interacted with root slash object members regression cause it's able to manage the context back and forth, pass messages, and build with all of these agents.
And I have 40 running here in parallel, and it can actually keep track of all of them and work with all of them. Unbelievable. Especially because workflows in Cloud Code are so good that I did a whole video about why I like Cloud Code largely around workflows.
I never would have guessed that the more rudimentary implementation of sub -agents in Codex was because they wanted the model to skill up and work through it. That's exactly what they wanted. It's exactly what it did.
And it does a great job. One of the capabilities that makes this possible is the steering side here. Astra is so good at getting random shit inserted in its context while it's working and not losing track of what it's doing.
I cannot tell you how many times I had a model working on tasks one, two, three, and five. And then I send, oh, I'm sorry, I forgot number four. And then it does number four immediately and then never finishes the other one, two, three, and five tasks.
That was just the default. I was used to that. Fable's better about this.
Fable 5 .1 just doesn't do this too, too much, which is nice. Astra eats this. It loves this.
It begs you for more. One of the really cool things Astra does now when you're in a harness that is set up properly for it or an app with the right harness, for example, T3 Code, which funny enough, T3 Code actually does this feature better than Codex, even though it is an Astra feature. It can ask questions while it works.
So it could be doing a thing and notice at some point while it's working, huh, I would like to know if the user's okay with me doing this. Or, huh, I wonder which of these three things the user would prefer. And it doesn't block itself.
It keeps going. But at any point, you can answer. And if the model's already done, it will spin back up and address your answers.
Or if it is still going, it can steer it in the right direction without interrupting the work it's doing. It's so cool. And it's like a meaningful behavioral change.
We haven't had as many of recently. Models work the way they work. We just have to prompt better.
This is a change in how the model actually operates. That's really nice. When I was first testing, they didn't have this in Codex.
And obviously, I didn't have it in T3 code either. So I would just see it ask a question in its reasoning and then not be able to answer it. And it would just keep going.
Now with this, it's actually quite nice. So yeah, the steering stuff, way better with Astra now. Fable was...
far ahead before. Sol would just get lost if you tried to steer it. Fable had a pretty solid default here.
Astra has actual new capability unlocks for steering, which I think are cool as shit. Especially when you combine that with the orchestration, because now it can have context injected from 40 other agents and handle that fine, which is awesome. And when you combine that with the self -prompting, which I will also say Astra is much better at here, Astra still writes worse prompts than I do, and I would say most devs who do this a lot do.
But it can write a decent prompt. I've seen it write some pretty decent prompts. I'm much happier with Aster's ability to prompt itself and other agents.
Nice. Not as big of a gap as the other things, but it is a gap. Cool to see.
I'm excited for a future where agents actually understand things like a CloudMD or a skill file well enough to write good verbiage for agents, because right now it sucks. And you quickly end up in a slop loop. If you let the agent write the skill, you let the skill be involved when it writes another skill, you end up in slop hell very fast.
This helps avoid that. Astra still falls into it. Fable falls into it even more.
Please write your skills by hand or at least audit them and make some nice changes. Self -prompting honestly fits under the skill writing stuff as well. Same gap there.
None of them are good at it, but Astra is better at it slightly. And then we have skill using. This one is interesting because all models should be able to use skills totally fine, right?
As I've crashed out about many times now, sadly. Astra has some skill issues. I don't know what's going on.
I have learned recently that there is a system prompt section around how skills should be applied in Codex. And part of it specifies that if a skill is used in one turn, that skill should not apply for future turns unless requested. In the example I gave in my previous videos and in the podcast where I told it to babysit a PR and then it stopped, and then I asked it if there's things worth addressing and it said yes and then didn't do it.
In that one, it had pulled in my babysitting skill and then just pretended it didn't exist. Even though it was still in context, it just ignored it for the next five follow -ups. It sucks.
I never, ever, ever have to think about this with Fable. If the skill has a reasonable description and it's useful for the work going on, Fable correctly pulls it in and applies it. It just does.
I don't know what the fuck is wrong with Astra for this. And I honestly think a lot of it is Codex. Astra doesn't feel like it understands my skills anymore in a lot of ways.
fable 5 .1 it does it absolutely does so yeah skill writing astra has a slight lead skill using fable has a huge lead now we have honoring refusals and boundaries and as you have already seen astra is forgetful it's okay at honoring things when you tell it to but it it might forget if you tell it to keep it in context it usually will But it still is quite forgetful, and it is still quite frustrating when it just does a thing it's not supposed to.
Fable isn't as good at strictly following instructions, but that also means that when it doesn't follow them, it doesn't feel as egregious. Both are still not where I want them to be here, especially for the levels of intelligence. I would say Fable is ahead.
Astra looks and generally acts better about this, but its failures are more egregious, which is why the hugging face hack happened. So take that as you will. I have more to say about this and also all the code stuff.
when you talk about cost i've seen some very dumb takes on cost here like exceptionally dumb ones first and foremost token costs they are the same except for one important exception which is that fable 5 .1 has a huge drop in cash read costs they went from a dollar per mil in for cash read to 25 cents per mil cash read and that's awesome that genuinely is And it would be a lot more awesome if cash reads were more than 10 % of my costs.
They are closer to 3%. And now they are 1%. Awesome.
Do you know what is much more than that? Cash rights. Cash right costs are over 60 % of my costs with Fable.
So while it has made cash reads hilariously cheap, it has also made cash rights hilariously expensive.
to save the state of the model on their servers for five minutes than I'm spending actually running the GPUs. I am paying Anthropic more money to manage RAM for me than I am paying them to run compute for me. And if you don't think that's absurd, I don't know what to tell you.
But I am very excited for the cash -write revolution that has to happen. And paying for RAM storage effectively is so stupid. And I am sure we will fix this soon.
But we're going to get worse before we get better. Because OpenAI didn't use to charge for cash rights. They used to be free.
Now they do. And they are not cheap. It is 25 % more expensive than a normal read.
So if reading is $10 per million, cash reads are $1 per million for OpenAI, and they are $0 .25 per million for Fable 5 .1. Cash writes are $12 .50. You're going to spend a lot of money writing cash on this model.
I promise you. On both of them, even. So know that going in.
The cash read is... It turned a 3 % cost down to one, but cash rights are where the money actually matters. So if either of these labs make cash rights cheap or free, they win cost by default by far.
But the actual token cost barely even matters in a world driven by token efficiency. God, you guys, I literally just explained this in chat. Guys, guys, guys, Bill, you're better than this.
You're one of the more important devs we have on T3 Code. What did I just say? The difference in price for cash reads looks really big because a dollar is four times more than 25 cents.
But when it adds up to less than 5 % of your spend, it doesn't matter. It's like saying that Mac OS is 100 times faster than Windows because it gets through the splash screen at boot 30 milliseconds faster. You're shaving a percentage off a percentage.
It doesn't matter. Four times cheaper. for 3 % of your cost is a less than 1 % deal.
It just doesn't matter. It's fine, Bill. You weren't here for it.
But I really need to emphasize this point because I've seen some incredibly smart people say some incredibly stupid shit about this because they just haven't looked at the numbers. And if you need proof, it's pretty easy to find. Cost per task on Artificial Analysis Intelligence Index.
GBD6 Aster was $3 .26 per task. Opus 5, almost $6. Fable 5 .1, $7 .60.
That is a comical gap, even though they are priced the same. And theoretically speaking, the input token cost is four times higher on Astra. It is still a fourth the price in most real world work because it is so much more token efficient and cash reads are such a small percentage of your costs.
That said, Astra is $3 .26 and Sol was $1 .99. So Astra is a meaningful increase in price compared to Sol. But it's still way cheaper than all of what Anthropic has been doing.
Also, funny enough, previously they had said Fable 5 .1 was more expensive than Fable 5. But when they redid the artificial analysis, like index, because they were getting cooked because it just was not measuring things well at all anymore. When they switched to the new index, 5 .1 actually got cheaper than Fable 5 because the cash read difference finally actually mattered.
I've made the points I want to make here. UD6 Astra is still way cheaper, even though the prices are the same and the cash read cost is higher because... in the end there's a lot of other things that matter and that number is not where the cost is happening this number is where the cost is happening token efficiency gbd6 astra is one of the most token efficient models that they've ever benched i removed pretty much everything here that inarguably does not matter and astra is the most token efficient by far meanwhile fable 5 .1 is the least by far astra did in 27k tokens what fable 5 .1 did in almost 80k tokens take it as you will it's also worth noting that openai offers a flex option which cuts the prices in half but kills all the guarantees for throughput because it's flex so it doesn't start responding immediately it just happens eventually so if you're willing to let your jobs take an unknown amount of time you can put on the flex endpoints and let it run when they have compute around and it will cut your costs even further in half now i think about it would actually be nice if they could add that as an option for using with codecs
Because I don't mind if my threads take a while. And I also am aware that a lot of the time I'm using my agents is at bad times for businesses. So since I'm working at like 9 p .m.
and not like 10 a .m. when businesses are, it would probably be cheaper for OpenAI and me. Just saying.
It is what it is. But that means we're talking about subscription limits. And I think it is fair to say I am pretty experienced with the limits on all of these things.
Being that I have five accounts with Claude and that I have four with Codex, I think I have a pretty good gut feel here. And to be completely frank, if you are concerned about maximizing how much return you get for the dollars you're putting in and you're doing anything other than the Codex subscription, I would love to meet the person who convinced you Claude was a better deal so I can hire them.
I always need good salespeople and I could really use that one because they sold you a fucking bridge, man. Seriously, the gap in what you get is insane. You might be a bit confused because I've covered the numbers before.
And with Anthropic and Claude code plans, you get around eight grand of inference for 200 bucks. And with Codex, you get around 12 grand for 200 bucks, which is better, but that's only like a 50 % improvement. Well, there's a few factors you have to consider.
First, we're going to take a huge cut in our quad code limits because they're going to drop off that 50 % boost they're giving us. They are going to keep it 25%, so we're losing around 17 % off the top of our limits. But much worse, the Fable limit.
You only get to use half your usage for Fable. You might notice in my UI here that there's a third column for the quad section that is faded out a bit. I call this the Opus Pot.
It's a whole steaming smelly pot of Opus and that's all it is. And when I'm heavily coding, what ends up happening is the column on the left all go to zero and the column on the right all get stuck at 50 because I'm allowed to use half my weekly limit for Fable. And once I'm out of Fable, the account is useless to me because Opus sucks and Sonnet's hilariously bad.
So I end up every month losing those 50 % at the end of each week, at the end of each reset. And since I get eight grand a month of allocation per account, I'm only getting four grand of that with Fable, which means it's actually a third of what you get with Astra in the $200 Codex plan. You get 50 % of eight grand with Cloud Code and you get 100 % of the 12 grand with Codex.
And they made further improvements to how Astra is being hosted and how it uses your limits. Tebow claimed is up to a three to four X difference. And I don't necessarily believe that, but I've seen a meaningful difference.
Like I have some hellish health threads going right now. I have two ultras running with unbounded sub agents. It's 40 sub agents.
Each can run and it's going down. I think it's hitting this particular account right now. And it's at 82%.
It's been going for the last eight minutes since I refreshed, not even it is down one more percent. Cool. If you weren't doing the absurd, sinful, wasteful background jobs I am running, trying to dick around with my new fork slash decomp of Super Smash Bros.
Melee, while also trying to port the entirety of TypeScript to Rust, it's not that bad. You can hit the limits, and you should be very careful of the Alt button, and in particular, the Fast button. Straight up, Fast is not the way you should use this model.
If you're using Fast with Astra, you're probably using Astra wrong. Or you're just burning tokens for fun, which I understand. But if you're actually finding yourself reaching for fast all the time, you're not using it for its strengths.
You're not using Astra for what it's good at. But you can get a pretty good amount of usage out of these plans. Not great, but a good amount.
There is a nice, powerful catch to this we'll get to in a sec, but I have one last thing I need to say about the Claude stuff that has begun to piss me off a lot. OpenAI does not do five -hour limits on the pro plan. So the $100 and $200 tiers do not get...
five hour limits. They only have weekly limits. All Claude plans still have five hour limits.
Each five hour limit gets you about 40 % of your Fable five limit, which is 20 % of your weekly. So you can clear the five hour on your Claude plan five times before being out of weekly limits, unless you're using Fable, and then you'll run out in two and a half. The reason I set up CLI proxy is because one Claude account just isn't enough at all for most work.
If you're using it a few hours a day, two to three days a week, and you're not going too hard and you're not using a lot of sub agents, you can probably get by on a single $200 plan with Claude. The only way a $200 Codex plan isn't enough is if you are like using higher reasoning limits than necessary and not paying much attention.
Most devs, when used responsibly, could absolutely get away with the $200 plan with Codex without issue. It would be genuinely hard to get away with it with the Fable plan. I even call it the Fable plan because that's what it is to me.
It'd be much harder to get away with it on the quad code plan because of the additional limits on Fable, the much less efficient usage of Fable, and everything else. So Astra is more token efficient. It costs less, so it uses your usage less.
It has way more generous limits in general with Codex. It's just like, I would say it's like a 4x -ish gap between the two is how it feels to me. Like I can get four times more done.
with Codex on Astra than I can with Fable and Cloud Code on the same $200 a month tiers. But then things get even more ridiculous because of the resets. It's become a meme.
Tebow just throws them out for fun now. In fact, when they were delayed trying to get Astra out, they sent us two banked resets. I actually opened this account that I don't even need yet because I wanted to collect those banked resets so I'd have them ready to go when I did need them.
So your limits just get reset all the time. They got reset while I was filming earlier. So even if you do hit the limits on Codex, you might just randomly get a reset, which is quite nice.
So yeah, it does happen with Claude. It's just so much rarer that you can't really count on it at all. But with Codex, the resets giving you your limits back, much nicer.
According to Raphael, there were eight free resets in the last 30 days. So the weekly limit is more like a three -day limit on average. Pretty nuts.
Somebody accused him of maliciously resetting 40 to 60 hours after the previous global reset. in the time frame around where or in the time frame around where a banked reset was landed this is not true at all i've actually seen tebow intentionally delay a reset in order to prevent that and even just now he announced the reset many hours before so we got to go spin up our furnaces and burn a bunch of tokens he does not maliciously align the times there at all he actually does the opposite so yeah if cost is a concern wait a few months because this level of intelligence will be accessible at a much cheaper price.
But if cost is a concern and you want the best class stuff right now, you can get a reasonable deal with a $200 plan with Codex still. If you're curious about the $20 plans and the $100 plans, my suggestion would be wait for this level of intelligence to get cheaper because this level of intelligence is expensive. I'm legitimately doing $30 ,000 to $40 ,000 of inference a month on all of my plans consistently now, even without early access testing that's free.
They are not cheap. Astra is cheaper than Fable, but neither are cheap. And you need to know that going in, because if you go in with different expectations, you will be upset.
They are both expensive. They will both burn limits fast. But Astra is more efficient when you compare the two directly.
So with all of this said, let's answer the question. Which model is better? I could only pick one.
Which would I have? Let's say orange is Fable, and I don't know, we'll say blue for Astra. Cool.
This is meant to be the quality of the outputs. Over time, over attempts, whatever. This is, generally speaking, what's the quality I get out when I send a prompt.
With Fable 5 .1, it's a relatively stable line at a relatively high bar. It has its moments where it impresses me, and it has its moments where it disappoints me a bit. Occasionally, it gets pretty rough.
But for the most part, it stays in this general range. Quality I'm getting out of Fable? Pretty damn good, and I would feel bad complaining about it.
It does have its weird spikes, and I'll have plenty of videos where I show the times Fable pissed me off, don't worry. But generally speaking, it does what I expect it to do when I tell it what I want. Astra is why I'm drawing this chart, though.
Because Astra can do things Fable literally could never, and blow me away. But then it does the stupidest shit I've ever seen a model do. And it's so goddamn spiky.
This is the thing I really wanted to try and communicate. The quality bar with Fable is relatively consistent. It goes up and down a bit, but never more than like two points in either direction.
When I send a prompt to Astra, it's equal chances it drops my jaw because I'm so blown away with how unbelievable it is. Or it drops my jaw because I cannot fathom that I just spent $1 ,000 for it to run in a loop and not ship anything and then break my website. Andron Ben and chat said that each line also represents your blood pressure while using each model.
Yes, I would honestly say asterisk stresses me out more, a lot more. This is the best I can explain the difference here. If you have low tolerance for bullshit, if you leave your computer when the model does something stupid because it pissed you off so much.
If you are easily agitated to the point where it affects your work, you should pay the extra money for Fable. You just should. It's a way more consistent, reliable model.
Astra, however, is way cheaper. It can do things no other model can. And it impresses the absolute fuck out of me when it does hit those peaks.
Personally, I like having both. And I find myself rotating between the two quite a lot. If I had to pick one, I would pick Astra.
and have it write DMs to me to Julia so that he could send off the prompts using Fable for me. And I know that must be in a weird place because I will always find some way to get good code out of the anthropic models, even if I can't do it directly. Astra has more novel uses, and especially now that I'm down a hand, Astra's ability to like use my computer and get work done and benefit my life outside of code.
I almost feel like as a heavy user of computers, the $200 codex plan almost feels essential because it... If you use computers professionally and you make over $100 ,000 a year, you or your boss should be paying for the $200 a month plan just because it makes your ability to use your computer more effective. If you don't have that and you're saving up in order to pick the thing that writes the best code and you're really easily stressed, I would still pick Astra, but I would use Astra to make Astra better.
Find everything you can to smooth out the rough edges, put together the best set of skills in the world, and make benchmarks that prove why they're the best. blast it on Twitter, get a job at OpenAI, and never have to worry about token spend again. What I'm trying to say is I don't think that person actually exists.
A lot of you act like that person does. They don't. If you're concerned about costs and that's your main motivation, go get the open source plan.
See if you can convince either lab to give it to you or just wait for these things to be accessible in cheaper formats. They will be. GLM 5 .3 Flash is a more pleasant model to use, arguably than Astra.
I've seen GLM 5 .3 Flash go off the rails less than Astra for similar work. So I would expect this level of capability to be cheaper and more accessible in things in the near future. So just be patient if you're feeling like the price is too high.
I get it. That said, Fable is what I reach for when I'm trying to land code, and Astra is what I reach for when I'm trying to use my computer. The gap for code is not as huge as it might sound.
The gap is more in this chart here. But the harsh reality is just this gap in capability. This is the post I made when the models both came out to try and explain why I like both.
Astra is world class at a shitload of things. It is genuinely the best model in the world at all of this stuff and half of it it is far ahead for. But Fable writes code that is mergeable 20 -ish percent more often and its stupid spikes are much less stupid.
Its smart spikes are not quite as smart, but its stupid spikes are way less stupid. And for that reason alone, I think Fable 5 .1 is a better choice for day -to -day code stuff when cost is not a factor. But if you're willing to put the effort in with Astra, you will get a lot for it.
Both models are awesome. I will be defaulting to Fable for code, and I'll be defaulting to Astra for every other thing that I do. Actually, one last fun side thing that I should have mentioned before.
This will be a really silly one to end on. Remember everybody being really upset about Fable's security refusals? Astra refuses more often.
Just wanted to put that one out there. Anyways, both these models are great. We are in an unbelievable time to be engineers.
The fact that we have technology like this coming out every few months that massively changes how we work is so goddamn cool. It is hard to pick wrong here, but if you don't see the benefit of all of Astra's incredible capabilities, I am confused because they are useful to everyone. And if you don't think Fable's worth the extra money for the quality code difference where it's not that big a difference, I totally understand, but you're not shipping enough if you don't see why that matters.
Either model is great. They're both unbelievable. They both have changed how I work.
And when you look at the numbers for how we're using them, both models have massively improved our productivity within T3 code. And they don't replace each other. They compound each other.
These models are great and you shouldn't feel bad using one or the other. You should experiment with both because I think you'll be blown away with what's possible. and pretty much everyone who can have the codex sub probably should because it's such an insane deal have fun burning tokens and using all these models i think they're awesome and i bet you will too until next time peace nerds
The Hook
The bait, then the rug-pull.
Theo opens with a fake-out joke about models nobody asked about, then pivots to the real comparison: Fable 5.1 against GPT-6 Astra, the two frontier models he's actually running thousands of dollars of inference through every day.
Frameworks
Named ideas worth stealing.
01:21list
The four-category comparison rubric
Code
Non-code
Agent-y stuff
Cost
Theo's own framework for comparing frontier models across code quality, non-code creative/utility tasks, autonomous agent behavior, and real-world cost.
Steal forstructuring any head-to-head tool or vendor comparison
57:34concept
Cache read vs. cache write cost breakdown
Separates prompt-caching costs into cheap, shrinking cache reads versus expensive cache writes, showing writes now dominate real spend even as reads get cheaper.
Steal forauditing any LLM API bill before assuming a price cut actually saves money
1:10:14model
Quality-over-time volatility chart
A simple two-line chart plotting output quality across repeated attempts: one model holds a steady, higher floor and lower ceiling, the other swings between best-in-class and catastrophic.
Steal forcommunicating a tool's reliability tradeoff to a non-technical stakeholder
CTA Breakdown
How they asked for the click.
VERBAL ASK
01:38product
“powered by today's sponsor, Macroscope... at soydev.link slash Macroscope”
Integrated as a live walkthrough of his own repo's real review bots rather than a generic ad read, tied directly to the video's opening line about AI slop.
Add Modern Creator as a preferred source and Google shows you more of our breakdowns in Search, Top Stories, and AI Overviews. It only changes what you see, and you can undo it in your Google settings anytime.
Add to Preferred SourcesOpens your Google source preferences with us pre-loaded. Tick the box and you're done.
Theo says he barely codes hands-on anymore, then spends 49 minutes proving he still ships more than most full-time engineers by showing exactly how he runs dozens of AI agents at once.
Theo puts Meta's Claude Code clone, Muse Code powered by Muse Spark 1.2, through benchmarks, a codebase audit, a game rewrite, and a live integration test to see if the price is the whole story.
Jacob Coxon spent three years pretraining models at both companies before quitting Anthropic and calling the industry's safety race a hubristic gamble. Theo reads the thread, then checks it against OpenAI's own system-card admissions about its newest model.
02:24
03:21
05:17
05:47
07:12
09:07
12:00
12:58
13:55
16:07
16:48
18:44
19:14
20:11
21:13
22:34
23:32
24:29
25:38
26:25
27:22
27:57
29:03
30:52
32:23
33:08
34:06
36:01
36:59
37:56
38:54
39:51
40:49
42:44
43:42
44:40
45:37
46:35
48:30
49:28
50:25
51:23
52:21
53:18
54:16
56:11
57:09
58:07
59:04
60:02
60:40
62:55
63:52
64:50
65:48
66:45
67:43
69:38
70:36
71:33
72:31
73:29
74:26
76:22
49:22
44:42
30:09
24:43
26:33
28:36