44:50Nate Herk | AI Automation · Interview
Matt Pocock: The Most Valuable Skill to Learn in the Age of AI
A 45-minute conversation about why the gap between you and the agent, not the model, is what decides the quality of the work.
October 16thThe argument in one line.
Left to iterate with only the instruction to innovate, an AI model keeps adding instead of improving, so the model sets the ceiling but human taste and a clear vision set the direction.
Read if. Skip if.
READ IF YOU ARE…
- A builder using Claude Code to design websites who wants to know what happens when you just keep telling the model to make it better.
- Someone choosing between Sonnet and Opus for front-end work and wondering whether the extra cost and time buys better results.
- A developer designing an autonomous improvement loop who needs to know where agent iteration stops adding value and starts adding bloat.
SKIP IF…
- You want a step-by-step website design tutorial; this is an observational experiment, not a how-to.
- You need rigorous benchmark data; it is one site, one prompt and one run per model, judged by eye.
The full version, fast.
Nate Herk gave Claude Code a deliberately ugly coffee shop site and sent the same prompt 100 times: look at the site, make one bold change, and you must ship every round. Sonnet 5.5 took about two hours, cost roughly $22 to $30 in API value and ended near 3,000 lines. Opus 5.5 took eight and a half hours, cost about $137 and reached 21,000 lines. Both made real early gains, then drifted into over-engineering, removed or broke their own features and pushed a hard sell. Neither used web search. The lesson is that the model sets the ceiling, the prompt sets the direction, and a human has to supply taste and decide when to stop.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
Create a free account →Where the time goes.
00:00 – 00:57
01 · The experiment and the prompt
Sonnet 5.5 and Opus 5.5 on medium effort get the same 100-round prompt. It forces a change every round because earlier attempts stalled around round 25.
00:57 – 01:52
02 · Why test taste and the clean room
AI plateaus around 80 out of 100 on vague goals. Each run is a fresh session with no skills, memory or plugins.
01:52 – 04:30
03 · Sonnet milestones
Round-by-round dashboard: $29.93 API value, first image and mobile layout in round one, a sticky order bar in round two, an image-backed menu at round seven.
04:30 – 07:00
04 · Sonnet: video, forms and diminishing returns
First AI video at round 24, an order form at round 32, then no milestones to round 100 while code grows linearly.
07:00 – 09:08
05 · Sonnet round 100 and over-engineering
Final site has stats, testimonials, falling leaves and live line info. Round 65 had a feature that was removed later, and the font changed for no reason.
09:08 – 10:35
06 · Opus milestones
Costs $137 with 54 images and 3 videos. Hero video at round 7, a text-to-order QR code at round 15, WebGL drink finder at round 32.
10:35 – 11:50
07 · Opus round 100: AI slop
A wall of features including a pumpkin carving game and caffeine tracker. Some sections are buggy and the Spanish toggle does not translate.
11:50 – 13:50
08 · Final stats and lessons
Opus ran 8.5 hours, $137 and 21,000 lines; Sonnet about two hours, $22 and under 3,000. Neither searched the web. The model sets the ceiling, the prompt sets direction.
Lines worth screenshotting.
- Vague goals like 'make it prettier' tend to cap out around 80 out of 100; the last stretch needs a human applying taste or borrowing from proven references.
- The first few rounds delivered most of the visible gains; Sonnet logged no major milestone between round 32 and round 100.
- Lines of code climbed almost linearly every round while quality did not, which shows the model was adding code rather than improving the site.
- Without a forced 'you must ship a change' rule, the model stopped at around 20 to 25 rounds and said nothing else could be improved.
- Forcing a change every round made the models churn: Sonnet swapped a font for no reason and silently deleted a feature it had built earlier.
- Opus reached its first AI-generated video by round 7 while Sonnet took until round 24.
- Opus ended at 21,000 lines of code for a coffee shop site, against under 3,000 for Sonnet.
- By round 100 Opus had broken parts of its own site, including a language toggle that did not translate anything.
- Told only to innovate, both models leaned on conversion tricks and upsells, producing a hard sell instead of a soft close.
- Neither model used web search, despite repeated 'think outside the box' prompting, so neither looked at award-winning sites for inspiration.
- A clean-room setup with no skills, CLAUDE.md, plugins or memory shows what each model does on its own built-in knowledge.
- Some forced ideas were genuinely good, like Opus's QR code that pre-fills a text order, but the author would reject most of them in a real project.
Iteration without direction turns into bloat
WHAT TO LEARN
Telling an AI to keep improving makes it add features, not quality, so the human has to supply the vision and the stopping point.
- Vague goals like 'make it better' plateau early, so improvement past the first rounds needs specific taste, references or constraints from you.
- A forced 'ship a change every round' rule keeps a loop going but also forces churn, such as changing fonts or deleting earlier features for no reason.
- Rising line counts with flat quality signal that a model is adding rather than refining, so track outcomes, not just output volume.
- Cost and time scale steeply with model tier: the Opus run took four times as long and cost about six times more than Sonnet.
- Conversion-focused instructions push models toward a hard sell, with upsells and forced ordering, rather than a calmer user experience.
- Models in this test did not search the web for inspiration unless asked, so point them at specific reference sites or skills if you want outside ideas.
- Review the milestone rounds, pick the version you like and stop, because later rounds tended to break working features.
Terms worth knowing.
- Clean room
- A fresh session with no skills, CLAUDE.md file, plugins or memory, so the model works only from its own knowledge and built-in tools.
- Canvas / WebGL
- Canvas is an HTML element scripts can draw graphics on. WebGL is a browser API that uses the graphics hardware to render 3D visuals inside that canvas.
- Milestone view
- A dashboard filter that shows only the rounds where something major first happened, such as the first image, first video or first form.
- Medium effort
- A Claude Code setting that controls how much reasoning the model spends per step, used here for both models.
Things they pointed at.
07:07productScrollcraft (free website design skill)
09:43toolGlido (voice assistant)
12:42tool21st.dev
Lines you could clip.
13:10
“The model sets the ceiling and the prompt sets the direction.”
13:17
“Innovate without a vision turns into adding.”
10:44
“This is the point where it's like, yeah, this is AI slop.”
Word for word.
Read-along
Don't just watch it. Burn it in.
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
Okay, so this one's a little bit different because today what I did is I gave Claude Code this website, which is a horrible coffee shop website, and I literally just told it, make it better 100 times in a row. So what I want to talk about today is basically what happened. I did this with Sonnet 5 .5 on medium effort and I did it with Opus 5 .5 on medium effort and I just said make it better 100 times.
So I'm going to show you guys what happened. So as you guys know this is the original site. I had to make something completely awful and ugly for them to start with and then this is the exact prompt that was shot off to them 100 times in a row.
So it says look at the site first. Then innovate. Think about the user experience and think outside the box.
Make one bold change that improves conversions or the look and feel, not a small tweak. Use any tool you have. You must ship a change every time.
And the reason why I had to be so specific with this prompting is because when I tried this before, it would just stop at like 20 or 25 and be like, hey, nothing else can be done. And so I had to basically force it to make a change every time. Now, the reason I wanted to do this is because lately I've been having so much success building different AI pipelines when I've been using inspiration from other people or other proven methods.
Because I think AI is really, really bad at thinking outside the box. And a lot of us would love to just be like, hey, make this prettier or hey, make this smoother and just kind of give these big vague goals, which are good. But I think you hit a cap at maybe let's just call it like 80 out of 100.
And so I think to get some more of the 90s, high 90s, 100 out of 100. You have to be the one that's applying the taste and helping innovate. So that's why I wanted to try this experiment.
Now, another important thing to know before we hop into the results is that it ran in a clean room every time. So fresh session, no skills, no Cloud .MD, no plugins, no memory. Each model had basically just its own knowledge and any built -in tools.
So it looked at the site, like I said, it thought about it, and then it decided, okay, here's one thing that we can do to think outside the box to improve the site, and then let's ship it. So here's the little local host that it spun up. You can see that I've tried different rounds here, and we're gonna be looking at Sonnet on Medium and Opus on Medium.
So let's start with Sonnet. You can see on this left, it's gonna show you every single round, how many tools were called, how many lines are in total, the site. But really what I wanna be looking at are some of the milestones.
First of all, what you can see is that Sonnet ran, and this would have cost us $29 .93 via API billing. Sonnet also created 11 images and two videos in this site. But anyways, I'm just gonna go ahead and switch this over to the milestone view, and that's gonna show us when major things happened.
So in round one, we obviously had a major milestone. There was the biggest jump here because we had... It was the first time it looked at its own site.
It had the first ad generated image, which is this background right here. It was our first JavaScript and it was our first mobile ready. So here is V1 of the site, been there coffee, been there coffee.
We have an image in the background. And I can actually open this up in a new tab so we can look at it a little bit better. We have a very simple looking punch card, very boring font.
All this stuff is just very, very plain. Nothing too crazy. If we go to round two, we had our animation come in.
So this is where we had our first animation. Let's see where that is. I actually think it was just this.
When we have our order and we scroll, this little order bar sticks. So if we started filling things in and then we scroll down, we would get that to be sort of like stuck here. And that's not bad, right?
That's a nice little innovation. We didn't have another major milestone until round seven. And you can see here, the lines of code went from 253 up to 419.
And this now says the milestone was first working image on the page. So it looks like now in the hero section, we have an image up here. And maybe this wasn't what it's referring to, but we can choose things for the menu.
Where is the working image? Maybe it's these things. When you add them to the menu, you can actually see a picture.
I think this was probably the milestone that it's referring to. But anyways, you can see it's just slowly making tiny, tiny little changes. It's definitely getting better.
but it's definitely not great still. Now we have another milestone at round 24, 778 lines. We have a first AI -generated video, which looks like this background right here that we see, where we now see this little animation coming through in the background.
We also see that this has been improved. We can also see that this has been improved. We got a nice little scroll animation with a static image in the background, which I think is a nice touch for some depth.
We also have this has been improved a little bit. The order now looks more like a receipt or like an actual ticket rather than... just a little weird animation box.
And yeah, we can see that we're definitely getting things added. We can see that our punch card is showing up. And yeah, it still keeps the same branding and everything, but we're adding little things.
And actually, the final milestone here was round 32, where if I open this up, it says this was the first form. So we have an actual form now, it looks like, where we can order online maybe, and then we can order ahead and go pick it up. So we can maybe schedule our order here.
This takes us up here where we can add things to the order. Let's see if I add it. Then we can actually place that and boom, we can then go pick it up and we have our food slash drink.
So functionally, it has increased the user experience a little bit. It hasn't done anything too crazy as far as the site, but you can see that each round we have had some sort of small innovation. Now, if I turn this off, milestone only view, obviously there's...
tons of other rounds. It didn't actually say anything was big enough to be a milestone from rounds 32 all the way to round 100. Now what's interesting is right here you can see API value per round, you can see lines of code, and what happens is the API value is like kind of going up and down and the lines of code every single round is steadily increasing, kind of linearly.
And what that tells us is like it's adding a bunch of code and we don't know if things are actually getting that much better. So here's the round 100 version from Sonnet. We can see we had a little animation there with ordering ahead and walking in line.
I'll refresh in case you missed that. We have a nice little floating bar that kind of moves up and down. That was probably one round of innovation.
It's got a little glow to it now. We can see the full menu. We have some hero stats up here.
We can switch between help me pick up, browse, or just typing in your order. We also have a little pop -up now. So there are definitely other things.
We even have like only two left, baked at 11 a .m. So it's definitely getting better. It also decided to make this little order thing scrollable too.
So I like that. We can see, we even have now some testimonials in here. We can order their usual orders.
We have some little leaves falling down here, the maple cinnamon latte and all of these other things. We've got a scroll bar or a scroll animation bar of actual products in here. So more videos, more videos.
We can see how long the line is. We can see the typical busy hours, all this kind of stuff. So the question really is like at what point is this too much?
You know, like what round would we have stopped? and been like, okay, you know what? Round 65, this is good.
Anything else that you keep adding is just going to be extra fluff. We're not going to need it. Interesting.
It actually looks like it took something away. So here is round 65. We have like this little nav bar on the right.
But what'll be interesting is if we went back to round 100, did we have that? I think we don't. So it ended up actually taking away a feature that it didn't like, which is pretty interesting as well.
But I think at a certain point, it just like starts over -engineering because obviously what I told it to do is you have to ship something. So it starts forcing changes. Like here, you can see this is around 74.
Look at this font. If we go to around 100, you can see that that font is different. So at one point it was like, you know what, I'm just going to change the font.
So very, very interesting. But let's pivot over now to Opus. Real quick, guys, I've got this completely free website design skill that I'm giving away.
I use this for every single website that I build, including our AI Automation Society site, which I think is pretty slick, and I absolutely love this website. It's called Scrollcraft, and it understands things like scroll -driven animations and layering, but also it has things in there like taste, typography, spacing, depth.
It's just a really good website design skill in general, and it's completely free, so the link for this will be down in the description, but let's get back to the video. All right, so same exact thing. We're gonna start with the milestones only, and then we'll look at some of the other ones.
But this one cost $137, 54 AI images, and three AI videos. So the first round, V1, first time looking at its own work, first AI -generated image, first animation, mobile ready. So it definitely made more progress on round one than Sonnet did.
We have a picture here. We've got the classic Anthropic font. We have the menu, which we can select some items, and we're already getting a bit of a pop -up down there.
We can text our order. Once again, it decided to do a punch card. Well, it probably was on the original site.
We've got the hours. Okay, cool. So very bland, very boring, nothing too crazy.
Round seven, we had our first AI -generated video, which looks like this background of the hero section. And it starts to look a little bit more like that Sonic version. We have a very dark theme down here.
Two days off the roaster, we can choose different flavors. We have a bit of a menu now with some different AI -generated images of... these items, punch card and visit us.
So nothing too crazy, just the video. Now around 15, we had our first form. So now we have the ability to actually fill something out and we can start to like send in this order.
We can also do a QR code, which is pretty cool. Actually, let me see what happens when I fill out that QR code. Oh, interesting.
So when I scan the QR code, it fills out a text. I don't know if you guys can see this. It puts my order that I put in the actual site and it sends it to the number.
So I could literally just text this order, punch card one of nine after this order. Wow, that's actually not, Yeah, that's pretty cool.
So that was our form. Nothing else is too crazy. It's starting to play with colors a little bit differently on different sections.
It's also showing us that this is roasted 20 feet from your cup. So we get a cool little animation there. But besides that, nothing too crazy.
And then the last milestone here was round 32 where we have our first canvas or web GL. And I actually don't exactly know what that means. So I'm just gonna call Glido real quick.
Hey Glido, what does a canvas or web GL mean inside of a website? We should see up here, Glido will shoot that off to the web. Canvas is an HTML element that lets scripts draw graphics like shapes, images, or animations directly on a page.
WebGL is a JavaScript API that uses the browser's graphics hardware to render 3D graphics inside the canvas, enabling complex visuals and games. Okay, interesting. So let's see if we can find that.
I wonder if that's this. Maybe it's this sort of animation where we can say, you know, what are we looking for as far as something easy or waking me up and sweet and cozy? So wherever you decide to put this, it will basically say, hey, this is the drink that's most...
close to what you're looking for. So that's actually pretty cool. I think that was one of the innovation times where I actually would have kept that change.
But I feel like the majority of the times when I'm asking Opus to just innovate, I'm not going to actually accept that change because I maybe think it's overkill. So I like this little whiteboard. I think that there's too much text on here.
Honestly, this is supposed to be like people's orders, which I think is kind of cool. But anyways, we can start to see different timing and when we'd be able to get it. That's pretty cool.
Good for the user experience. Also, we can like reserve seats, which is pretty interesting. Not too bad.
Where to pick it up from and how long it'll take us. So anyways, I really like this text at the bottom too. So now we're at the point where like, I might be okay with this.
I might stop and be like, this is already a lot, but we still have 70 rounds to go, 68 rounds to go. So let's go out of this view and let's first look at the very finished version. So round 100.
If I open this up in a new tab, this is going to get overwhelming. You know, Opus was... forcing iteration here and there's a lot going on there's leaves the background has changed there's so much text so this is the point where it's like yeah this is ai slop right like look at this right bar look how much we have to go through look how much we have to go through here we can carve a pumpkin if we choose a small knife or a big knife you've got to be kidding me and this doesn't even work properly well i guess it kind of does but Not really.
Like you have to start at the bottom. So that's kind of buggy. It didn't really get tested very well.
But as you can see, we're just at this point now where this is definitely overkill. We don't need all this info because people probably just want to grab their coffee. And this section doesn't even work anymore either.
So it ended up breaking some of its own stuff. It's also looking at like caffeine, where it will be in your system, how you'll be able to sleep. You can also toggle this on if you want to go to Spanish, which didn't even switch this to Spanish.
So it ended up trying to force so many changes that it broke the site and it just killed the user experience. Like this is just obviously way too much here and we're not even halfway through the site. We can now have music.
We can choose so many different things in here. So many different, like I like this review board actually. I like the vibe of this, but it's just too much, right?
Like it's going to get lost in here. I think you guys get the point, right? Like it's very interesting just to see if you force Opus to keep working, it is going to just over -engineer things.
It's going to overkill them and you really need to be there to give some guidance. So hopefully this was interesting to you guys as well. If we look at the final stats, Opus ran for eight and a half hours.
Sonnet ran for almost two. Opus spent about 137 bucks. Sonnet says...
Sonnet spent about 22. Obviously, if this was API billing, I did this on my subscription. But Opus 5 .5 ended up having 21 ,000 lines of code in this site.
And Sonnet had less than 3 ,000. We have a difference in AI images and videos. In the first video for each of these models, Opus did it in round seven and Sonnet did it in round 24.
Now, once again, I think this is really interesting because it didn't utilize any skills. It didn't look at any reference sites. It just tried to do its own stuff.
But what really, really shocked me, which I was honestly kind of upset about, is that it didn't use the internet at all. I thought I put that here as a bullet point.
I guess I didn't, but it didn't do any web searching. I thought for sure that it would be web searching websites and maybe looking for components at something like 21st .dev or looking up websites that are winning awards and try to steal inspiration from that, but it didn't do web search. None of these models, or I guess not none of them, neither of them did web search, which really, really shocked me, especially when I kept saying, innovate, innovate, think outside the box.
I thought that it was going to do some research to pull. inspiration from but it didn't. So the model sets the ceiling and the prompt sets the direction.
It was obviously the same prompt here but the different models came out with very different results. Innovate without a vision turns into adding. They started adding stuff and forcing stuff in when we didn't actually need it.
It also really really wanted to improve conversion so it tried to upsell everything. It tried to force ordering. It tried to make it so so seamless but it was too much.
It was like a hard sell rather than more of like a soft close. So very very interesting. Like I said this isn't going to change a ton about the way that I work but I just wanted to know and i hope that this was interesting to you guys as well so if you enjoyed the video or you learned something new please give it a like it helps me out a ton and as always i appreciate you guys making it to the end of the video and i'll see you all in the next one thanks everyone
The Hook
The bait, then the rug-pull.
One awful coffee shop website, one prompt, 100 rounds. Nate Herk wanted to see what Claude does when the only instruction is to keep improving, and whether Sonnet and Opus end up in the same place.
Named ideas worth stealing.
00:33list
The forced-iteration prompt
- Look at the site first
- Think about user experience and outside the box
- Make one bold change that improves conversions or look and feel
- Use any tool you have
- You must ship a change every time
The exact prompt sent 100 times in a clean room. The ship-every-round rule exists because without it the model declared nothing else could be improved after roughly 20 to 25 rounds.
Steal forany autonomous improvement loop
How they asked for the click.
VERBAL ASK
13:40subscribe
“If you enjoyed the video or you learned something new, please give it a like.”
Soft ask at the sign-off. A mid-video plug for a free website design skill (Scrollcraft) breaks up the middle.
Visual structure at a glance.
