A walkthrough of a private, self-hosted system that runs every coding-agent task through the same process, so prompt and model changes can be measured instead of guessed at.
Posted
2 weeks ago
Duration
Format
Tutorial
educational
Views
53.6K
627 likes
57 · 43
Big Idea
The argument in one line.
A software factory turns ad-hoc AI coding agent usage into a repeatable, measurable process by running every task through the same prompts and infrastructure, so a change to a model, prompt, or workflow can be proven with data instead of guessed at.
Who This Is For
Read if. Skip if.
READ IF YOU ARE…
You already use coding agents like Claude Code or Codex regularly and want a way to know whether a prompt or model change actually helped.
You work on a team where different people use different models, prompts, and harnesses, and want a shared process everyone benefits from at once.
You want to queue up large batches of well-defined coding work to run unattended, then review the results later.
You're comfortable with infrastructure basics (SSH, virtual machines, config files) and want a concrete example of running agents on private cloud infra.
SKIP IF…
You're brand new to AI coding agents and haven't built a repeatable workflow with one yet.
Your work is mostly exploratory or design-driven, where you need to iterate live and look at results in a browser rather than delegate a defined task.
TL;DR
The full version, fast.
The video argues that ad-hoc use of coding agents makes it impossible to know whether a change to your model, prompt, or harness actually helped. The fix is a self-hosted 'software factory' (built with Machinist) with a control plane that queues and tracks work and a data plane of workers that delegate to Codex or Claude Code, run privately behind an SSH tunnel on a cloud VM. Every task runs through one 'foreman' coordinator prompt, so changes to that prompt apply consistently everywhere, and triggers (a repo health-check cron job, a GitHub label) feed work in automatically. Historical run data (task time, token counts) becomes an eval you can act on: reviewing it surfaced a concrete efficiency fix, which was shipped as a tracked GitHub issue and measured before being folded back into local, day-to-day agent use.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
States the problem: changing models, prompts, harnesses, or skills gives no reliable way to know if the change helped.
00:55 – 02:16
02 · Control plane vs data plane
Defines a software factory as infrastructure that runs a defined process every time; splits the architecture into a control plane (manages work) and a data plane of workers (delegate to agent harnesses).
02:16 – 03:12
03 · Running agents through an SSH tunnel
Explains running the factory on a VM with no public internet access, reached only via an SSH tunnel, for a solo developer's security needs.
03:12 – 04:04
04 · Inside the Machinist dashboard
Tours the control plane UI: completed and in-progress work, analytics showing 18-minute average task time and 15 million tokens used after one day.
04:04 – 04:57
05 · The foreman: one coordinator, defined sequence
Every task is handed to a foreman coordinator agent that works through a fixed sequence and delegates to sub-agents, which is how consistency is enforced.
04:57 – 05:50
06 · Triggers: repo health checks and GitHub issue intake
Two configured triggers: a cron job that scans the codebase with a cheap model and files bugs, and a GitHub trigger that watches for a label to pull work in.
05:51 – 07:14
07 · Inside the config: workers, models, evals
Code walkthrough of the SQLite-backed config, foreman and audit job prompt definitions, worker executor config (Codex/Claude), and deterministic automated checks.
07:15 – 08:38
08 · Cloud vs local: pricing Hetzner against Google Cloud
Compares hosting cost: a Google Cloud e2-medium ran about $32/month before disk, versus a cheaper Hetzner Ubuntu VM; notes you can also just run it locally.
08:38 – 10:28
09 · Using evals to find a token-efficiency win
Points an agent at the factory's own historical run data to find inefficiencies; surfaces the idea of replacing an agent loop with a deterministic Python script for a CI-monitoring task.
10:29 – 11:24
10 · Shipping the fix: GitHub issue #432
Turns the eval finding into a tracked, labeled GitHub issue ('bounded code-review loop'), lets the factory build and measure it before adopting it locally too.
11:24 – 13:11
11 · Solo dev vs team: why shared infrastructure compounds
Contrasts personal, siloed agent setups with a shared, version-controlled factory process; draws the CI/CD analogy for why shared improvements benefit everyone at once.
13:12 – 14:04
12 · When not to use a factory
Design and exploratory work should stay local and iterative; the factory is for queuing large batches of well-defined work reviewed later.
14:05 – 14:31
13 · Closing pitch: AI Engineer community
Points viewers to the aiengineer.co community for the deeper behind-the-scenes guide and further training.
Atomic Insights
Lines worth screenshotting.
Without a repeatable process, you can't tell whether changing your model, prompt, or coding harness actually improved anything.
A software factory splits into a control plane that queues and tracks work and a data plane of workers that execute it, so you can scale or change either half independently.
Running agent infrastructure on a VM with no public internet access, reached only through an SSH tunnel, keeps it private without needing dedicated security infrastructure.
One coordinator prompt (a 'foreman') delegating to sub-agents in a fixed sequence is what makes every task follow the same steps, which is the real source of consistency.
After one day of use, the system reported an average task time of 18 minutes and 15 million tokens used, turning workflow quality into a number you can track over time.
A cron trigger can scan a codebase with a cheap model on a schedule and file bugs as tickets, so repo health checks happen without anyone remembering to run them.
A GitHub label alone can pull an issue into an automated agent queue, no manual submission required.
A small Google Cloud VM (e2-medium) ran about $32 a month before disk costs, meaningfully more than an equivalent Hetzner instance, so the cloud provider you choose changes whether self-hosting this is worth it.
Feeding an agent your own historical run data and asking it to find inefficiencies is itself a use of the system: the factory can be pointed at analyzing its own output.
When an agent's job is simple and repeatable, like watching CI and reacting to it, swapping the agent loop for a deterministic script is often cheaper, more predictable, and easier to test.
A workflow improvement discovered through evals gets run through the same tracked process as any other task (a labeled GitHub issue) before anyone trusts it.
In a team, everyone running different models, prompts, and harnesses means no one can tell which combination is actually working, and no individual's improvement helps anyone else.
Shared agent infrastructure means one person's improvement to a shared prompt benefits every person on the team the moment it's merged, the same way a CI/CD change does.
A rigid, multi-step factory process is the wrong tool for design or exploratory work where you need to iterate live and look at results immediately.
Takeaway
Run every agent task the same way so you can measure what actually helps.
WHAT TO LEARN
A repeatable process around your coding agents, not the agents themselves, is what lets you prove a model, prompt, or workflow change is actually an improvement.
01Why measure your agentic coding
If you can't measure whether a new model, prompt, or harness actually improved your results, you're just guessing every time you change your setup.
Running the same task through a fixed process every time is what turns agentic coding from anecdote into data you can compare.
02Control plane vs data plane
Splitting the system into a control plane that manages work and a data plane of workers that execute it separates what needs doing from how it gets done, so you can scale either half independently.
The worker's job is to prep the environment and hand code changes to whichever agent harness is doing the actual coding, keeping that choice swappable.
03Running agents through an SSH tunnel
Running your coding agents on a VM with no public internet access, reached only through an SSH tunnel, keeps automated agent infrastructure private without needing a full security team.
A local forward through SSH lets you use a browser-based control-plane UI without ever exposing that UI to the open internet.
04Inside the Machinist dashboard
Tracking average task time and total tokens per run turns 'is this working' into a number you can watch move over time.
A dashboard listing every completed and in-progress task gives you a paper trail for changes you make to prompts or models later.
05The foreman: one coordinator, defined sequence
Routing every task through one coordinator prompt that delegates to sub-agents in a fixed sequence is what makes every task go through the exact same steps, which is the real source of consistency.
When you edit that one coordinator prompt, every future task uses the new version automatically, so improvements compound instead of living in one person's local config.
06Triggers: repo health checks and GitHub issue intake
A cron-style trigger that scans your codebase with a cheap model and files bugs as tickets turns repo maintenance into something that happens without anyone remembering to do it.
A GitHub label can be the on-ramp into an automated system: tag an issue, and the factory pulls it into the queue on its own.
07Inside the config: workers, models, evals
Defining your coding executors in one config file means you can swap models or agents without touching the workflow logic.
Writing deterministic, automated checks alongside your agent tasks catches process drift that a human reviewer would eventually have to catch by hand.
08Cloud vs local: pricing Hetzner against Google Cloud
A cloud VM to host this kind of infrastructure isn't free: pick the provider carefully, since the difference between options changed the monthly cost several times over.
You don't have to run agent infrastructure in the cloud. Running it locally works too; the cloud version mainly helps separate your day-to-day environment from where agents operate.
09Using evals to find a token-efficiency win
Pointing an agent at your own historical run data and asking it to find inefficiencies is itself a use of the system: the factory can analyze its own output.
When an agent loop's job is simple and repeatable, swapping it for a deterministic script is often cheaper, more predictable, and easier to test than paying for another agent loop.
10Shipping the fix: GitHub issue #432
A change identified through evals gets turned into a normal tracked task and run through the same process used for everything else, so the improvement itself is measured before you trust it.
Before a workflow change gets adopted everywhere, prove it moves the numbers on the system where you can measure it, then port the same change into your everyday local setup.
11Solo dev vs team: why shared infrastructure compounds
In a team, everyone using different models, prompts, and harnesses means there's no way to know which combination is actually working, and no way for one person's improvement to help anyone else.
Shared infrastructure is what makes an improvement to your process benefit every person on the team at once, the same way a CI/CD change benefits everyone, not just the person who wrote it.
12When not to use a factory
For design or exploratory work, where you need to iterate fast and look at the result immediately, a rigid multi-step factory process gets in the way rather than helping.
The strongest use case for this kind of system is queuing large batches of well-defined work you review later, not back-and-forth thinking-out-loud work.
Glossary
Terms worth knowing.
Software factory
Infrastructure for running coding agents through a fixed, repeatable process every time, recording metrics about each run so changes to that process can be measured.
Control plane
The part of the system that manages and queues work, holds shared prompt/agent definitions, and stores the history of every run.
Data plane
The workers that actually execute queued tasks by delegating to a coding agent harness, setting up environments, and reporting results back to the control plane.
Worker
A process that pulls a task off the control plane's queue and hands it to a coding agent harness, then sends the results back once the work is done.
Agent harness
The specific coding agent doing the work, such as Claude Code or Codex, which the worker delegates to but doesn't itself implement.
Foreman
A single coordinator prompt that every task in the factory runs through, responsible for working through the task in sequence and delegating to sub-agents.
Eval
A structured check, or a review of historical run data, used to measure whether a change to a model, prompt, or workflow actually made a difference.
Trigger
An automated way of getting work into the factory, such as a scheduled repository health check or a GitHub label that queues an issue.
SSH tunnel
A forwarded local connection to a private virtual machine, used here to reach a browser-based control plane UI without exposing it on the public internet.
Machinist
The open-source software factory system used throughout the video to run and manage coding agent tasks.
“This could save you a ton of money, make your process more reliable.”
punchy payoff line right after the fix is found→ TikTok hook↗ Tweet quote
12:17
“Any change you make to your software factory configuration benefits every single person in the team.”
the team-leverage thesis in one sentence→ IG reel cold open↗ Tweet quote
The Script
Word for word.
Read-along
Don't just watch it. Burn it in.
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
17px
analogy
Today we're talking about software factories and how they can make your agentic coding more consistent, more measurable and more scalable. This isn't going to be one of those hype videos where I tell you to stop using coding agents locally because I think that would be terrible advice. I'm still using local coding agents for a lot of my work but I think there are many advantages of using a software factory.
One of the big challenges with agentic coding is that when you change your models, your prompts, your harnesses or your skills, it's not very easy to know whether those things make any valuable difference. So the advantage of a software factory is it gives you a repeatable way of measuring and improving your process over time.
So in this video, I'm going to break down a software factory that I built. I'm going to show you how to set it up and configure it. I'm going to show you the architecture of a software factory.
And more importantly, we're going to talk about how you can use evaluations and software factories to improve your coding process over time. You can make some drastic improvements in terms of your own efficiency and learn a lot about your own agentic coding process by following some of the ideas in this video. this is going to be a big topic so all of the resources and everything you need to get started i'll link for free in the description below so let's get into it i personally think about a software factory as infrastructure for running coding agents through a defined process so what i mean by that is you put work into the system and it will run through a defined process the same way every single time and you record metrics and data about that that particular run so this allows you to see everything that's going on to manage coding without having to think about, you know, work trees and branches and how to set up the local environment.
You just delegate the work and the factory or the system will take care of it for you. I think about two parts of the architecture, the data plane and the control plane. So the control plane is where you manage the work.
And then the data plane is made up of the workers. The workers are essentially delegating to coding agents. They're setting up your environments.
They're doing the work. So the useful distinction really is between the worker and the agent harness. So the worker lives on the data plane.
You could have multiple workers across multiple different virtual machines. You could run this on a Kubernetes cluster if you wanted to. So you admit work to the control plane, which will queue up the work.
And then the workers are just going to pull tasks off the queue and then delegate those to the right coding agents and make sure the work gets done. And then the workers are going to record the results of that work and then send it back to the control plane so that you can actually see what's going on in a cycle. So the way that I personally like to set this up is using an SSH tunnel because I'm just a solo developer.
I don't need to deal with any security headaches or complex infrastructure setups. I just put this onto a virtual machine. I run it on the virtual machine, but then I use an SSH tunnel to forward to access this machine.
So essentially what this does is we can run our software factory and our agents on a virtual machine with no public internet access. So we can have these completely secure and private. But then we can use an SSH tunnel to connect to them on our local machine.
So I'll show you how that works now. So I have this deployed on a virtual machine. So I can go onto the Machinist virtual machine.
And this is just an Ubuntu instance. And then if I type the Machinist command, you can see here we have a CLI tool that has a bunch of different options. We can run our workers straight away.
You can see here I'm signed in with my subscription, but the agents are running on a virtual machine, not on my laptop. So when I run this command, it will allow me to go to my browser and now access the UI. So this is the machinist UI.
You can see here we have a bunch of work that's been completed, a bunch of work in progress. You can see a list of all of the stuff that we've been working on. You can see analytics here about what's going on.
So you can see the average task time is 18 minutes and we have 15 million tokens used so far. I only set this up yesterday. And we have workers.
You could have multiple different workers. You could run the workers on cloud instances, Kubernetes jobs, however you want to set this up. I only have one worker right now, which is running on the same virtual machine as the control plane.
This is connected and the worker will then just take the tasks and dispatch them to coding agents. If you're working on many things at once or you're working in a big team, you might want to have more workers added to the pool. And then you can see here the prompts that we're running.
These are the actual agent definitions. So every single task when we send a task through Machinist is given to a foreman coordinator. The coordinator will be responsible for taking the task and then basically working through it in a sequence, delegating the work to coding agents or sub -agents.
The goal here is to have this one agent spin up sub -agents, implement the code change and follow a defined sequence. So even down to how we manage labels, it's all configured here, it's all defined. But this is the repeatable process that every single coding task will go through in the system.
So as you make changes to this specific prompt, every single task that you do will be using the same prompt every single time. So this is how you build the consistency. I have two triggers set up.
You can define triggers within the factory. So this first trigger will monitor the repository. So every so often it will run a health check.
It will scan the code base using a cheaper model, find bugs, find issues in the code base and raise tickets. And then the other trigger that we have is a GitHub watching trigger. You can configure all kinds of different triggers within your repositories.
And this will then monitor for a specific label on a task. When the agent sees that there's a particular label, it will then pull that work into the factory.
So it's a way of getting work into the system. And so the nice thing about having a factory system like this is you get a historical record of all of the work that you've been doing. And so you can see as you start to change your models and you start to change your prompts, you can see the difference it makes on the system overall.
So if you want to make a change or use a different model, you can run that for a couple of days, collect a bunch of data, and then start to see what difference it makes in terms of the cycle time. how effectively the tasks are being completed. So this is kind of a way of experimenting a lot with your coding setup.
So just quickly within the code base, you can see here that we have our configuration. So let's go ahead and take a look at some examples. So this is the configuration for the server.
So you can see here we have the database, which is just using SQLite. So we have the max concurrent jobs. Then we have a foreman job or a foreman agent.
And this is a prompt definition. We're using Codex. And then we have a audit job as well.
Again, these are just prompt definitions. And if we go into the worker configuration, you can see here we're configuring the executors. So we're using Codex or Claude, and we've defined the models that are available to the executor.
You can build in any executor here. You could have multiple different coding executors, and you can start experimenting then with different setups to see what makes a difference. This is the foreman prompt.
This is the prompt that every single task inside the system is going to be running through. It controls the workflow and it delegates work to sub -agents. So we're just working on a single GitHub issue.
I also have a bunch of somewhat deterministic evals as well. So I can do automated checking and automated testing to make sure that the right labels, the right workflow is being followed. So you can also define a bunch of automated tests as well in the spirit of like evaluating and improving your system over time.
So when it comes to software factories, you can either run your infrastructure locally or you can run it in the cloud. I've chosen to run my infrastructure in the cloud. So both the control plane and the workers are running on virtual machines.
You could also run this using things like Kubernetes containers. You can run these using cloud run jobs. You can run this in a million different ways.
I like to keep it simple. And because it's just me working on this, I'm using a simple virtual machine setup. If you're working in a team environment, you might do this differently.
Taking a look at the pricing across all of these different providers, there are a whole bunch of options when it comes to this. i personally use google cloud for almost all of my professional work but i find that the virtual machines are pretty expensive you can see here even for like an e2 medium in london is 32 a month plus the disk usage so this starts to add up quite quickly whereas if you use a hetsna virtual machine it's significantly cheaper so you can see here i'm now running an ubuntu 16 gig in helsinki so this is just a cheap -ish virtual machine like i said If cost is a concern, you can just run this locally.
You don't need to run it in the cloud, but I find running these agents in the cloud helps me think better about the infrastructure. It helps me kind of separate out my local development environment from the infrastructure where I'm running my coding agents. So Google Cloud is where I run most of my infrastructure, but I find that the virtual machines are relatively expensive when compared with something like Hetzner, where you can get a bigger virtual machine for much less money.
The next thing I want to show you is the idea of evals. And this is kind of where I started with this whole experiment. The idea that I want to start improving my process over time and experiment with different models and see with data what's actually working.
You can obviously do this with your local coding setups. So all of your coding agents will emit logs or JSON files containing a bunch of metadata that an agent can interpret. The difference here with a factory is that we're using the same prompt every single time.
So every single task that runs through our factory goes through exactly the same series of steps. So this was a very simple example. I just said, can you connect to the remote virtual machine instance, review the outputs of the latest coding runs, find ways to improve our prompts to make it more efficient in terms of token usage.
And so the agent then would SSH into the machine. It's reviewed all of the output and the data in the database. And you can see here, we have a bunch of interesting data.
So you can see here, the time is taken for these particular issues. You can see here the number of tokens used, the number of commands run, the number of agent weights as well. So what I'm thinking now is I might actually experiment with using more deterministic Python scripts.
So within the machinist setup, you can run an agent, but you can also run a script. And so what I might do is experiment with something like this, where we have essentially a deterministic script that will monitor CI, delegate to an agent, and this might be a lot more token efficient generally when working with agents. So by moving this to a script it's going to be cheaper, more predictable, more testable and gives the same structured output every time and so I think this is something that I'm going to do next.
So this is useful feedback just by analyzing our runs looking for ways to improve. Now what I can do is go ahead and write a script to improve. So I'm going to go ahead and cut a task to do this and then I'm just going to go and try experimenting with this now over time.
So now we found a very specific improvement to my workflow. And this would improve everything about my workflow, to be honest, because I'm using the same coding loop on every single task every single day. So if you could imagine that implementing this one change to my workflow could save me a significant number of tokens every single time I use a coding agent, this is going to be a massive improvement to my overall workflow.
So you could see that this could save you a ton of money, make your process more reliable. having this ability to do this in a kind of like a repeated way is super useful. So what I'm going to do now is add a label as before.
So we're going to add a machinist queued label or requested label, sorry. And then we're just going to basically get the agents to go and build this for us and then start to measure and experiment and see if this makes an improvement to our overall workflow. If this does make an improvement to the workflow on the factory instance, I will then also incorporate the same code.
the same workflow on my local development setup as well, just day to day. So when it comes to software development, there's a big difference between working by yourself as a solo developer or working in a team environment. So I personally work both as a solo developer, I run my own AI business, I do a lot of side projects, but I also work in a large enterprise environment with hundreds or thousands of developers.
And there's a big difference between those two situations. In a team environment, I think there's a lot of benefits from having shared infrastructure where everyone is using the same workflows and systems to implement code changes. That way, as a team, collectively, you can make changes to your setup, your prompts, your configuration, and those changes benefit every single person in the team.
This is far better, in my opinion, than every single developer in the team. Using different models, using different harnesses, using different prompts, different skills, there's no way to kind of have consistency across the team in terms of the workflow. or get any kind of repeatability.
And it's just very hard to make improvements in that situation. With a software factory, everybody is using the same process, any change you make to your software factory configuration benefits every single person in the team. And this is the same going back to the analogy about CICD, it's the same idea that the system can be improved, and it benefits everybody, so that you have massive, massive leverage compared to individual people.
doing everything completely differently with different agent harnesses and different skills, different prompting abilities, different techniques, etc. I much prefer this idea of a common systems approach to implementing software. So I personally found it really valuable to be using a software factory.
I think this idea of experimenting and trying out different techniques and seeing what works and then improving your process is probably the most valuable part of the software factory process, as well as having a very repeatable and reliable way of working with these coding agents. So I'm going to be using this in my own workflow.
But like I said, you don't need to use a software factory for everything you do. If you're working on design or iterative work, it doesn't make sense to me to be using a software factory because you need to stay close to the code. You want to iterate.
You want to go back and forth. You want to see things in the browser and move very, very fast. So I don't think that's a good use case.
I think I'm still going to be using coding agents locally, but in conjunction with a software factory. So I'm going to be using software factories for things like queuing up large amounts of work that happen overnight. And then just delegating those and sequencing them and making sure they run repeatedly and get done.
And then in the morning, I can review those tasks. But any kind of work where I need to do experimentation, you know, I need to talk to an agent, think through a problem, anything like that, then I'm going to still use a local coding setup. I hope you found the video useful.
If you did, please remember to like and subscribe. It massively helps out the channel. And I'll see you in the next one.
Take care. If you want to go deeper on building AI systems, using agentic coding tools, leveling up your career as a consultant, as a freelancer, or inside a company, I cover all of those things inside my community. I've been a software engineer for the last 20 years.
I've been both an independent AI engineer, I've worked as a director of engineering in a large organization, and I also freelance and build software every single day. So if you need help with any of these things, you can check out the link in the description below.
The Hook
The bait, then the rug-pull.
He opens by ruling out the obvious hot take: this isn't a video telling you to stop using coding agents locally. It's about a problem anyone running Claude Code or Codex day to day will recognize: when you tweak a model, a prompt, or a skill, you have no real way of knowing whether it helped.
Frameworks
Named ideas worth stealing.
01:16model
Control plane / data plane
Control plane: queue, shared definitions, run history
Data plane: workers, delegating to agent harnesses
The core architectural split: a control plane manages and records work while a data plane of workers actually executes it by delegating to coding agents.
Steal forany system that queues AI agent work across more than one machine
12:56model
Personal snowflakes vs shared factory
Personal snowflakes: personal prompts, custom skills, model choices, learning stays on individual laptops
Shared factory: Observe, Improve, Measure, Roll out
Individual developers experimenting alone produce learning that never leaves their machine. A shared, version-controlled factory turns local experiments into an Observe/Improve/Measure/Roll-out loop that benefits the whole team.
Steal forjustifying shared tooling investment to a team
CTA Breakdown
How they asked for the click.
VERBAL ASK
14:05newsletter
“If you want to go deeper on building AI systems, using agentic coding tools, leveling up your career as a consultant, as a freelancer, or inside a company, I cover all of those things inside my community... you can check out the link in the description below.”
Soft pitch at the very end, after the content is fully delivered, pointing to a paid community (aiengineer.co) rather than interrupting the tutorial.
Add Modern Creator as a preferred source and Google shows you more of our breakdowns in Search, Top Stories, and AI Overviews. It only changes what you see, and you can undo it in your Google settings anytime.
Add to Preferred SourcesOpens your Google source preferences with us pre-loaded. Tick the box and you're done.
The same update that let Claude Code and Codex turn a screen recording into a skill quietly taught both of them to watch raw video, no plugin required.
Theo runs OpenAI's GPT-5.6-Sol through Claude Code instead of Codex and gets visibly better designs and cheaper orchestration — then reads Codex's system prompt on camera to find out why.