24:23Mark Kashef · Tutorial
Master All 5 Layers of Every Agentic OS
A 24-minute Earth-layers framework for building AI operating systems that don't decay.
June 30thThe argument in one line.
Open-weight AI models running on hardware you own are now good enough for most day-to-day work, and the fastest way to set one up is to point a closed-source coding agent like Claude Code at the task and let it build, configure, and maintain the open-source stack for you.
Read if. Skip if.
READ IF YOU ARE…
- You've relied on Claude or ChatGPT for months and are starting to feel exposed to a single company's pricing and availability decisions.
- You have a spare machine — a Mac Mini, old laptop, or GPU box — sitting around and want a legitimate reason to put it to work.
- You're comfortable pasting commands into a terminal but have never touched Ollama, llama.cpp, or Hugging Face.
- You already use Claude Code or Codex daily and want to point that same harness at a free or cheaper open-weight model for routine tasks.
SKIP IF…
- You need bleeding-edge model quality right now — the video is upfront that open source still trails frontier closed models.
- You have zero interest in touching a terminal or reading hardware specs; this is a hands-on technical setup, not a one-click app.
The full version, fast.
The video's argument: relying entirely on closed-source AI carries real vendor risk — models can vanish or get quietly nerfed overnight, and today's subsidized subscription pricing won't last as costs catch up to the venture capital funding it. The fix isn't abandoning Claude or GPT, it's adding a local backup. The stack breaks into five layers: hardware, open-weight models, an inference engine (llama.cpp or vLLM), a gateway (LiteLLM) that lets any app swap models without rewiring, and an interface (Open WebUI) plus optional agent harnesses. The unlock is using Claude Code or Codex to research your hardware limits, download and configure the right quantized model, wire up Tailscale for remote access, and even build a full local dashboard — turning a historically painful setup into one guided conversation.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
Create a free account →Where the time goes.
00:00 – 01:42
01 · The two AI scares
Anthropic pulled public access to Claude Fable 5 and Mythos 5 overnight, and Kashef frames that outage as the reason to have an open-weight backup.
01:42 – 02:01
02 · The 3 sections of this course
Kashef lays out the video's structure: key concepts, picking the right model and tools, then building an open-source command center.
02:01 – 05:00
03 · My local AI command center (demo)
A tour of Kashef's own dashboard running entirely on a Mac Mini: AI chat, a Hermes agent, document Q&A/RAG, image generation, and Tailscale-based remote access.
05:00 – 05:56
04 · The fake war between closed and open source
Kashef rejects the 'pick a side' framing — he uses closed models heavily but argues open source is essential as a redundancy plan.
05:56 – 08:42
05 · 4 reasons to run AI locally
Vendor risk (models can vanish), the closed-source subsidy ending, data privacy for sensitive work like personal DNA analysis, and cheap 24/7 ambient agents.
08:42 – 09:39
06 · "But local models were toys"
Acknowledges that Ollama-era models like Llama 2 disappointed people, but argues today's models plus agent harnesses have changed that.
09:39 – 12:52
07 · The 5-layer stack (most important diagram)
The core framework: hardware, open-weight models, inference engine (llama.cpp/vLLM), gateway (LiteLLM), and interface/harness — the mental model for the rest of the video.
12:52 – 15:45
08 · Find the right models with OpenRouter
Uses Claude Code to pull OpenRouter's public rankings API and categorize the last six months of open models into text-only vs multimodal.
15:45 – 18:20
09 · What can YOUR computer run
Feeds hardware specs (his M5 Max, 128GB unified memory) to Claude Code to get a shortlist of models that will actually run at usable speed.
18:20 – 21:32
10 · Downloading models from Hugging Face
Claude Code checks Hugging Face for a downloadable, Mac-tailored quantized build, installs prerequisites, and gets a 65GB model serving at 60 tokens/sec.
21:32 – 22:05
11 · Same workflow on a Mac Mini over SSH
Repeats the exact same hardware-check-and-install prompt over SSH on a headless Mac Mini instead of the local laptop.
22:05 – 23:13
12 · llama.cpp vs vLLM in plain English
llama.cpp is 'cooking for a party of one' (single-user local inference); vLLM is the industrial-kitchen version built to serve many concurrent users.
23:13 – 24:24
13 · LiteLLM, the switchboard
LiteLLM exposes an OpenAI-compatible API so every app talks to one gateway, and swapping the underlying model is just changing a config value.
24:24 – 26:33
14 · Tailscale, your private mesh network
A private encrypted mesh (free personal plan, up to 6 users/unlimited devices) that lets any device reach the home AI rig without port-forwarding.
26:33 – 28:26
15 · The best open source models right now
A named roster: Gemma (Google, sometimes overly restrictive), the Chinese labs (Qwen, GLM, Kimi) which Kashef says now outperform expectations, DeepSeek, and MiniMax for voice.
28:26 – 31:55
16 · Using Claude Code's harness with open models
Wires OpenRouter into Claude Code so a 'Claude OS' alias can run open models (Kimi K3, GLM 4.6) through Claude Code's own tool-calling harness.
31:55 – 33:38
17 · Parameters, what the B actually means
Parameter count is a rough proxy for capability; the rule of thumb is divide the parameter count by roughly two to estimate required RAM in GB.
33:38 – 34:56
18 · Quantization, JPEG for brains
Compressing a model's weights (16-bit down to 4-bit and below) trades fidelity for size and speed, the same way image resolution degrades.
34:56 – 35:57
19 · GGUF vs MLX
GGUF is the universal, cross-platform format; MLX is Apple Silicon-only but performs better on that specific hardware.
35:57 – 37:49
20 · KV cache, why chats slow down
Long conversations accumulate context that has to be re-processed each turn; shrinking the system prompt, trimming context, and cache quantization all help.
37:49 – 38:20
21 · Sparse attention
Instead of reading every token of a conversation every time, sparse attention selectively picks the most relevant past tokens, saving compute.
38:20 – 39:13
22 · Tokens per second
Generation speed is bottlenecked by memory bandwidth, not raw compute; smaller quantized weights move through that 'pipe' faster.
39:13 – 40:47
23 · Harnesses and pi.dev
A harness is what turns a model into an agent that can read/write files and run bash; pi.dev is a minimal open-source harness alternative to Claude Code/Codex.
40:47 – 43:34
24 · Build your command center in one command (ODS)
Feeds Ahmed Osman's open-source ODS/Osmentic GitHub repo to Claude Code, which fans out sub-agents to install and configure the entire local dashboard in about 45 minutes.
43:34 – 44:41
25 · Wrap-up + your free resources
Closes over the finished personal dashboard with a pitch for the free Local AI Engineering Guide plus his paid Early AI Adopters community.
Lines worth screenshotting.
- A rule of thumb for local AI: divide a model's parameter count by two to estimate the RAM in gigabytes you need to run it.
- Kimi K3's full model has roughly 2.8 trillion parameters, and running it at home could cost over $10,000 a month in electricity alone.
- Anthropic's $100-$200 monthly plans have been benchmarked as equivalent to $6,000-$8,000 of pay-as-you-go Claude usage, and OpenAI's equivalent plans to $13,000-$15,000 of usage.
- Pruning an oversized agent system prompt took one local model's response time from two minutes down to thirty seconds.
- GGUF works on any hardware; MLX only pays off on Apple Silicon — downloading MLX on a non-Apple machine gets you nothing.
- vLLM exists to serve one model to many simultaneous users; llama.cpp is built for a single user on their own machine.
- Below roughly 1-bit quantization, a compressed model loses so much fidelity that even a strong agent harness can't compensate for it.
- LiteLLM's whole value is that swapping one local model for another only means changing a config string — every app that talks to it stays the same.
- Tailscale's free personal plan supports unlimited devices and up to six users, enough mesh networking for most solo builders.
- Speed on a local model is a function of memory bandwidth, not raw compute — a wide memory pipe with small, quantized weights is what actually produces fast tokens per second.
- The RAM gap between a 14B and a 70B parameter model is roughly 5-7GB versus 35-40GB to run comfortably.
- Sparse attention lets a model selectively pick which past tokens matter instead of re-reading the entire conversation on every turn, which is what keeps long chats fast.
- OpenRouter's public rankings API can be fed to a coding agent, which will cross-reference your hardware specs against the leaderboard to shortlist runnable models.
- Pointing Claude Code or Codex at an open-source project's GitHub repo and letting it fan out sub-agents can fully configure a local AI dashboard in about 45 minutes.
- Using an OpenRouter API key inside Claude Code or Codex lets you borrow a closed-source coding harness while running an open-source model underneath it, so you can trial a model before committing to downloading and hosting it.
Own a backup AI stack you control
LOCAL AI STACK
Running open-weight models locally is now a realistic backup plan, and the fastest path to owning one is pointing a closed-source coding agent at the setup instead of doing it by hand.
04The fake war between closed and open source
- Treating open source vs. closed source as a war to pick a winner in is a false binary — the strongest setup uses both for different jobs.
054 reasons to run AI locally
- Vendor risk is real: a single directive pulled public access to two frontier models overnight, and the versions that came back were noticeably worse than before.
- The subscription pricing on closed models is subsidized by venture capital rather than sustainable margins, so expect prices to rise once those companies need to turn a profit.
- Local models are the practical choice for sensitive data — process something like a full DNA analysis on-device so it's never sent to a third party.
06"But local models were toys"
- Models that felt like toys a year or two ago (Llama 2, early Mistral) are meaningfully better now, and pairing them with an agent harness closes even more of the gap.
07The 5-layer stack (most important diagram)
- Every local AI setup breaks into five layers: hardware, open-weight model, inference engine (llama.cpp or vLLM), a gateway (LiteLLM), and an interface/harness on top — map any project to this stack before buying anything.
- Open weight means the actual model file is downloadable and inspectable; closed models like Claude and GPT never expose those weights to you.
08Find the right models with OpenRouter
- OpenRouter's public model-rankings API can be handed to a coding agent as a data source, letting it build a categorized, up-to-date shortlist of the newest open models instead of you researching it manually.
09What can YOUR computer run
- Feed your exact hardware specs (chip, unified memory, GPU) to a coding agent and ask it to cross-reference the model list — it will tell you which models actually run at a usable speed, not just which ones technically fit.
- Don't assume all of your machine's advertised memory is usable for a model — other running apps eat into that headroom, so budget well under the sticker RAM number.
10Downloading models from Hugging Face
- Hugging Face is the download repository for open models, including quantized and fine-tuned variants, while OpenRouter only lets you query and serve, not download, the underlying weights.
- A capable coding agent can install missing prerequisites (Ollama, a Python version, a serving library) on its own mid-setup, without you needing to know what any of the errors meant.
11Same workflow on a Mac Mini over SSH
- The exact same 'check my hardware, recommend and install a model' prompt works identically over SSH on a headless machine, so one workflow covers both a laptop and a remote server.
12llama.cpp vs vLLM in plain English
- llama.cpp is built for a single user on their own hardware; vLLM exists to serve the same models to many concurrent users at scale — pick based on whether you're the only one hitting it.
13LiteLLM, the switchboard
- A gateway like LiteLLM exposes one OpenAI-compatible API in front of every model you run, so switching from Qwen to GLM is a config change, not a rewire of every app that talks to it.
14Tailscale, your private mesh network
- Tailscale's free personal plan supports unlimited devices and up to six users, enough to mesh a laptop, a phone, and a home server without touching port-forwarding or a public-facing server.
15The best open source models right now
- As of mid-2026 the Chinese labs — Qwen, GLM, Kimi — are the strongest general-purpose open-weight options, with MiniMax standing out specifically for voice/TTS work.
- Quantized versions of huge frontier-scale open models keep getting cheaper to host even as full-precision hosting costs rise.
16Using Claude Code's harness with open models
- You can wire an OpenRouter API key into Claude Code or Codex and alias it to an open model, letting you test-drive a model's real-world performance inside a familiar coding harness before committing to downloading and hosting it yourself.
17Parameters, what the B actually means
- Parameter count (the 'B' in a model name) is a rough proxy for capability, and a quick rule of thumb is: divide the parameter count by about two to estimate the RAM in gigabytes you'll need.
18Quantization, JPEG for brains
- Quantization is compression for a model's weights — 16-bit down to 8-bit, 4-bit, and lower — and once you go below roughly 1-bit, the model's tool-use ability breaks down even with a strong harness.
19GGUF vs MLX
- GGUF is the cross-platform format that works with llama.cpp on virtually anything; MLX is Apple-Silicon-specific and performs better, but only on Apple hardware — there's no reason to seek out MLX if you're not on a Mac.
20KV cache, why chats slow down
- A bloated system prompt is a common hidden cause of slow local-agent responses — pruning one oversized prompt cut a response time from two minutes to thirty seconds.
- Three practical ways to manage a growing KV cache: shorten the agent's system prompt, keep individual turns/context small, and periodically summarize or quantize the accumulated conversation history.
21Sparse attention
- Sparse attention lets a model selectively decide which earlier tokens are worth re-reading instead of processing the entire conversation history on every single turn, which is what keeps long chats fast.
22Tokens per second
- Generation speed is bottlenecked by memory bandwidth, not raw compute — think of it as a pipe: smaller, more compressed weights move through it faster, which is why quantization directly improves tokens-per-second.
23Harnesses and pi.dev
- A harness is what turns a static model into something that can read, write, and manipulate files — the tool-calling scaffolding around the model is doing more of the work than people give it credit for.
- Open-source harnesses like pi.dev exist as alternatives to Claude Code or Codex, but expect to keep tuning them yourself as new models come out with different tool-calling habits — that maintenance is done for you on the closed-source side.
24Build your command center in one command (ODS)
- The fastest path to a fully-configured local AI dashboard isn't following a manual setup guide line-by-line — it's handing the project's GitHub repo to a coding agent and letting it fan out sub-agents to read the code, tailor the install to your specific hardware, and execute the whole setup.
Terms worth knowing.
- Open weight model
- A model whose trained parameter file is publicly downloadable, so you can inspect and run it yourself — unlike closed models like Claude or GPT, where the weights are never released.
- Quantization
- Compressing a model's weights (e.g. from 16-bit down to 8-bit or 4-bit) to shrink its size and speed it up, at the cost of some accuracy — similar to lowering image resolution.
- GGUF
- A universal file format for open-weight models that works with llama.cpp on virtually any hardware, including Windows, Linux, and Mac.
- MLX
- Apple's model format and framework, optimized specifically for Apple Silicon — the same model runs faster in MLX than GGUF, but only on a Mac.
- KV cache
- The accumulated context a model has to re-process on every turn of a conversation; it grows as a chat gets longer and is a major cause of slowdown.
- Sparse attention
- A technique where a model selectively attends to the most relevant past tokens instead of reading the full conversation history every time, saving compute on long chats.
- Tokens per second
- The rate at which a model generates output; bottlenecked mainly by memory bandwidth rather than raw processing power.
- Harness
- The tool-calling scaffolding wrapped around a model that lets it read, write, and manipulate files, run bash commands, and otherwise act as an agent rather than just chat.
- LiteLLM
- A gateway that exposes one OpenAI-compatible API in front of any number of local or remote models, so applications can swap the underlying model without being rewired.
- Tailscale
- A private, encrypted mesh network that connects your devices directly to each other without port-forwarding or a public-facing server.
- llama.cpp
- The inference engine that runs open-weight models on a single machine for a single user; the backend most local AI tools like Ollama and LM Studio are built on.
- vLLM
- An inference engine built to serve the same model to many concurrent users at scale, rather than a single local user.
- OpenRouter
- A platform that provides one API and a shared credit wallet for accessing dozens of both closed and open AI models, plus public rankings/leaderboard data.
- Hugging Face
- A repository site where open-weight models — original, quantized, and fine-tuned versions — can be downloaded directly.
- Parameters (the 'B' in a model name)
- The number of weights in a model, usually in the billions; a rough proxy for how capable the model is.
- ODS / Osmentic
- An open-source local AI command center project (by developer Ahmed Osman) that bundles chat, agents, document Q&A, and image generation into one self-hosted dashboard.
- Ambient agents
- Always-on local agents that continuously scan a system for security issues or do background housekeeping, cheap enough to run 24/7 on local hardware instead of paid API credits.
Things they pointed at.
05:00toolOpenRouter
18:20toolHugging Face
01:58toolOllama
18:20toolLM Studio
23:13toolLiteLLM
39:13toolpi.dev (agent harness)
40:47toolODS / Osmentic (Ahmed Osman's local AI command center)
Lines you could clip.
22:25
“You're basically cooking for a party of one.”
23:00
“As soon as you're forced to drive that same car on a highway with a speed limit of 300 miles an hour, you're gonna have problems.”
11:45
“You have this open source model as a brain in a jar. Harness allows you to attach limbs to this brain.”
35:04
“You can think of GGUF as this universal passport.”
13:15
“The model is not the product.”
23:48
“Swap the model. The apps never notice.”
43:34
“Make sure you own your AI. AI in the cloud is not aligned with you — it's aligned with the company that owns it.”
Word for word.
Read-along
Don't just watch it. Burn it in.
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
metaphoranalogy
00:01Over the past few months, we've had multiple AI scares. Last week, Anthropic pulled access to its newly released AI models. Fable five and Mythos five models.
00:10Neither version is available. Overnight, we lost access not just to one, but to two frontier models. It was too dangerous for everyone to have.
00:19Too powerful to be released to the public. Now, we've got them back. But those few weeks set a precedent.
00:25It was a reminder that at any moment, whatever you do day to day, however you use however you leverage it, could all change in the snap of a finger. Open source or open weight models by their nature are not on par with the bleeding edge AI. But for many day to day use cases, they're becoming more than good enough for the work that you do and spend tons of money on closed source AI.
00:46And the best part is that they'll keep getting better. So more than ever, it's important to have the mental insurance policy of knowing how to use them, how to enable them to their fullest potential, and whether or not it might make sense to invest in hardware.
00:58So the goal of this mini course is to break down a lot of the hairy concepts that might have held you back from wanting to explore your open source journey in the past. There's a lot of intimidating terms and jargon that probably pushed you away, and I wanna demystify as much as possible. And throughout this video, I'm gonna show you a bit of a twist.
01:15I'm gonna walk you through how you can use closed source models like Claude and Codex to help you set up, build, and even maintain and bug fix anything that goes wrong in your open source setup. So instead of making you pick sides, I'm gonna show you how to get the best of both worlds. If you watch this till the very end, then you'll be at peace because you'll be nimble, informed, and a lot more comfortable with most of the technical aspects of open source setups.
01:40Let's jump in. Alright. I'm splitting this video into three core sections.
01:43First The thing we're gonna do is go over the key concepts, and then I'll show you how to pick the right model for the right use case at the right time and use things like Tailscale, OLAMA, and others. And last but not least, I'll give you the path of least resistance to create your own open source command center.
02:01Alright. So take a look at this. This is my personal AI command center running only on local models hosted on this Mac Mini right here.
02:08Now for your purposes, depending on what models you wanna run or what your current specs are of your hardware, of your laptop, your desktop computer, you might not need one. But if you want a separate environment to isolate and run these kinds of experiments until you get the hang of it, something like a mini can be helpful.
02:24But the bottom line is that you have this dashboard with a series of functionalities that you can take off the shelf that you would have otherwise had to set up manually one by one just 24 ago. So you can do things like having an AI chat, running and setting up Hermes agent out of the box, having document q and a, which is basically rag, which used to be a huge headache to set up.
02:44And have you a series of other functionalities like even creating workflow automations using a local version of any of them. So if you click through, let's say, AI chat, this will open a brand new tab using what's called open web UI.
02:56And the whole purpose of this is to give you the look and feel of ChatGPT or Claude where you have this environment where you can drag and drop a file as you please, you can do speech to text, and most importantly, you can send a query, and that query will run on your Mac mini, not on this actual laptop. So if I send a question like, what is the best type of AI consultancy for me to start in 2026?
03:19And I send this over. When this runs, if we go back to the dashboard, you'll see we'll have a bit of a spike in the CPU of my Mac mini.
03:27And this is how much memory is being used. This is the context window, and this is the primary model that we're using. And if we go back, this will obviously take longer than usual than a normal chat you'd have on the Internet, but it will still give us back the result.
03:40And it will look something like this where you have the thinking patterns, you can go through the response, and if you wanna take this on the go, and when I mean on the go, even on your phone, if you just have Tailscale set up or something like Tailscale, then you'll be able to interact with your dashboard, even something like Hermes agent just by clicking through, and it will take you to your Hermes console session.
04:02You can go to chat, and you can have a full conversation with this. So the beauty is not only this private mesh network that you can take anywhere, customize it to your own specific desires. And if you wanna do things like image generation, you can click through here, and after you set up what's called Comfy UI, you can create a whole open source pipeline to generate images like this, which aren't the highest quality like g b t two image or Gemini 3.1, but still good enough for the majority of use cases.
04:28And on top of that, you have an extensions tab here where it's basically a small app store where you can add more open source technology to help you with orchestration, vector databases. You have things like integrations, which shows you a full map of your entire system, a full cross section of it.
04:44And then you can see whatever models you have running on your Mini. So I have a Quen visual model. I have this Quen 14 b model, and I have Mistral for things like email triage.
04:53Now the point of me showing you this is not to dangle something fancy in front of you. It's just to show you where we're gonna end up through the remainder of this video. Now before we dive deeper into any concepts, it's important to clear the air on this apparent war that's happening online.
05:06You have team closed source that thinks that open source is not only useless but way behind and can't be used for day to day work or at least effectively. Then you have team open source which make jokes that people only using closed source saying your time is coming to an end. You will be stuck and you'll have no idea how to do your day to day work.
05:24As usual, you'll find me right in the middle where I think both sides are wrong. I think open source is essential to have as that backup redundancy in case all of these costs explode and you have all of these operating systems that don't need that much firepower to run day to day.
05:39At the same time, myself and my team use closed source AI heavily from building custom development projects to teaching enterprises and other businesses how to use AI. I have to know how to use both worlds and both worlds have their pros and cons. So it is a life skill in my opinion in today's age to learn how to use both effectively.
05:57If you're still not convinced, let me give you four more reasons why it's worth considering running AI on a local machine. First and foremost, you have what's called vendor risk and this is what happened with things like Fable and GPT 5.6 and will likely happen with subsequent models where some models might never see the hands and the terminals of the general public.
06:17You might have forgotten, but when we lost Fable initially and then we got it back, we didn't get the full version that we originally had back. There was noticeable differences in quality, in the length of time it could run and not make mistakes, its level of ingenuity, etcetera.
06:31And this is a model that is the light version of mythos that we have yet to still see or use. Consider the bill because the subsidies of all these plans is coming to an end by nature of pure economics. If you have venture capital that is subsidizing these companies like Anthropic and like OpenAI to be able to offer $100, $200 plans that have been shown to be equivalent to 6 to $8,000 of usage on Claude and 13 to $15,000 of usage on OpenAI every single month.
07:01It doesn't take a mathematician to realize that at some point, if these companies wanna go public and they want their public offerings to go well, they're gonna have to make a lot of money very quickly. Companies like Uber and Airbnb were lost leaders and didn't make profit for many years to get people into their ecosystem and addicted in using their products.
07:19This might be the exact same story with closed source models. So if you have an entire company who are addicted to using co work and that is their source of truth and all their transactions run on there, all their automations run the cloud on that platform, if the bill were to five x, they might change over the course of a year, but still they'd have to pay for it for a while because they would be locked in.
07:39Now third is privacy. And the first use case of this is obvious, which is not having your personal information being trained on. So a good example is recently I built this whole health operating system and I processed all of my DNA.
07:51I had it analyzed by local models that weren't sending that information anywhere else. Once I had that synthesis and analysis, then I could use closed source models to help me fine tune and build this operating system.
08:02And the last reason is something that people either don't know about or don't know about well enough, which are ambient agents. And the whole point of ambient agents is that they can constantly scour your operating system, look for security flaws, double check on the health of apps, do constant rag on any files on that system.
08:19Basically, have a twenty four seven running team of agents that might not be doing the most magnificent bleeding edge activities, but are doing dirty work that you shouldn't be spending API cost to do. So instead of running OpenClaw on even the cheapest closed source models and spending thousands a month, you would spend tens of dollars or maybe the early hundreds of dollars a month on the equivalent electricity to keep this system running.
08:42Now some of you might have already used things like Olama in the past and you downloaded these models like Lama 3.1, Lama two, Mistral, and you were completely underwhelmed.
08:52And you were right to be underwhelmed because two years ago, even twelve months ago, a lot of these models were toys. But with today's models like Kimi k three, not only do you have better models, better systems, but you also have things called harnesses that you can attach local models to to make them work that much better.
09:08And like I said before, you can use a $20 all the way to a $200 plan on codecs or Cloud Code to help you build the entire infrastructure. This command center that I showed you at the beginning of the video, this entire thing was set up by taking someone's GitHub repo called ODS, feeding it to Cloud Code, having it learn and fan out a bunch of agents to look through all the code, then help me set up the perfect environment, tailor it to me, find the best model for my specs, and take care of all the plumbing.
09:37So the goal here is we can use the easy button to creating our local open source system. Now this is the most important point of the whole video. And once you understand this ecosystem diagram, then everything else will fall into place.
09:50So step one, if we start from the back all the way to the front, you have your hardware layer. And this could be your Windows laptop, your Mac, your Mac mini, your Mac Studio, whatever it is that you own, this decides pretty much what models you could run, at what capacity, and at what latency or what we call tokens per second.
10:07Once you understand what we're dealing with, then we can understand which models. And when we refer to local models, some people might say open source models, local source models, or open weight models. Open weight just means that we have the entire file of weights that represent how these models tick.
10:24So with things like Anthropix Cloud and OpenAI's GPT models, you don't actually understand how they work. They don't have the weights publicly available. But with these Chinese models, Gamma models from Google, you can physically see these weights so you can see exactly how it derives the answer comes up with.
10:41Once you know what models you can use, you then have your inference engine layer. And this is basically involving something called LAMA CVP usually or VLLM. LAMA CVP is basically the bridge through which you have local models, take the input information or the prompt from you, run it through this engine, and come up with the output.
10:59And VLLM is just an alternative that is used more at a team scale. Once you have your engine established and that's all taken care of, then you get to what's called the gateway portion. And the gateway is typically something called Light LLM, and this allows you to create what are called APIs that allow you to connect any service to these local models running on your computer.
11:19So you can think of it as a that allows all of your applications to talk to whatever model you want. And once you have the gateway, you have the final layer on top which allows you to interact with these models through platforms like Open Web UI where you can have a back and forth just like you're using ChatGPT or Claude.
11:36And on top of that, if you wanna make these models more effective, then you hook them up to what are called harnesses. And harness is basically allowing you to have this open source model as a . Harness allows you to attach limbs to this brain so it can do things like write, read, manipulate files, use things like bash to control and move files around your computer, etcetera.
11:57And this harness is the very reason why people love Claude code and codecs. They think that all of the orchestration is the model itself, but it's actually the symbiosis of the model with all of its limbs that it uses to actually manipulate its environment. And outside of that, if you wanna be able to use the model with something like Cloud Code, you can always hook up open router through API and use Cloud Code's harness and borrow it with open source models or you can do this locally as well.
12:23And by the way, if you enjoy the way I teach and break down some of these more sticky concepts, this is what I do day in and day out in my early AI adopters community. I go through all kinds of concepts from how to use Claude code to codex to harnesses to even how to monetize AI responsibly in this day and age.
12:40So if you're looking for a one stop shop without the hype, without the obsession of things that don't actually matter and move the needle, then check out the first thing down below and maybe I'll see you inside. Alright. Back to the video.
12:50So let's tackle all of these layers. And what we can do is bundle the first and second layer together. So we can use something like a Claude code, a codex, or even something like a QEMI three to look at your current hardware and understand what is your RAM, what is your memory, what does that mean in plain English, what is your vRAM, and then understand based on this what are the best models that you can run where they'll actually practically run and you won't wait five business days for an answer to hello.
13:18So if we hop into a terminal, we can send over a very basic prompt like this, and I'm using Opus 4.8, and we can say, can we use open router API to pull the latest open source models in the past six months as of today's date. And then we can say categorize it into text only generation models versus multimodal. There are even some models that aren't that efficient, but they let you run completely for free.
13:41Not the most performative, but they can actually work. And before I keep going, I wanna zero in on what OpenRouter is just in case you don't know.
13:49It's a platform that pretty much allows you to talk to any model, whether closed or open source. You would basically have this credit wallet, and you can pull from that credit wallet by using that API so you can access all kinds of models from one main bridge. And the core thing we're using it for is this explorer models feature where it has rankings and entire leaderboard, and they have a public API that you can tap into.
14:11So in this case, all we're doing is pulling the comprehensive list from that source as our proxy to avoid us to have to do something like web fetch and researching where sometimes it's on and off on how comprehensive it is. So once we get that full list, it will dump it as a JSON file. You can see here all of the models.
14:29And in this case, we care about text only versus multimodal because if you wanna be able to generate images and understand images, create videos and understand video, and create audio and vice versa, you wanna know exactly what is the right model for what you're looking for. And one of the benefits they have on top of having an entire database of the latest and greatest models is this rankings tab where you have all the top models, a whole leaderboard, top models by task.
14:55So if you wanna be able to know which models can consume and create images, our multimodal, this would be a very easy way to understand that, which are great at debugging, planning.
15:06And, obviously, you can see Anthropic owns a lot of the leaderboard here. But you can then filter on open source models, see which ones would fit the best for your specific tasks. But with that, we can have Cloud Code figure out how to use the open public API and process the information.
15:23So that's what it does here. It pulls the information. You could see here all of the text based models, and then it categorizes it in these tables.
15:31So these are all of the text only generation models in the past six months. And then we have the multimodal ones, the ones that can sometimes understand and create things like image, video, and audio. So this is the full list, and this is the key thing here.
15:47Now we have the list, we and could say something like this. I want you to go through my current operating system specs and tell me what models I'm eligible to run if I wanted a model that's really good at generating proper and quality text, but also can understand image and potentially video and create audio.
16:07If I need multiple models to accomplish this, then let me know, but I wanna make sure that it doesn't run very slow on this computer. Now with this seemingly vague prompt, you can send this over and it will be able to access the specs of your operating system, then compare that to what these models need and give you the right shortlist to get started.
16:24If you're curious what I'm running on, this is an Apple m five max with over a 128 gigabytes of memory, 18 cores, a GPU, and overall, it's a beast, but it's very expensive. You will not need this necessarily for whatever you wanna run day to day.
16:39I wanted to make sure that I could invest in all the hardware I could before all the prices inevitably rose a couple months ago. So if we scroll down, it will break down exactly depending on your laptop and your OS, what makes sense for you. So it breaks down the chip, the unified memory, and really important, of this memory, how much of that memory can be used towards models?
17:00Sometimes people will read the box of their laptop and say, oh, there's a 128 or there's 50. That means I can use 50. You can't necessarily use that because you have other apps that are running on your computer, and you need to take that into account.
17:14Then you have storage and then it breaks down exactly what models might make sense for me. So you'll see here I can run the Quinn 3.5, 3.6, Gamma 34, 31 bit.
17:26If you don't know what the word bit means, we will get to that in a bit, pun intended. And then you have the NVIDIA Nemotron, and then it tells you which ones are way too big to run.
17:35So although this is a beast, it's not beast enough to run something like a Kimi k two. You would need what's called the quantized version, which we'll talk about later, but it's basically a compressed version of that model that is meant to run, but maybe not run at the fullest capacity. Now in this case, it couldn't see which models have the best text to speech.
17:53So one thing you could do is ask it to spin up or fan out a series of agents or sub agents to look for the best models to allow you to create and do text to speech. And one additional thing here is it tells you exactly how you can run this. So it says you can run both simultaneously, and the easiest way to serve these locally on Apple Silicon is LM Studio or Olama.
18:13You can see right here, Lama CPP back end, so the wrapper around this specific engine. And then there's something called MLX that's optimized for Apple products. Now if you wanna be able to download these without adding additional software, then you can literally say, can we download this specific quen model that you recommended?
18:29It will check what's called Hugging Face, and Hugging Face is basically the equivalent of an app store for models. So whether they are brand new models, whether they're quantized, meaning compressed versions of existing models, or fine tuned models, you'll find them all here. So OpenRouter can serve it, but you can't necessarily download it.
18:47Hugging Face is this entire repository where you can download the models. These are models released by vendors themselves, let's say, like Kimi, Quinn, GLM, as well as open source models that have been distilled, meaning maybe someone took a version of Kimi and not only compressed it, but made it really great at understanding images for some reason.
19:06You'll find all of these different permutations of models here, and it was smart enough to know to have to investigate and see, is it downloadable from this database? If we scroll to the bottom, you'll see, yes.
19:16It's downloadable, public, and there's a version tailored for your Mac. Here's what I found. So in this case, it'll give us some options.
19:23I will go with whatever is recommended. You just wanna double check that you have more than enough room to support this because in this case, it is 70 gigabytes. And it seems like it has both image and video reasonably fast, and it looks like we can handle it.
19:37So I will click on enter. This will start the downloading process. Now in my case, I already have a llama and brew installed on my computer.
19:45But if it's missing from yours, it should be able to understand that it's not there and install it as a prerequisite so then it could set up the download process. And you'll see this will take a little bit of time. It will install it, optimize it, and then it will be ready to use on this laptop.
19:59It not only pulls and downloads the model, but the best part of running this terminal with a closed source model as Smart as Claude is it realizes that I need to update my version of Python to run it. You could see right here. The 65 gigabyte download was perfect, but the serving library needed two fixes.
20:15And instead of me having to figure out what that means, it would go and update what's called your virtual environment and then the model architecture to make sure it runs as well as possible on my specific computer. And if we scroll to the very bottom, it tells you that it's fully operational.
20:30It tells you the number of tokens per second. So this model will work at 60 tokens per second. This is exactly how much memory it will take of my one twenty eight.
20:39Typically, you don't wanna exceed more than 50%. You'll start running into problems, especially if you're running other things in parallel.
20:45And if we scroll to the bottom, I had one issue. This one issue was I wanted to be able to interface with it using something like LM Studio or Olama. If you remember, because both Olama and LM Studio are wrappers around LMACPP, Cloud Code was able to stitch together the connection to this model.
21:02So now we can load it, and we can actually have a conversation with it. Now, hopefully, the video doesn't collapse. But if I say, hi.
21:09How are you? We should get, like it said, 60 tokens per second. We can already see that.
21:14It's thinking. We could take a look at its thinking traces, and we should get, hopefully, a very simple response fairly quickly.
21:22And I'm running this while I'm running this video in four k, so it's doing a decent job. And before we keep going, I'm gonna eject the model so I can make sure we have as much memory as possible. We'll close this.
21:32We'll close Olama. In the same way we did this locally, we can also do this on my Mac mini. I can connect to it through screen sharing through what's called SSH secure shell, and I can run that exact same prompt over there where I can say, as of today's date, find the best performing models for text to speech on this specific hardware.
21:50And now you know the the whole pattern. It will scroll down. It will find the best matches.
21:55It'll create recommendations. I can tell it to install it on my behalf. And then once it's done, it looks like it's pretty much close to that.
22:02I can always tell it go and update our dashboard to make sure that I can see this under the models tab if it doesn't happen automatically. So we've already tackled three of the five layers. And before we move on to the gateway layer, I just wanna close the loop on what the difference is between LAMA CBP versus VLLM.
22:19So if you think about it mentally, when we are working with a local based model on our personal laptops or hardware, you're basically . When you're cooking for a party of one, you don't have to worry as much about scaling that inference across the board.
22:34But if you are serving the equivalent of an industrial kitchen, you need a way that this can scale and multiple people can hit the same local models at the same time, and that's where something like VLLM comes in. It basically allows you to do what we did locally but apply it at scale.
22:49Now another way you can think about this is through taking the analogy of a car. For yourself, you might be more than fine with driving a beater that will take you from point a to point b even if it's a little bit slower, it has some weird sounds on the road as long as you get the output you're expecting. But as soon as you're forced to drive that same car on a highway with a speed limit of 300 miles an hour, you're gonna have problems.
23:11So the point of VLLM is it allows you to run these engines at a much higher capacity so that multiple people can use it concurrently. Now let's take a look at the penultimate layer, which is the gateway, and one of those gateways is called Light LLM.
23:25You can think of Light LLM as a , and the whole point of it is that if you wanna be able to interface with your local models through things like chat, through the open web UI, through things like agents that can either code in parallel, asynchronously, or run-in the background as ambient agents, or run automations through hosted locally, all of them can go through this middleman, which is the LightLM layer.
23:49And the beauty of it is that it has what is called an OpenAI compatible API, which allows you to have one back end that can speak to hundreds of models.
23:58So all you have to do instead of having to download a model like you saw right now and constantly rewire it to the chat or to work with Olama or everything else, you have this middle layer that helps with the orchestration itself. So if you were using quen and you wanna move on to using GEMMA, then all you'd have to do is swap this cable for this cable right here.
24:18The whole point of this is to keep you as nimble as possible.
24:24Now before we come close to the last layer, is actually setting up the final interface that hooks up to everything running on your system, we have to understand a few more key concepts. One of those concepts is the private mesh, and this very mesh is the reason why I can access this dashboard that is running infrastructure on my mini.
24:41And the way it works is it's using something called Tailscale, and Tailscale is basically a private encrypted network that makes it possible to have one central hub where I can tap into every single device as if they were sitting in the same room. So instead of doing things like port forwarding or hosting a server where someone could possibly compromise it and get access to those devices, you basically have them all chat to each other secretly in a way that nobody else can see.
25:07And one of the many reasons why I like it is it's not only very easy to set up, but also when it comes to pricing and again no affiliation here whatsoever is you can use the personal plan completely free and that gives you access to unlimited user devices. So whether I connect one Mac mini or five, it'll be the same price and you can add up to six users, which for even small businesses, very small businesses, this is more than fine.
25:29And not only is it economical, but it's also very easy to download. You could just say tail scale download app Mac.
25:37This will take you directly to the download page, and this will give you this little app that sits on the very top of my screen. But if you click on this and you click on network devices and my devices, you'll see right here, I have my iPhone, my old laptop, and the Mac mini all hooked up together, including this specific laptop.
25:54So if I were to run an AI chat from my phone and I go on Quen 14 b and I just say, how are you? And then we send this over. This is what's happening behind the scenes.
26:05Like I said, we have an m two, we have an m five laptop, we have an iPhone, and a Mac mini in this network. When we send that request from the phone, it will directly travel encrypted to the Mac mini.
26:17The Mac mini will actually run the inference. It will run the engine, and then it will transport back the response securely back to the phone on that dashboard.
26:25So the whole point of this is it allows you to take your entire network on the go. You can integrate it with all kinds of tools, and you can even share it with someone else if you want them on your private network. So now that we're more comfortable with the architecture, let's go back and look at the best models as of today.
26:41And naturally, just like the closed source space, by the end of August, by the end of September, by whenever you end up watching this, the model names might have changed or the model versions. But most likely, the providers will be similar. So from left to right, we have GEMA, which is released by Google.
26:55It's a decent model, but sometimes it's weirdly prohibitive. It will stop you from doing very normal legal things for whatever reason. Once you get to the Chinese models, I'm gonna say the quiet part out loud, they really outperform a lot of what's on the market.
27:10So whether it's Quen, GLM, Kimi, not only are they releasing brand new versions all the time, but their token windows are now reaching that million context window limit.
27:20And their costs, if you host the entire model, are actually going up, but to host the quantized versions, the compressed versions is increasingly cheaper. Now obviously, DeepSeek gave us the DeepSeek moment a few years ago where open source really became a part of the conversation and MiniMax interestingly offers alternatives to things like 11 labs.
27:39So when it comes to voice cloning, it has very unique characteristics versus the other models. If I had to pick my favorites having used all of them now, the Quinn models and MiniMax give you really great agentic coding options, but also you can use GLM as a workhorse.
27:54So let's say you're using a Claude code or a codex, and you wanna be able to still pay for closed source, but maybe offload admin tasks to workhorse models, you can use both. We have the closed source take care of really detailed planning, bug review, testing, and then for small tasks, like go and change all the names of these files.
28:13Go and search for this specific file using GREP, which basically lets you search through a variety of databases, files, etcetera. You can start to be more intentional with what you offload to what kind of models you can save on cost and tokens. So naturally, you can try before you buy, and you can use something like OpenRouter that we talked about before to test out all these different models on a series of tasks.
28:36So you could hook it up to ClawCode or Codex natively, which I'll show you in a second, and then you could test five different models on 10 different tasks and see which ones perform the best on the workflows of choice. Once you have that, then it's easy for you to decide what deserves to live on this local hardware permanently.
28:53What we can do is say something like, Cloud Code integration open router docs, then we send this over.
28:59You'll see the first link is Cloud Code integration. We'll click on that. You can read it yourself.
29:05You can feed in the page, or for me, I'll just feed in the raw URL. So I'll go into a brand new session in Cloud Code, and I'll say something like, I wanna be able to use OpenRouter's API natively through Cloud Code, read through all the documentation, spin up any sub agents if you wish to look at any details, and create a full plan on exactly how we can communicate through the API, maybe through switching the model to an alias that's called open source.
29:31So we'll send that over, and then I will paste this link. It should be able to now start using its web fetch tool, pull the information, and create a full plan.
29:41So after a few minutes, it comes back with a full assessment of our options. And if we scroll to the bottom, it tells me exactly what I need to decide on. So what is the primary open source model that I wanna use?
29:51And it tells me that one way you could implement it is to create a specific mode that's called Claude OS. If we pick Claude OS, then it will run using the open source models. Otherwise, if you use Cloud, then it will use the standard models out of the box.
30:04So in this case, I'll just say, let's scroll down and let's go to something and say, want Kimi k three. And then how should I handle your open router API key? Now multiple ways you could do this.
30:16You could use the key chain. You could use a private source file. Now for simplicity, I'll just use a dot n file.
30:23We'll do isolation. We'll send this over, and it should be able to come back in five, ten, fifteen minutes and hopefully have this set up. And just like that, everything is set up, and it created a shortcut command that's called DSP OS.
30:35And DSP stands for dangerously skip permissions, which is basically running Claude codes harness in YOLO mode. So if I go into a brand new terminal and I write d s p dash o s, this will open up Claude, and then you can see right here, it's using Moonshot Kimi k three.
30:52If I wanna be able to switch the model, all I'd need to do is write the same command, but with the name of the language model right after. So if we pop back over, it created this other shortcut command that's called DSP OS GLM. If we run this in a new terminal, d o s one more there.
31:09GLM, It should ask me permission, and there we go. It's now using GLM 4.6.
31:15And if we say build me a tic tac toe app, what's happening behind the scenes is that this is being sent to the open router API.
31:24It's interfacing with this model. You can see right here it's entering plan mode, and it's borrowing the harness of Cloud Code.
31:30So you get the best of both worlds if you wanna be able to use this kind of model at a lower rate of inference, see how it works before you decide to actually download it on your hardware and using it all over the place. And one thing I wanted to show you is you could see that it's executing this task, and at the very bottom here, it's running some agents and sub agents.
31:49So even though it's still not the Claude code model, because we're using the harness, it has access to all the tools that we know and love that we typically use closed source models for. Now leading up to the final build, I'm gonna go over some key concepts in quick succession of words they've either heard of before and have no idea what they mean, or you might think you know what they mean, but you wanna be 100 sure.
32:09One of those concepts are the number of parameters of a certain model, which are usually in the billions. So if we pop over to our command center and we hover over the AI chat and we go back to our model, you'll see we're running QUEN three, but we're running the 14,000,000,000 parameter model. What does that mean in plain English?
32:26Now for all intents and purposes, you can assume that the number of parameters is a proxy for how capable that model is. So if we take a model like Kimi k three, it has almost 2,800,000,000,000 parameters at the full scale model.
32:40Running that at home could cost you $10,000 a month plus in pure electricity costs, let alone all the hardware you'd need to use it.
32:47So when you compress these models, you make them less capable, but at some point, these models have become good enough that a 14 b, a 27 b, even a 50 b can be more than sufficient for a lot of your day to day work. And this will keep increasing over time as closed source improves and open source improves in tandem. And one rule of thumb that you can use is you can take the total number of parameters in a model, divide it by roughly two, and that will give you an approximate amount of how much memory you need on your computer to run it freely.
33:17So if we take the 70,000,000,000 parameter model that we downloaded earlier, it needs roughly between 35 to 40 gigabytes of RAM depending on your computer and a few other factors. But overall, you can start to see that if you run a 120, you need almost 60.
33:31And if you have a very small, not as agile computer, then 14 would need anywhere between five to seven, which could be feasible. Now one thing we didn't touch on is what four bit means.
33:41It's pretty much a proxy for quantization, and quantization is a proxy for compression.
33:47So if we jump into quantization, imagine you have a very smart brain, and as you compress its knowledge and its fidelity of knowledge, over time, it's kind of like resolution of an image or resolution of this video. This is being streamed in four k.
34:01It could be in ten eighty p. It could be in seven twenty. Once you get to four eighty, it becomes really rough, and the exact same thing happens with compression.
34:09So on average, you could have 16 bit, you could have eight bit, you could have four bit, and you can continue until you go to one and below that. And at that point, once you get to one and below, you've over compressed the brain to the point where a lot of the intelligence, a lot of the capability, the ability to know how to use tools even with a great harness becomes severely prohibited.
34:30And if you ever see any one of these words in the wild, I don't care to go through these compression algorithms specifically, but just know that they're different versions of compression. So you have a combination of how many bits and what compression algorithm, and then you come down to the number of billion parameter models.
34:45So what you should care about if you're running things locally is finding the smallest quantization of the smallest billion parameter model that still passes your tests and accomplishes whatever it is that you're looking to do in your day to day workflow. Now the next key terms here are GGUF and MLX.
35:01You can think of GGUF as this . It can work with pretty much anything.
35:06If you use a GGUF model of which you could find tons of them on Hugging Face, it is the most generic and versatile type of model. So if you download this nine times out of 10, it will work with LAMA CPP, meaning it will work out of the box with a LAMA and LM Studio.
35:22When it comes to MLX, this is basically an . So the same way that GGUF works with all gates, MLX is specific to Apple.
35:32So it is meant to work with Apple Silicon, so you can use the same model, but it will perform much better because it's optimized for that specific stack in architecture. So the rule here is simple. If you're not running on Apple, then you have no reason to look for GGUF.
35:45And of all the models you'll find on Hugging Face, you usually will find an equivalent of MLX to GGUF for every model. But for some, they just have the general version.
35:54That's one thing to keep in mind. You can run GGUF on everything including a Mac OS, but this one is specific to this operating system. Now this next term trips a lot of people up, and it's called KV cache.
36:05And the way you can think of it is when you start a conversation, you're gonna have a blank slate or in this case, a blank desk. As you have more back and forth and you have a multi turn conversation and potentially that multi turn conversation is calling tools, maybe it's using a harness.
36:20Now you have the system prompt of the agent you're using. Let's say it's Hermes agent, which has a mammoth system prompt, and then you have back and forth conversations. As you keep going, you accumulate tons of context.
36:32So this desk here on the left hand side keeps accumulating. So one way to optimize the use and the performance of local models is to manage your KB cache. This could mean adding or basically distilling a shorter system prompt for whatever agent you're running.
36:48Now of three ways that you can optimize this cache, one of them is to have a shorter system prompt for your agent. So exhibit a would be for our Hermes agent. When we clicked into it and I chatted with it using the 14,000,000,000 parameter model on my Mac mini, it took around two minutes to respond to hi.
37:04And then I ran a diagnostic, and this diagnostic showed that the system prompt being pushed through the model was humongous, and a lot of it I didn't actually need. So as soon as I pruned it, I went down from two minutes for a hello to thirty seconds.
37:18So doing this kind of pruning on the system prompt is helpful. And then obviously adding smaller context, meaning if you don't need to write a whole essay or a story for a specific instruction, then avoid it.
37:29And lastly, similar to quantizing a model, you can also quantize or compress the cache. So as you have a conversation, which is something that a lot of chat agents and chatbots do in the wild, maybe after 20 messages, you summarize those 20 messages as one message, and only that one message is persisted in the next parts of the conversation.
37:49And this keeps going to give your conversation as much bandwidth as possible. Now there's something that's called sparse attention that became popular with the mini max local model, and the way it works is as follows. On the left hand side, you have your standard model where it reads every single word of every single part of the conversation at all times.
38:06On the right hand side, you have sparse attention where it picks and chooses which tokens might be the most relevant when it looks back at the past. So this makes it a lot more token efficient and helps alleviate a lot of the performance issues that can come with larger models. Now before I briefly touch on harnesses, I wanna mention one last term which is tokens per second.
38:27And the way you can think of this is as follows. Your memory bandwidth on your computer is like a pipe, and the width of the pipe allows it to dictate how large of a block or blocks can pass through and at what speed. The smaller the blocks or the smaller the bits in this case, the faster they can run.
38:44If they're very large blocks in a very tight pipe, then you're gonna have anywhere between five to 10 tokens per second, which is actually very slow. That's equivalent of getting five words over the span of maybe, let's say, minute.
38:57You'd almost have a paragraph. Whereas with 60 to 70 tokens per second, now you can start to have a paragraph within ten to fifteen seconds. So in terms of the quality of life, having a more quantized model allows more agility and allows you to have a better experience on your specific hardware.
39:12And if you're curious on how speed is derived, this is the equation for it. I won't dive into it just in case you're nontechnical and if you already had enough. When it comes to harnesses, we already touched on being able to use something like Cloud Code and combining it with open router.
39:25You can also find ways to pipe things like Olama and use that exact harness locally without the open router. You can use Codex's harness. You could use Gemini's, and you can also use open source harnesses like py dot dev.
39:38And the way this works is it's a very minimalistic, very malleable framework that allows you to start to build the limbs that you can hook up to your brain of choice. Now you can set this up by copying and pasting this curl command and running it in a terminal and following all the onboarding, or you could do what I do and give Claude code or Codex the link, tell it to read the documentation, fan out a series of sub agents, become familiar with it, then set it up for your specific operating system.
40:04Now the key trick with this is once you have it initially set up, it will not be as robust as Claude codes or codexes, but it's gonna be your harness. Meaning, as you monitor performance, keep giving feedback to your language model of choice to improve the harness because after five, ten, 15 iterations, it might get to the point where it's optimal.
40:23Now the main trade off with using py dot dev versus something like a Claude code harness is if you just wanna use closed source models, a lot of the harness engineering is done for you. They keep updating it to be optimized to work with the right model at the right time with the right tool calls.
40:37But with py dot dev, as you keep having brand new models come out, you will incrementally need to keep tweaking it because all models will have slightly different behaviors and different patterns at which they'd like to call tools. Now with all this information, you have more than enough to go to the last step, which is to set up your local command center.
40:55And there are all kinds of frameworks that you can choose. I'm just gonna show you one that worked really well for me. Now if you search for ODS GitHub, this will take you to this Osmanic ODS and this is a newer repo by someone named Ahmed Osman.
41:09He's actually very gifted at this open source stuff. I'm giving him a huge shout out because he's done incredible work with this, and he's open sourced this entire command center that I've built on called Osmentic. Now using the same technique that I've shown you thus far, is taking repos and websites and documents and feeding them to your language model of choice, you can take this GitHub link, paste it into your language model of choice, and ask it to, one, inspect it and audit it, and then fan out some sub agents to plan how you could implement it on your specific computer.
41:39And this is exactly what I did. So I said fan out some sub agents to read through this GitHub repository to comprehensively come back with a plan on how to set this up locally on our system with our downloaded Quinn models.
41:51So this is the GitHub link. It then does some web search. It goes through the entire repository.
41:57It looks through everything we might need to make this work. It comes back with its initial findings. So in my case, we have an MLX model that we downloaded.
42:04So we just need to make sure that it works well with the LAMA servers. And any issues that pop up, even if you're nontechnical, you typically just have to speak to it in plain English, get to the bottom of basically what is the path of least resistance.
42:17And once you understand that, it can go and actually execute all the steps. So I was secretly running this the entire time I was filming this video, and it took around forty five minutes with a couple queries on preferences. And once it's set up, it now should work out of the box.
42:31If I go to here and I put this local host address, it should initialize, ask me to label the platform.
42:37So I'll say, Mark's ODS or maybe I'll just keep it Mark's ODS.
42:44There we go. We'll click on continue. I'll just say username is Mark.
42:48Continue. Let's do everything and then continue.
42:52We'll click finish and here's your dashboard. And the core differences between what I showed you at the beginning and this is one, we don't have the logo at the top left hand side.
43:00Two, I enabled all different features here by going back and forth with Cloud Code, and this is using instead of the out of the box model that it comes out with, which is the fee model. It is using the one that we downloaded earlier. So now we have a full pulse on exactly what's happening in our system, where everything's located.
43:17We can start to install the extensions, add the integrations, and you can see, we we can start to really stack this up and get it to be as sophisticated as this one is here.
43:26And naturally, you can take this and combine it and Frankenstein it with a series of other frameworks to make your perfect snowflake version of your local setup. So hopefully, entire walkthrough gave you the full TLDR of exactly how you can go from not knowing how to use local models to using them actively and using them effectively with the path of least resistance.
43:46And I know I threw a lot of you in this mini course, so on top of giving you all the documentation that you need to review everything that I went through, I've also put together this local AI engineering guide. It is extremely sophisticated. It documents everything that I've read, seen, and surveyed in the past month and a half that I've been setting up my own rig.
44:04So you'll get both of these resources along with a few other goodies completely for free down in the second link below. And as always, if you wanna level up your AI game beyond just local models, but also how to use things like Cloud Code and Codex to their fullest potential and be supported by a team of 12 plus coaches, myself, an entire community, and all the other resources that you'll never see on YouTube, then make sure to check the first thing down below, and maybe I'll see you in my early AI adopters community.
44:29And for the rest of you, if this gave you a good primer on how to use local models and you feel that much more confident, I would beyond appreciate a like on the video and hopefully a comment to expand the reach. I'll see you all in the next one.
The Hook
The bait, then the rug-pull.
Last week Anthropic pulled public access to Claude Fable 5 and Mythos 5 overnight — a reminder that whatever closed-source workflow you depend on can change in the snap of a finger. Kashef uses that scare as the on-ramp into a five-layer stack for running your own AI, with a twist: he uses Claude Code and Codex to build the open-source setup for you.
Named ideas worth stealing.
09:39model
The 5-Layer Stack
- Hardware
- Model weights (open weight)
- Inference engine (llama.cpp / vLLM)
- Gateway (LiteLLM)
- Interface + harness (Open WebUI / agents)
The mental model Kashef uses for the entire video: any local AI project is a stack of these five layers, built from the hardware up.
Steal forScoping any local-AI or self-hosted-infra project before buying hardware or picking tools
05:56list
4 Reasons to Run AI Locally
- Vendor risk
- The subsidy is ending
- Privacy
- Ambient agents
Kashef's case for local AI as insurance rather than replacement: sudden access loss, unsustainable subscription pricing, sensitive data handling, and cheap always-on background agents.
Steal forJustifying local infrastructure spend to a cost-conscious stakeholder or client
35:57list
3 Ways to Manage KV Cache
- Shorten the system prompt
- Keep context per turn small
- Quantize / summarize the accumulated cache
Practical levers for speeding up a local agent that's slowed down over a long conversation.
Steal forDebugging any self-hosted chat agent or assistant that has gotten sluggish over a long session
31:55concept
RAM Rule of Thumb
Divide a model's parameter count by roughly two to estimate the gigabytes of RAM needed to run it comfortably.
Steal forAny hardware-shopping decision before buying a GPU box or Mac for local AI
How they asked for the click.
VERBAL ASK
43:34product
“check out my early AI adopters community... I've also put together this local AI engineering guide... you'll get both of these resources completely for free down in the second link below”
Soft dual CTA delivered direct-to-camera over the finished dashboard at the very end (free lead-magnet guide + paid community), echoing an earlier lighter mention of the same community around 12:23 mid-video — no hard sales pitch either time.
MENTIONED ON CAMERA
FROM THE DESCRIPTION
PRIMARY CTAWhere the creator wants you to go next.
OTHER LINKSAlso linked in the description.
Visual structure at a glance.
