A solo-cast map of the local AI stack, from Hugging Face and Gemma model sizes to three niche businesses built on models that never leave the machine.
Posted
4 days ago
Duration
Format
Tutorial
educational
Views
69.4K
783 likes
57 · 43
Big Idea
The argument in one line.
Local AI is now good enough, cheap enough, and simple enough to run on a laptop or phone, which turns overlooked niches with sensitive data and repetitive review work into buildable, cash-flowing software businesses.
Who This Is For
Read if. Skip if.
READ IF YOU ARE…
A non-technical founder or operator who has only used ChatGPT or Claude and wants a plain-English map of Hugging Face, LM Studio, Ollama, and Google's Gemma stack.
Someone hunting for a boring, niche B2B software idea in a vertical that still runs early-2000s tools and handles sensitive customer data.
A builder who wants a concrete first project: which model to download, which app to install, and what prompt to run today.
SKIP IF…
You already run local models daily and know the difference between llama.cpp, MLX, and quantization formats cold.
You want frontier-model benchmark comparisons or deep fine-tuning guidance; this stays at the beginner-to-practitioner level.
TL;DR
The full version, fast.
Local AI means the model runs on hardware you control instead of a website or API, and the space breaks into four pieces: the model (Gemma, Llama, Mistral), the warehouse where you find it (Hugging Face), the software that runs it (LM Studio or Ollama), and the workflow you build around it. Google's Gemma 4 family runs from a 2B edge model to a 31B workstation model, with specialized variants for embeddings, function calling, and vision. The practical path is LM Studio or Ollama today, Google AI Edge and LiteRT-LM once a model needs to live inside a shipped app. The real business filter isn't whether a model is smarter, it's whether it's good enough for the job and whether running it locally makes the product better, which matters most for sensitive data, repeated review work, and work that happens away from a desk.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
Cold open with the episode's thesis (local AI as a 24-month business opportunity), a full roadmap of what the episode will cover, and a brief sponsor credit to Google.
01:35 – 03:09
02 · The Open Model the Landscape
Defines local AI (runs on hardware you control) versus cloud AI (runs elsewhere via API), then introduces the four-piece landscape: model, warehouse, software, workflow.
03:09 – 06:48
03 · Vocab Decoder
Plain-English definitions of parameters, tokens, context window, quantization (Q4 vs Q8), GGUF, and LiteRT-LM.
06:48 – 10:29
04 · Google Gemma Clearly Explained
Walks the Gemma 4 size ladder (E2B, E4B, 12B, 26B/31B MoE) and the specialized models: EmbeddingGemma, FunctionGemma, PaliGemma, ShieldGemma, Gemma Scope.
10:29 – 14:20
05 · Other Open Model Families
Quick primer on Llama, Qwen, DeepSeek, GLM/z.ai, Mistral, and Phi, with the licensing and data-sovereignty caveats for each.
14:20 – 18:17
06 · Path 1: Run Gemma in LM Studio
The six-step LM Studio path: download, search Gemma 4, pick E4B or E2B, grab the quantized file, paste real notes, then start the local server.
18:17 – 20:15
07 · Path 2: Ollama
The command-line path: ollama pull gemma4, ollama run gemma4:e4b, and the local API exposed on port 11434.
20:15 – 21:07
08 · Path 3: Google AI Edge
When to reach for Google AI Edge and LiteRT-LM instead: only once a model needs to live inside a shipped mobile, browser, desktop, or edge app.
21:07 – 21:52
09 · Hardware Cheat Sheet
A RAM-based cheat sheet (8GB, 16GB, 32GB, strong GPU) plus a phone-specific framing: ask about the job, not the model size.
21:52 – 22:47
10 · First Workflow to Build
The recommended first project: a Customer Notes folder, a local Gemma model, and one markdown memo covering repeated complaints, root cause, and the week's top fix.
22:47 – 25:06
11 · Workflows Before Fine-Tuning
Argues for workflow-first, eval-second, fine-tuning-last, defines what an eval is, and shows how the same folder-to-artifact pattern repeats across use cases.
25:06 – 26:33
12 · Local vs Cloud vs Hybrid Eval
Lays out when to use local (private, repetitive, fast, offline, device-native) versus cloud (deep reasoning, giant context) versus a hybrid local-first, cloud-escalation, human-approval architecture.
26:33 – 27:22
13 · Framework for Local AI Startup Ideas
The five-point filter for a local-AI business: sensitive data, repeated review work, bad software, expensive mistakes, and work that happens near the device.
27:22 – 29:24
14 · Startup Idea 1: Home Health QA Reviewer
A local desktop app that reviews home health visit notes and care plans before billing or audit, flagging missing vitals, unclear instructions, and service-level mismatches. Grow it by starting as a manual-plus-AI review service for five agencies.
29:24 – 32:10
15 · Startup Idea 2: Offline Field Report Copilot
A mobile app for restoration contractors that drafts the damage report on-site from photos and voice notes, flagging gaps (missing photos, unclear explanations) while the technician is still on site.
32:10 – 34:47
16 · Startup Idea 3: Pre-Send Reviewer for Professional Services
A local desktop app that acts as a second set of eyes on outbound drafts for wealth advisors, law firms, HR, agencies, and accountants before they go out. Nicknamed 'schmuck insurance.'
34:47 – 37:55
17 · Build Your Local AI Lab
A personal-productivity exercise: a Local AI Lab folder of 10 files run through a model to produce one reusable artifact, plus a closing checklist of where to look for opportunities (private data, offline work, camera/audio, low latency, repeated review, high API cost, buyer trust).
37:55 – 38:46
18 · Closing Thoughts
Reframes the whole episode as a product conversation rather than a model-benchmark conversation, then signs off asking viewers to share the episode.
Atomic Insights
Lines worth screenshotting.
The real question isn't whether a local model is smarter than the biggest cloud model, it's whether it's good enough for the job and whether running it locally makes the product better.
The local AI landscape has four pieces: the model (the brain file), the warehouse where you find it (Hugging Face), the software that runs it (LM Studio or Ollama), and the workflow built around it.
Hugging Face was reportedly in talks to be acquired for $13 billion, and it remains the first place to go to find, compare, and read licenses for open models.
Quantization is compression for models: Q4 is easier to run, Q8 keeps more quality but needs more memory, and Q4 is the reasonable starting point for beginners.
Gemma 4 E4B is the practical starting point for most local tests; E2B fits phones and older machines, and 26B/31B needs workstation-class hardware.
A hybrid architecture beats an all-cloud or all-local approach: local strips and summarizes sensitive drafts first, cloud handles deep reasoning on the sanitized version, and a human approves anything important.
An eval for a local workflow can be as simple as running the same 10 documents through a local model and a frontier model, then comparing whether the local model caught the same details.
A local-AI business idea is worth pursuing when it hits five markers at once: sensitive data, repeated review work, bad incumbent software, expensive mistakes, and work that happens near the device.
16 GB of RAM is enough to run useful experiments with models like Gemma E4B and smaller quantized versions; 32 GB opens up larger local workflows.
The best first local AI project isn't fine-tuning a model, it's picking one folder, one model, and one output, then running it ten times to find where the prompt needs work.
A folder of customer calls becomes a market research memo, a folder of support tickets becomes a product roadmap signal, and a folder of PDFs becomes a risk checklist, the same workflow pattern repeats across use cases.
Verticals like home health, water damage restoration, and professional services still run early-2000s software, creating roughly a 24-month window to build local-AI-native tools before incumbents catch up.
Takeaway
The Business Filter Behind Local AI
LOCAL AI PLAYBOOK
A practical map of local AI, from vocabulary and model picks to running Gemma yourself, built around one filter: is it good enough for the job, and does local make the product better?
02The Open Model the Landscape
The local AI stack breaks into four separate layers: the model itself, the place you find it, the software that runs it, and the product workflow you build on top.
Hugging Face works like a warehouse, not a single vendor: model cards there list purpose, size, license, hardware needs, and whether a quantized version already exists.
03Vocab Decoder
More parameters usually mean more capacity for hard tasks, but the tradeoff is memory: a 2-4B model suits edge devices, 12B is a laptop middle ground, and 26-31B needs workstation-class hardware.
Quantization compresses a model so it fits on a normal machine; Q4 is the easier, more common starting point, while Q8 preserves more quality at the cost of memory.
The context window sets how much information a model can work with at once, and locally the constraint that matters is speed and memory, not a per-token bill.
04Google Gemma Clearly Explained
Gemma 4 E4B is presented as the practical starting point for most local experiments, with E2B reserved for phones and older hardware, and 26B/31B for workstation-class use.
Beyond general-purpose sizes, Google ships specialized variants: EmbeddingGemma for semantic search, FunctionGemma for structured tool use, and PaliGemma for vision tasks.
05Other Open Model Families
Llama remains the default reference point for community tooling and examples, but its license and model card still need reading before commercial use.
Performance from Chinese open-model families like Qwen and DeepSeek is real, but enterprise, government, healthcare, or finance use cases need to separate running open weights locally from sending data to a hosted service.
06Path 1: Run Gemma in LM Studio
The fastest way to feel local AI's value is a business prompt on day one: paste real (or realistic) customer notes and ask for a one-page memo on what's breaking.
Starting the LM Studio local server turns a laptop into a private API endpoint, which is the step that turns a demo into something other software can call.
07Path 2: Ollama
Ollama's command-line path exposes a local API on port 11434, which is the more builder-oriented route into local models.
Scaling to a bigger local model later is mostly a hardware question: try the 12B, 26B, or 31B versions once the machine can handle them, or ask an LLM whether it can.
08Path 3: Google AI Edge
Google AI Edge and LiteRT-LM only matter once a model has to live inside a shipped product, not just run as a personal demo: think a mobile app, browser app, or edge device.
The runtime question of whether a model ships inside an iOS, Android, web, or desktop app is distinct from whether it can simply run on a laptop.
09Hardware Cheat Sheet
16 GB of RAM is enough for genuinely useful local experiments with models like Gemma E4B; 32 GB opens up larger workflows, and a strong GPU makes bigger models realistic.
For phone-based local AI, the right question isn't model size, it's the job: can it read a photo, summarize audio, or classify something quickly without a strong connection.
10First Workflow to Build
The recommended first project is narrow and concrete: ten real support tickets in a folder, one local model, and one markdown file summarizing complaints, root causes, and what to fix this week.
The point of the first local workflow isn't the output quality, it's proving the model ran next to genuinely private data without anything leaving the machine.
11Workflows Before Fine-Tuning
Fine-tuning a model is the advanced move people jump to too early; the practical starting point is finding one repeated workflow, running it ten times, and improving the prompt.
An eval doesn't need to be complicated: run the same input through the local model and a frontier model, then check whether the local model caught the same details.
The same folder-to-artifact pattern repeats everywhere: customer calls become a research memo, support tickets become a roadmap signal, PDFs become a risk checklist.
12Local vs Cloud vs Hybrid Eval
Use local for private, repetitive, fast, offline, device-native, and high-volume work; use cloud for deep reasoning, giant context, and cases where the strongest model changes the answer's quality.
The most durable architecture for sensitive workflows is hybrid: local handles the private first pass, cloud does the heavy reasoning on a sanitized version, and a human approves anything that matters.
13Framework for Local AI Startup Ideas
A local-AI business idea is worth pursuing when it stacks five markers at once: sensitive data, repeated review work, bad incumbent software, expensive mistakes, and work that happens near the device.
The businesses worth targeting are niche and cash-flowing on purpose, tied to one painful, recurring workflow rather than a broad platform play that needs venture funding.
14Startup Idea 1: Home Health QA Reviewer
The wedge into an established, paperwork-heavy vertical is a service, not software: manually review a batch of documents first, write down the twenty issues that keep recurring, then turn that list into the product's checklist.
The buyer in a compliance-heavy niche cares about one thing: fewer documentation problems surfacing at billing, audit, or supervisor review, not the AI model behind the review.
15Startup Idea 2: Offline Field Report Copilot
For work that happens on-site and away from a desk, the product has to flag gaps while the person is still standing there, not after they've already left.
A killer demo for a niche field-service tool is disarmingly simple: ask for three old jobs and show how fast the same report could have been produced.
16Startup Idea 3: Pre-Send Reviewer for Professional Services
Almost every professional service firm already has an informal 'have someone check this before it goes out' step; a local AI reviewer just makes that existing habit faster and more consistent.
The fastest way to build a checklist-driven review product is to interview ten practitioners about which specific documents make them nervous, then encode their actual concerns as the checklist.
17Build Your Local AI Lab
The personal-productivity version of this same idea is a folder of ten files run through a local model to produce one reusable artifact, a memo, checklist, or weekly pulse, not just a chat answer.
Running that exercise repeatedly is what teaches you where private data is trapped in folders and which review tasks keep repeating, which is exactly where the business ideas come from.
Glossary
Terms worth knowing.
Parameters
The internal weights of a model, often described in billions (2B, 4B, 12B). More parameters usually mean more capacity for hard tasks, at the cost of more memory.
Tokens
The chunks of text a model reads and writes. Running locally, the concern is speed and memory rather than a per-token bill.
Context window
How much information a model can hold and work with at once during a single conversation or task.
Quantization
A compression technique that shrinks a model so it fits on ordinary hardware, usually at a small cost to quality.
GGUF
A common file format for local models, designed to make inference easier on normal machines running tools like LM Studio or llama.cpp.
LiteRT-LM
Google's runtime layer for running language models on-device across Android, iOS, web, desktop, and edge targets.
Model card
A page on a model's listing (commonly on Hugging Face) documenting its size, license, supported inputs, hardware needs, and available quantized versions.
MoE (Mixture of Experts)
A model architecture that activates only a subset of its total parameters per task, aiming for more efficient use of a larger overall model.
Embedding model
A model that turns text into numerical vectors so documents can be searched by meaning rather than exact keyword match.
Function calling
A model capability that lets software take structured actions based on the model's output, rather than just returning a text answer.
“It's basically schmuck insurance is the way I think about it.”
funny, brandable phrase for a whole business idea→ newsletter pull-quote↗ Tweet quote
The Script
Word for word.
Read-along
Don't just watch it. Burn it in.
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
17px
metaphoranalogy
I think local AI and open models are going to create a ridiculous number of business opportunities over the next 24 months, and I don't think most people actually have the map yet. They've used ChatGPT, they've used Claude, but when they hear local AI, Hugging Face, Ohlama, LM Studio, AI Edge, it sounds like it's for this developer world and that normal founders are just not supposed to touch it.
And I think that's a mistake because the opportunity here is actually pretty endless. By the end of today's episode, you're going to understand what local AI is, when it matters, how to run open models at work, where Hugging Face fits in here, which Gemma model I'd start with, how I'd run a model locally with LM Studio or Ollama, and how this turns into real business ideas.
And I'll give you three startup ideas I'd actually consider building using local AI, including who the customer is, what the first version does, why local matters, and how I'd sell it. Basically, this is gonna be a masterclass around local AI, how to run models, how to build apps, how to make money from it, and I'm gonna explain it for the average person who isn't technical.
quick shout out to Google for sponsoring today's episode and for caring about local AI and open models for entrepreneurs. Today's episode, I'm going to use Gemma and Google AI Edge as the main examples, but the goal is to give you a full map so you can actually understand the space and build with it and use whatever model suits you.
Okay, let's dive in.
So put simply, local AI means the model runs on hardware you control. The hardware could be your MacBook. It could be your Windows laptop, an Android phone, an iPhone.
It could be a browser, a Raspberry Pi. It could be in a workstation in your office. I just got a DGX Spark, which is like a high -end one.
But the important part to note is... A phone now could run local AI. Cloud AI means the model runs somewhere else and you access it through a website or an API.
And that's the basic difference. The business question is, where should the intelligence live? If I'm doing deep research and strategy and hard reasoning or something where I want the strongest possible model, I'm probably going to be using a frontier cloud model.
If the work involves private files like sensitive customer data, offline usage, field work, low latency, audio input, or an internal workflow that runs again and again and again, local AI starts to make a lot of sense. A smaller model in the right place can actually be very valuable. That is the idea I want you to keep in your head.
question most people ask is, is this model smarter than the biggest model in the cloud? The actual more useful question to ask actually is, is this model good enough for the job and does running it locally make the product better? Once you ask it that way, you start seeing these business opportunities which we'll go into.
So there's four pieces to the local AI landscape. The model, which is the brain file. That could be something like Gemma, Llama, or Mistral.
The warehouse, which is where you find the model. You might have heard of Hugging Face. I think they're trying to get acquired right now at $13 billion.
That's what they do. The software, which is what runs the model, that's something like LM Studio or Oh Llama. And then the workflow, which is the product you're building around all of it.
And those are the real four pieces. The model is the brain file. Gemma is a model family.
Llama is a model family. Quen or Mistral, you might have heard of Phi. These are model families too.
Some of these are actually better at reasoning and some of them are better at coding. Some of them are smaller. Some of them are faster.
Some of them are better for images. Some are easier to run on your own machine. And then you need somewhere to find these models.
That's what Hugging Face is. That's the first place I would go. They're the biggest at it.
The easiest way to explain Hugging Face is that it's a model warehouse. You go there and you can find model cards, licenses, file formats, examples, benchmarks, community versions, and sometimes versions that have already been compressed so they're way easier to run locally. If you're new to local AI, one of the best exercises is actually just to open Hugging Face and read a model card really slowly.
You're going to learn a lot. At first, though, you can ignore half the scary looking details and just look for a few basic things, in my opinion. What is the Model 4?
How big is it? What license does it use? What hardware are people running it on?
Does it support text, images, audio, tool use, or embeddings? Are there quantized files available? Once you can answer those questions, the space gets a lot more intimidating because I know when I first looked at these cards initially, I was like overwhelmed.
So just those are the key questions to ask. Then you need software that runs the model. For most people, I would just start with LM Studio or Ollama.
LM Studio feels like a normal desktop app. You download it, you search for the model, you click download, and then you can just chat with it. My opinion is it's probably one of the most friendly first -time user experiences if you're non -technical.
Ollama is a little more builder -oriented or developer -oriented. You install it, you run a command like Ollama run Gemma for... colon E4B.
And now you have a model running locally with an API your apps could talk to. Then underneath those tools, you're going to start hearing about things like Lama .cpp and MLX. And I'll explain what those two things are.
Llama .cpp powers a lot of the local model inference. MLX matters if you're on Apple Silicon. And if you're thinking about shipping real on -device apps in the Google ecosystem, that's where Google AI Edge and LightRTLM come in.
Basically, Google AI Edge is the broader on -device AI development world, and LightRTLM is the runtime layer. for the language models. This is what you study when you want to move from, I ran a model on my laptop to I want this model inside an iOS app or an Android app or a web app, desktop app, whatever it is.
We got to talk about some key vocabulary, just about the most important things you need to know about these words that come up time and time again in local AI. I'm just going to give you simple, clear definitions of what they are. By the end of this part.
You'll know just the core basics of local AI vocab. So I'm sure you've heard this one before, parameters like 2 billion, 4 billion. These are what's called the internal weights of the model.
And more parameters just usually means more capacity for harder tasks, but it does require more memory. So parameters are the internal weights of the model. The beginner shortcut is that more parameters usually means more capacity.
And more capacity can help with the harder tasks. So the trade -off is usually memory. speed, and hardware.
So a 2 billion or 4 billion model is the kind of thing you might use for edge devices, phones, fast workflows, and smaller tasks. A 12 billion parameter model is more of a middle ground and a 26 or 31 billion model is getting into the stronger workstation territory, depending on your hardware and how the model is built. I recommend like not going out there and spending $5 ,000, $10 ,000, $20 ,000 on a workstation just yet.
By the end of this episode, you're going to understand how to just set up some of these things on your phone or on a laptop, a spare laptop that you have from 2021. Then there are tokens. So tokens are the chunks of text that the model reads and writes.
Locally, you care about speed and memory rather than the per token bill. Then there is the context window. The context window is basically how much information the model can work with at once.
Then there is quantization. The word sounds more technical than it needs to. Honestly, I can barely pronounce it.
Quantization is the compression for models. It allows giant models to fit on normal laptops. For example, you might have heard of Q4, Q8 formats.
That's quantization. If the full model is the giant version, the quantized model is the version that can actually fit on a normal laptop. So you might lose a little quality, but suddenly this thing magically runs.
You will see things like Q4 or Q8. And as a beginner rule, Q4 is just usually easier to run and Q8 keeps more quality, but it needs more memory. If you're just getting started, Q4 is just a reasonable place to begin.
So I would start there. Then there is GGUF. It's a common file format for local models that make inference easier on normal machines like you and I have.
And in the Google AI Edge world, you'll see something called the LightRTLM. This is the model format and runtime path you care about when building on -device apps with LightRTM. So the simple map is this.
Hugging Face helps you find and understand models. Gemma is Google's open model family and Google's a trusted brand. I run my business on top of Google, so it just makes sense.
LM Studio helps you try models locally without much friction. Ohlama helps you run models locally in a way that a bit more technical people can plug into apps. GGUF is a common...
local model format. And Google AI Edge and Lite RTLM are the path towards shipping on -device AI products. That's what you need to know.
So let's talk about Google's open model family because I feel like there's a lot here. It's a bit overwhelming. And I'm just going to break it down so you understand what you need to know.
about the whole Google AI open model family. So Gemma is Google's family of open models, and Gemma 4 is built for the efficient, local, and on -device use. So they have Google 4 E2B, which is the smaller edge model for phone workflows.
You have a bigger E4B, Gemma 4 E4B, which is pretty much the most practical starting point. for most local tests. Then you have Gemma 4 12B, which is a middle ground with more capability for laptops.
And then you have Gemma 4 26B slash 31B, which is the stronger local workstation territory. That is the main model picker. Then, and a lot of people don't know this, there's specialized Gemma models that are just really useful to know.
So you have things like embedding Gemma, which is just for search. So specifically, it helps you turn text into embeddings, which lets you search by meaning. If you want to search your own docs or customer notes or support tickets, sales calls or knowledge -based localese, embeddings matter a lot.
Then they have something called Function Gemma, and that's a tool use and structured function calling. That means the model can help software take actions in a way more structured way. It is part of the path from the model gave me an answer to the model help the product do the next step.
Then you have a few more like Pali, Gemma, which is more vision focused. You have Shield Gemma, which is more safety focused. Then you have Gemma Scope, which is more understanding how models work under the hood.
You can leave most of the family alone on day one. The practical path, like on day one, if you're a beginner, start with Gemma for E4B, understand the workflow. Then you can move up or down or sideways, actually, depending on what you are building.
So the way I understand the whole Google AI ecosystem is you have Gemma as the open model family. You have Google AI Edge, which is the on -device AI development ecosystem. You have LightRTLM, which is the runtime for running languages models across all the devices.
You have AI Edge Gallery, which lets you try on -device models and see the experience just more directly. And if you need huge scale, things like strong managed infrastructure or frontier level cloud reasoning, you still have Gemini and Google Cloud that you can use or another frontier LLM that you can use.
The reality is a lot of big products and serious products are going to use a hybrid setup. They're going to use cloud for certain things and you're going to use local for other things. As an example, imagine a local AI tool for a professional service firm.
So the local model is going to read the sensitive drafts, checking for the issues. It's going to strip or summarize all the private details and prepare a clean version of the problem. Then when the customer wants deeper reasoning, a cloud model.
can help with the sanitized version. That to me feels like a more natural architecture than just putting everything into the cloud, which a lot of people don't want. You basically have local handling the private files as a first pass, and then cloud handles the heavy thinking when you need it.
A human can improve the work before anything important goes out. This is how I'm starting to think about building a lot of these products. Beyond Google Gemma, I'll give you a quick primer on the other families or other open model families you'll hear about and some of the pros and cons.
Lama is a metas model family, and it's probably the default. And it's probably the default open model reference point for a lot of developers because it's a pretty big ecosystem. The upside is the community, the tooling, the examples, support.
The downside is you still need to read the license and the model card, especially if you're building a serious commercial product. Quan is Alibaba's model family and has become very strong, especially around coding, multilingual work, long context, and agentic tasks. The China thing is real.
A lot of people use Quen because it performs really well. But if you're in an enterprise, a government, healthcare, finance, or sensitive data environment, you need to separate running open weights locally from sending data to a hosted service. And you need to check what your company is comfortable with or what you're comfortable with.
DeepSeq is similar in the sense that it's made a lot of people realize how strong Chinese -based open models really could be, especially for reasoning and coding. The upside is performance and cost. It's pretty cheap.
The trade -off is that some buyers will have procurement, security, or geopolitical concerns. So I'd be thoughtful about... where I'd use it, how I'd deploy it, even if you want to use it and go down that path.
There's also GLM, or people know it as z .ai. It's another one you'll see pop up a lot. I actually did an episode on it.
Especially if you spend time on Hugging Face and Ollama or just local model Twitter, you're going to see it a lot. The thing to know is that some of these models can be weirdly good for specific jobs. So I wouldn't ignore them just because they're not the obvious brand name.
You can test them. You can read the model card. You can check the license and just play with them.
But what you might deploy in the sense of for your business or for what you're doing might be very different. There's also Mistral, which is the European model family. I think they're based in France.
If you care about efficient models and they do a lot of releasing, a lot of practical developer use cases, they're pretty good. It's a strong model with a pretty builder -friendly posture. But the downside is the lineup is a little confusing.
Some models are open, some are commercial. So some question marks there. Microsoft also has their open model family.
It's called Fi. I think it's interesting if you care about smaller, faster, lower latency models. But for a lot of use cases, I haven't seen it work very well.
And honestly, there are new models showing up all the time. It feels like every other day. And that's why hugging face matters.
You're not going there just to find the big models. You're going to find these weird specialist models, these community fine tunes. quantized versions of stuff, these forks, and model cards that tell you whether something's actually usable for the workflow.
So you don't need to memorize all of this, but the takeaway basically is that there's these ecosystems and your job as a founder or just someone who's playing with these models is to pick a model family that fits your workflow. that you connect with that company, you like how they do things, and then go from there. You can play with a lot, learn a lot, and then pick a family.
So how do we make this whole thing real? If you actually want to run Gemma, here's how I would do it. I would start with LM Studio.
I would download LM Studio. It's free to download. You open the app.
You search for Gemma 4. If your machine is solid, try E4B. But if your machine is a bit slower, older, I would look for E2B.
And then I would look for the quantized version if you're using the GGUF path because you want the model to run just a lot more comfortably. Once it downloads, open a chat and just ask it something really simple. I would use a business prompt because I want you to feel the value immediately.
It's sort of an aha moment. Maybe it's something like... read these customer notes and turn them into a one -page memo about what customers are struggling with, what has changed, and what the business should fix this week.
And then just paste some customer notes or just fake customer notes, just if you want to see the value. The point of this exercise is just basic. The model is now running on your machine and you're using AI without sending that prompt to a cloud model.
I believe everyone should try that and feel what that is because I do think that it's just going to be a lot more common and it's going to unlock your brain in a completely new way. After that, go to LM Studio's developer section and start the local server because that just gets a lot more interesting because other apps can talk to the model on your laptop.
your computer becomes this little AI server. So you can have a script or a prototype or just an internal tool that can call the model through localhost and you get an answer back. I think that's when you start to see how products are going to get built in the modern age.
The second path is Ollama. So install Ollama and run Ollama pull gemma 4. Then run olama run gemma 4 colon e4b.
Now you have Gemma running locally from a command line. olama also gives you a local API port. I think it's on 11434.
It is useful because you can connect your own app or script to it. If you want to test a larger model later, you can try the 12 billion, 26 billion, or 31 billion versions, assuming your hardware can handle it. And you can ask an LLM if your hardware can handle it, or you can do it yourself and just suffer through the slowness and the pain of it.
The third path is Google AI Edge and LIDAR TLM. I would only use this path if I wanted to build an actual app and a model inside of it. For example, maybe I'm building a mobile app and the model is running on the phone.
Or it could be like a browser app where the model runs locally. Or maybe it's a desktop app with a private workflow. Or something on an edge device.
LightRTLM is designed for that world. Android, iOS, web, desktop, and edge environments. That is the path from local AI as a demo to local AI as a product.
So here's the hardware cheat sheet that I would use. If you have eight gigabytes of RAM, start small and keep the first test simple. But if you have something like 16 gigabytes of RAM, you can do some useful experiments with models like E4B and smaller quantized models.
If you have 32 gigabytes of RAM, you have way more room to do larger local workflows. If you have a strong GPU or a workstation, like a DGX Spark, the bigger models become just much more realistic. And for phones, I would think a lot less about model size and more about the job.
So can the model understand a photo? Can it summarize audio? Can it classify something quickly?
Can it help a worker in the field? Can it run without a strong connection? Can it do something useful inside the app before the user even thinks to ask?
Now, let's build the first workflow in our heads. So I would make a folder on your desktop called Customer Notes. And inside that folder, I would put 10 support tickets for a specific business.
Let's say it's a home health agency or a med spa. or a water damage restoration company. The notes might say something like, I tried to reschedule but couldn't find the link, or the technician didn't explain what happens next, or hey, no one actually confirmed my appointment, or I was charged twice here.
Then I would run a local model like Gemma and ask it to create a file called whatcustomersaretellingus .md, the markdown file. The output should include the repeated complaints, the exact customer language, the likely root cause, the part of the business that seems broken, and the one thing the operator should test this week, the high priority stuff.
This is a good first local AI workflow because it's useful and it's simple. What do you have here, right? You have this private messy data, the model runs next to it, and the output is a memo someone could actually use.
And then once you actually go and, you know, you're going to go and do this and get the output, you're going to, like the unlock I was talking... It's going to unlock something in your brain. You're going to see this pattern everywhere.
A folder of customer calls become a market research memo. A folder of support tickets become a product roadmap signal. A folder of PDFs become like a risk checklist.
A folder of drafts become a pre -send reviewer. This is why I always start with workflows before I'm fine -tuning anything. People here open model and immediately want to train their own model.
And I get it. I get why. I was actually the same way.
It sounds really cool. But I feel like that's an advanced move. The practical move, the beginner move, where you should start is just to find a repeated workflow first.
You pick one folder, one model, one output, and you run it like 10 times. you see where it gets confused. You see where you can improve the prompt and add examples.
You add a checklist and then you create a small eval. What's an eval? An eval is just a test that tells you whether the model did the job well enough.
For this workflow, for example, the eval could be really simple. It could be like, take the same 10 customer notes and run them through Gemma locally and then run them through a strong, cloud model, a frontier model, and then just compare the outputs.
And then you ask, did Gemma catch the same complaints? Did Gemma pull the right quotes? Did it follow the format?
Did it miss something? The comparison actually teaches you where local is already useful and where you still want that stronger cloud model and how you should think about the hybrid model I was talking about. That's really how I think about local versus cloud decisions.
Use local for private, repetitive, fast, offline, device native, and high volume workflow. Stuff that you want to run all the time. Use cloud for deep reasoning, giant context, broad research, and cases where the strongest model changes the quality of the answer.
So you use both when the product has sensitive data and hard reasoning. A lot of valuable products will work that way. Local first pass, you do the cloud escalation, human approval for anything important.
I think that's the way work's going to get done. So I want to give you three startup ideas where local AI actually matters. And these are the kind of businesses I would look for, niche, useful.
cash -flowing businesses that you don't need to raise venture for, and tied to a painful workflow. The filter is pretty straightforward. So I look for a customer with sensitive data, repeated review work, bad software usually, expensive mistakes, like mistakes that will cost them a lot, and a workflow that happens close.
to the device. That combination is like the interesting zone for me. So let's go through the three ideas.
I want you to steal these ideas. And at the very least, it'll get your creative juices flowing with how you can use local AI to run model, build apps and make money. Idea number one.
is a local QA reviewer for home health agencies. So home health agencies have nurses and caregivers and they go into people's homes and they write visit notes and updating care plans and dealing with billing and compliance. The paperwork is a pain.
It takes a lot of time if you've ever witnessed it in person, but it matters so, so much. Like a missing detail can create a billing delay and a vague note. can create extra admin work and a mismatch between the visit and the care plan can create a ton of risk and we don't want that.
So the first version is a local desktop app for the agency. The agency drops in visit notes and care plans and dictated transcripts. And then the model is going to review them before the submission and should look for flags.
So it's going to flag things like this note mentions dizziness, but vitals are missing. Or the caregiver described a medication change, but the follow -up instructions is pretty unclear. Or the note may not support the billed service level.
The buyer mostly cares about fewer documentation problems. before the billing or the audit or a supervisor review. So if you solve that, you have their attention.
Now, I don't want to just give you the idea. I mean, how would you actually grow this? If I was starting this business, how would I grow this business?
I would actually start it as a service. So I would find five small home health agencies and I would offer to review a batch of notes. I would do the review with AI, helping behind the scenes with the local AI.
And I would inspect everything manually with human beings, myself first. I would write down the 20 issues that keep showing up. And those issues become the checklist.
And then the checklist eventually becomes the product. So you have this wedge, it's pretty simple, where you're catching documentation problems before they cost the agency time or money, and then you build from there. I love this business and totally would start it.
The second startup idea is an offline field report co -pilot for restoration contractors. So think water damage or fire damage or mold remediation, things like that. Those teams are out there field taking photos, recording notes, documenting damage, and creating reports for homeowners and insurance adjusters.
I unfortunately had this, so I know a little bit about it. The job is actually pretty visual. It's also physical, right?
It happens away from a desk. And the report matters because the report becomes the handoff between the technician, the customer, the office, and the insurance process. So how would we build a product here?
The first version is a mobile app. So a technician walks through the property, takes photos, record voice notes, and the app drafts the report before they leave the site. So it can flag missing pieces while the technician is still there walking around.
You mentioned the basement, but there are no basement photos. You took a photo of ceiling damage, but there are no moisture meeting reading. Things like that.
Or maybe like... The affected room is like missing. Could be the homeowner explanation is way too technical.
Here's a clearer version they can understand. And the last part of that is underrated. In a stressful home damage situation, clear communication is part of the product, right?
So if you had that, that would be key. How would I grow this business? Well, I would pick one niche first.
I wouldn't go after everything. So say I'm going after water damage restoration. I would talk to owner operators.
I'd look at their current report templates, study the software they use, which is some old stack, and I'd build around the checklist that's already in their head. The demo is actually the easy part. Send me three old jobs and I'll show you how fast your techs could create reports.
If that works, then the product could expand from there. That's just the wedge, right? It can go into QA and estimates and insurance packets, customer updates, and training new technicians.
But I would start with the field report because it's specific and obviously super annoying. I just think that when you look at some of these old softwares that these people are using, I recently had some water damage at my apartment and I was seeing some of the software and it's antiquated.
It's stuff from the early 2000s. So I think that there's this opportunity to create local AI native software and wedge now. And that's why I said in the beginning, I think there's a 24 -month window and opportunity to do some of these products.
Let's go into startup idea number three. So startup idea number three, is a local pre -send reviewer for professional services.
So every professional service firm, or 99 .9 % of them, has a version of this workflow. Someone writes a client email, a proposal, a memo, a contract summary, an investment note, an HR note, and then asks someone, else to check it out before it goes out, like a review.
And it happens constantly. Law firms, accounting firms, wealth advisors, recruiting firms, even consultants have a version of this. So the first version is you build a local desktop app that reviews outbound drafts before they leave the company.
So for a wealth advisor, it could be flagging language that sounds like a guaranteed return, which is a definite no -no. For a law firm, it'll flag a sentence that sounds too definitive.
For HR, It's going to flag sensitive employee information that should stay out of the threat. For an agency, it flags a promise that the scope does not support.
And for an accountant, it flags a number that doesn't match the attached file. You'd be surprised how often that happens. The product is basically a second set of eyes for sensitive work.
It's basically schmuck insurance is the way I think about it. And maybe that would be the name, schmuckinsurance .com. Someone tell me if that's taken.
How would I grow the business? I would start with one vertical and one document type. For example, I would do email review for independent wealth advisors.
Not everyone, probably not the big banks to start independent wealth advisors. I would interview 10 advisors and ask them which emails make them nervous. I would collect.
anonymized examples. I would turn their real concerns into a review checklist and I would build a local tool that checks drafts against that checklist. Obviously, this is so sellable because the buyer understands this behavior and they already asked someone to check the draft.
So you're just basically giving them a faster first pass that lives closer. to their client data and internal rules. I love this idea and I hope a few of you take it.
By the way, if you're not building one of these ideas tomorrow, I still think you should learn local AI because it does change how you work with your own files. So I think just from a personal productivity perspective, it's still super, super helpful. So if you're working at a company, say, and you just want to be more productive, so you have more time to scroll TikTok or watch movies or hang with your family, make a folder called Local AI Lab and then put 10 files that matter to your work in that folder.
It could be anything from sales calls or meeting transcripts, old tweets, ideas that you have. Then run Gemma, whatever model you choose, to make it produce one useful artifact. And then ask it to create a weekly business pulse or ask it to find what's changed in customer conversations or meeting notes.
Ask it to group feature requests by the actual pain behind it. Ask it to review drafts and tell you what your audience keeps responding to. The key basically is to produce a file, a memo, a checklist, a brief, a report, or a review that you can reuse.
A chat answer is nice, but a useful artifact changes that workflow. This is the first rep I would recommend. A model reads the folder, the model writes the file, you inspect it, you improve the workflow, and then you run it again.
If you do that a few times, your brain really starts to connect the dots. You start noticing where private data is trapped in folders. You notice which reviews happen over and over again.
And you notice which workflows depend on someone checking a form, reading a note, comparing two files, cleaning up a report, or writing the same kind of memo week after week. Hopefully this episode got your creative juices flowing because once you see the pattern, you start spotting local AI businesses everywhere. You can learn enough of the map.
to spot where these models belong without turning yourself into a local engineer overnight. I believe some local AI belongs in the cloud and some AI belongs in the device. And a lot of the best products over the next couple of years are going to combine them both.
So if I was starting today. What I would do is I would run Gemma locally, I'd read model cards on Hugging Face, I'd learn the difference between LM Studio and Ollama, I'd play with Google AI Edge, and then look for one boring workflow where local AI actually makes the product better. Those categories are things like private data, offline work, camera or audio context, or low latency, or if there's a high repeated API cost.
I'll see you next time. treating it like a model benchmark conversation and start treating it like a product conversation. You have to ask yourself, where is the work happening?
Where is the data? Where is the device? Where is the trust issue?
Where is an annoying review loop? And then you answer those questions and you just start seeing the ideas. So overall, I hope you understand a little about the core things you need to understand about local AI, some of the models, some of the apps you need to download, some of the workflows that you can build, and some of the business opportunities that exist.
I just don't see that many non -technical people playing with local AI. Over the last two months or so, I've gotten deeper and deeper into it. Like I said, it's been connecting the dots and I'm grateful for it.
I hope you have a creative day. I read every single comment on YouTube. and respond to most.
So I'll see you in there. Share this with a friend who you think could benefit from understanding local AI in a clear way. And I'll see you next time.
Happy building.
The Hook
The bait, then the rug-pull.
Greg Isenberg opens with a claim: local AI is about to create a wave of business opportunities most non-technical founders don't know is reachable yet. He spends the next 38 minutes building that map himself, then hands over three startup ideas built on top of it.
Frameworks
Named ideas worth stealing.
03:13list
The Four Pieces of the Local AI Landscape
Model (the brain file)
Warehouse (Hugging Face)
Software (LM Studio / Ollama)
Workflow (the product)
Splits the intimidating local AI space into four legible layers, and argues the model is just the ingredient, the workflow is the actual business.
Steal forany explainer page that needs to make a technical stack legible to a non-technical buyer
11:18list
Gemma Model Picker
E2B — edge / phone, fast tasks
E4B — best first rep, normal laptop
12B — more capability, laptop/workstation
26B MoE / 31B — bigger, denser, quality plus hardware
Google's sizing ladder for choosing a Gemma model by hardware and job rather than by hype.
Steal forany vendor's own 'which size do I need' picker page
25:06model
Local vs. Cloud vs. Hybrid Decision Rule
Local: private, repetitive, fast, offline, device-native, high volume
Cloud: deep reasoning, giant context, broad research
Hybrid: local first pass, cloud escalation, human approval
A three-way routing rule for deciding where intelligence should live in a product, rather than defaulting everything to the cloud.
Steal forany product decision that routes between a cheap/fast path and an expensive/strong path
26:41list
The Filter for a Local AI Business
Sensitive data
Repeated review work
Bad software
Expensive mistakes
Work near the device
Five markers checked together before treating a niche as worth building local AI software for.
Steal forscreening any vertical-SaaS idea, not only AI-powered ones
CTA Breakdown
How they asked for the click.
VERBAL ASK
00:22link
“quick shout out to Google for sponsoring today's episode and for caring about local AI and open models for entrepreneurs.”
The sponsor credit is a single spoken line inside the cold open, about 10 seconds, with no accompanying on-screen graphic. The real conversion point, a link to a full written guide on running local AI, appears only in the video description and is never spoken on camera.
Add Modern Creator as a preferred source and Google shows you more of our breakdowns in Search, Top Stories, and AI Overviews. It only changes what you see, and you can undo it in your Google settings anytime.
Add to Preferred SourcesOpens your Google source preferences with us pre-loaded. Tick the box and you're done.
A two-person breakdown of OpenAI's top-tier model that skips the game demos and goes straight to code audits, nine money-making agent prompts, and a Raspberry Pi speaker built and shipped in about 30 minutes.
A solo walkthrough of five open-source GitHub repos getting traction right now, an AI writing editor, an agent-run CRM, a video-editing agent, a skill security scanner, and a phone-controlling harness, each with the exact install command and the first small workflow to try.
Greg Isenberg names the role he thinks AI agents are about to make the most valuable job in tech, and hands over the folder structure, tool stack, and 30-day plan to become one.
Greg Isenberg and developer Vinny break down WebMCP, the experimental browser feature that lets any AI agent search, compare, and buy on a website without scraping the page, then price two startup ideas built on it.
Cloudflare quietly built the payment rail for AI agents to pay tiny fees for web resources, and that rail is about to spawn a whole generation of businesses selling clean data, agent-readiness, and packaged expertise.
00:56
01:07
01:50
02:22
02:45
04:06
04:40
05:04
05:39
06:29
07:43
08:20
08:56
09:33
10:10
10:29
11:51
12:24
12:57
13:30
14:03
14:20
15:44
16:18
16:52
17:24
17:55
19:01
19:30
20:00
20:41
20:58
21:14
21:44
22:19
22:37
22:47
23:56
24:52
25:49
26:18
26:57
27:13
27:22
28:38
29:08
29:24
30:33
31:00
31:28
31:56
32:10
33:28
33:59
34:31
34:47
36:05
36:36
37:08
37:39
38:20
38:37
22:52
24:43
35:18
28:58
44:21
34:10