21:08Riley Brown · Tutorial
Building a Real AI Jarvis with Cursor and GPT Realtime 2
A step-by-step build of a talking, screen-controlling desktop AI companion — no coding experience required, just long prompts into Cursor.
July 1stThe argument in one line.
OpenAI's real-time voice mode turns Codex into a hands-free assistant that can read your screen, use your existing automations, and run tasks in the background while you keep talking, but it is built to never send, spend, or act outward without explicit permission in the moment.
Read if. Skip if.
READ IF YOU ARE…
- You already use Codex or ChatGPT's agent tooling and want to know whether hands-free, spoken control is actually usable yet.
- You're curious what a coding agent can and can't do unsupervised, specifically around sending messages, spending money, or acting on private data.
- You want to see a real (if rough) first-use test rather than a polished, rehearsed product demo.
SKIP IF…
- You want a step-by-step setup tutorial — this is a live first-impressions test, not an installation or configuration guide.
- You're looking for a verdict on production reliability — the presenter explicitly says he's only spent 30-45 minutes with it and plans a deeper follow-up later.
The full version, fast.
OpenAI added a real-time, talk-back voice mode to Codex that differs from dictation because the agent can hold a live conversation while it keeps working. The video walks through triaging a Gmail inbox for customer complaints, turning the results into a shareable anonymized doc and texting it to a teammate, controlling an Excalidraw whiteboard by voice to map the feature's capabilities, spinning research and build tasks into separate background chats without interrupting the live conversation, scaffolding a Swift iOS app clone in an in-app simulator, and finally starting a voice session from the ChatGPT iOS app that reaches the same Codex task running on a Mac. The agent volunteers its own guardrails unprompted: it can't swap its own model, and it can't silently take over apps, send messages, spend money, change permissions, or access private data unless explicitly told to in that moment — and a user-built skill can't override those limits either.
Free for members
Chat with this breakdown — free.
Sign in and you get 23 free chat messages on us — ask for the hook, quote a framework, find the exact transcript moment, generate a markdown action plan. Bring your own key when you want unlimited.
Create a free account →Where the time goes.
00:00 – 00:52
01 · Jarvis in Codex
Cold open framing the new real-time voice mode as different from dictation, plus a three-item agenda for the demo: email/productivity, browser use, and an iOS app.
00:52 – 02:20
02 · Email triage demo
Starts a live voice chat, asks the agent to find and draft responses to customer complaints in Gmail, and has it open all flagged threads in the Codex browser.
02:20 – 04:44
03 · Summaries and texting
Agent turns the complaints into an anonymized, shareable Google Doc, makes it public, texts the link to a named teammate, and adds a follow-up note before sending.
04:44 – 06:32
04 · Excalidraw voice control
Switches to a new voice session with an Excalidraw board open; the agent edits the whiteboard by voice to map out what the feature can and can't do.
06:32 – 07:30
05 · Background research tasks
Asks for in-depth research on the feature itself, which the agent spins into a separate background chat, then reports back and opens the finished Notion doc.
07:30 – 10:31
06 · Limits and safeguards
Agent explains its own guardrails: it can't swap its model, can't send messages, spend money, or access private data without being asked, and a user skill can't bypass those limits.
10:31 – 13:25
07 · Build iOS app by voice
Invokes a build-iOS-app skill to scaffold a Swift Notes-app clone in an in-app simulator, spins off two more app-clone tasks in parallel, and requests live styling changes.
13:25 – 15:36
08 · Phone remote control
Switches to the ChatGPT iOS app to start real-time voice remotely, connecting to the same Codex task running on the Mac and asking for an inbox summary and a build-status check.
15:36 – 16:27
09 · Wrap-up and next steps
Recaps everything demoed in roughly 30-45 minutes of first use and promises a deeper best-practices video after more testing.
Lines worth screenshotting.
- Codex's real-time voice mode differs from dictation because it holds a live, two-way spoken conversation while the agent keeps working in the background.
- The voice agent can read whatever's already on screen — a doc, sheet, or diagram — and act on it without being told to look there first.
- It can spin off a separate background chat mid-conversation so a long task like research or an app build runs elsewhere while you keep talking in the main session.
- By design, it will draft a text, email, or Slack message freely, but it won't actually send one unless explicitly told to in that moment.
- A user-created skill cannot grant itself permission to send messages or take consequential actions — only an explicitly authorized automation can do that.
- The live voice session can't swap its own underlying model; that setting lives in the task configuration, not the conversation.
- OpenAI paired the feature with remote access from the ChatGPT iOS app, so a phone can start a real-time voice session that controls a Codex instance running on a separate Mac.
- A new voice session started from the phone isn't automatically a new task — it can stay attached to the same desktop workspace unless a separate one is explicitly requested.
- Multiple parallel build tasks can be requested in the same breath — two extra app clones were spun into their own chats while the first app kept building.
- Opening an in-app browser tab before starting voice mode caused that tab to auto-close in this test, a rough edge worth expecting from a feature this new.
What Codex's live voice agent will and won't do on its own
WHAT TO LEARN
A hands-free coding agent can triage inboxes, edit documents, and run parallel background tasks by voice, but by design it never sends, spends, or acts outward without your explicit say-so in the moment.
02Email triage demo
- Voice-triggered inbox triage worked end to end: the agent found unresolved complaints, drafted responses to all of them, and opened every flagged thread as a tab without further prompting.
- Ambiguous instructions get filled in reasonably — told to open "all of them" in the browser, the agent opened every flagged thread at once rather than asking for a number.
- Working full-screen while the agent runs removes the need to babysit a chat window; a visible progress log (creating file, thinking, getting documents) lets you step back while it works.
03Summaries and texting
- The agent can generalize a document request on the fly, anonymizing names in a summary and formatting it as a shareable doc without being given a template.
- It chains actions across apps in one spoken request — make a doc public, then text the link to a named contact, then add a follow-up line before it's sent.
- Confirmation checkpoints still show up naturally in the flow, like asking "should I text her the link" even when it already has permission, giving a final say before an outward-facing action.
04Excalidraw voice control
- The agent can read and edit a live whiteboard by voice, adding and reorganizing content mid-conversation rather than just describing what it would do.
- Opening voice mode while a browser tab is already open can auto-close that tab — a rough edge worth expecting when combining browser-based tools with a new voice feature.
- The same running session can keep the user talking while spinning a separate task into its own chat, so long research jobs don't block the live conversation.
05Background research tasks
- Long-running work gets automatically split into a background chat when it would otherwise interrupt the live conversation, and its progress can be checked conversationally without switching windows.
- The agent applies a previously stated formatting preference (tabs, hyperlinks, a specific style) without being re-taught it in the moment, drawing on prior instruction rather than a fresh template.
06Limits and safeguards
- The agent volunteers its own limitations unprompted when asked, rather than only surfacing them after a failed attempt: no model swapping, no silent app takeover, no unsolicited messages, spending, or private-data access.
- Drafting an action — a text, email, Slack message, or post — is allowed by default, but actually sending it always requires an explicit, separate instruction in the moment.
- A user-authored skill cannot bootstrap its own permission to bypass those safeguards; automations still require explicit user authorization to send or take consequential actions.
07Build iOS app by voice
- A named skill can be invoked by voice to scaffold a real app (a Swift Notes-app clone) and run it live in a simulator, with the agent stating its plan before starting.
- Multiple parallel build tasks can be requested in the same breath — two additional app clones were spun into their own chats while the first kept building, without derailing the live conversation.
- Style and behavior changes can be requested against a running build by voice (switching a default to light mode, upgrading a component's visual polish) and then verified by asking the agent to demo it.
08Phone remote control
- Remote access lets a phone start a real-time voice session that reaches a Codex instance running on a separate Mac, rather than being limited to the desktop app.
- A phone-initiated voice session isn't automatically a new task — it can stay connected to the same task and workspace already running on the desktop unless a new one is explicitly requested.
- State is shared across devices, not siloed per session: a remote session can reference and report on tasks that were created earlier from the desktop.
Terms worth knowing.
- Real-time voice (Codex)
- A live, two-way spoken conversation mode in Codex where the agent can talk back and keep working between turns, distinct from voice dictation which only transcribes what you say.
- Codex browser
- An in-app browser panel inside Codex that the agent can open, navigate, and read from to complete tasks, without switching to a separate window.
- Skill (Codex)
- A pre-configured, reusable workflow the user has set up in advance that the voice agent can invoke by name during a live conversation.
- Task
- A separate background chat or session in Codex that can run independently of the main voice conversation, letting long jobs finish without blocking live talk.
Things they pointed at.
00:52toolGmail
02:39toolGoogle Docs
03:36tooliMessage
04:44toolExcalidraw
06:47toolNotion
10:31toolSwift / iOS Simulator
10:45toolGoogle Images
13:25toolChatGPT iOS app
Lines you could clip.
04:41
“This is Jarvis. You're literally Jarvis.”
02:59
“I don't even need to have it open. I can just kinda have this real time voice just kind of, like, hovering over everything.”
07:52
“I also can't silently take over apps, send messages, spend money, change permissions, or access private data unless you ask.”
10:25
“You're like my little buddy that can fully control my computer.”
Word for word.
Read-along
Don't just watch it. Burn it in.
See every word as it's spoken — crank it to 2× and still catch all of it. The same dual-channel trick behind Amazon's Kindle + Audible.
metaphor
00:00OpenAI just released real time voice directly inside Codex, and this is the closest thing I've seen to Jarvis. And this is actually just a voice agent that connects to all of my tools, can do basically all of my work, build and edit applications, and fully control my computer.
00:16And arguably, the coolest part about this new feature is you can use real time voice on the ChatGPT app to control codecs on your computer. And I'm gonna show you exactly how to do that today.
00:27In this video, I'm gonna be using codecs with real time voice for the first time. I'm going to test codecs real time for email and other productivity workflows. I'm also going to use the in app codecs browser and Excalidraw with my voice, then I'm going to build an iOS app just by speaking, and then finally, I'm going to test out real time voice on the Chai Chi PT app and control Codex from my phone.
00:52Let's go into Codex. At the bottom right, you will see this button right here, which is start new voice chat. And this is what I call the new Jarvis feature, which is different than dictating.
01:01Right? Jarvis is real time voice. Let me show you.
01:08Hey, buddy. Can you hear me?
01:12Yeah. Loud and clear. What's up?
01:14Yo. I need you to go to my email right now. I want you to find any complaints for customer support.
01:20I want you to draft a response to all of them, and I want you to open them inside the Codex browser.
01:27Sure. I'll take care of that.
01:34And as you can see here, right, and notice here I can actually, um, mute the microphone, so now I can speak to you as I use it. But it can use all of the skills that I've set up when using Codecs, and I've set up so many skills. Right?
01:48I have all of these skills that I've been setting up for the last six months, and now I can use them in real time voice, which is just absurd. Amazing.
01:58Yeah. Just make sure these are open in the Codex browser, please.
02:02Got it. I'll use the Codex browser. Oh,
02:06there we go. Look. It opens up all of them up at the same time.
02:10Here we go. And boom. Look at that.
02:14Okay. Those 12 complaint braids are open in the Codex browser, and I saved draft for 11. Okay.
02:20One thing that's really cool about this is I can close the side panel, and I can just enter this full screen mode. And are are you still there, Codex? Yep.
02:31I'm right here. Very cool. Um, can you just actually turn all of these into don't use any names.
02:36Just create a new Google Doc for these and open it up in a new tab in the browser and and, um, just so I get a good summary of the issues going on, uh, with our product.
02:48Sure. I'll take care of that.
02:53And I love that I can just see this right here. It's creating file, thinking, getting documents, and I don't even need to have it open.
03:02I can just kinda have this real time voice just kind of, like, hovering over everything. It does take a bit to get used to, but I think I will.
03:12Done. I created a doc that groups the issues without any names, and it's open in a new Codex browser tab.
03:19Take a look. Very cool. Can you actually make this document public?
03:23And then I want you to text it to Emily Lambert for me.
03:29Okay. Checking that now. I'm setting the doc to public and sending the link.
03:37This is pretty cool. So I'm gonna open up my messages.
03:42I've made the doc public. Should I text her the link with the Yeah. Here's the customer support issue summary.
03:49Yep. Please do that. On it.
03:55Sending that now.
04:10Amazing. So it just sent a text message with yeah.
04:15This is very cool. Hey. Can you actually just add a little extra note there that says, like, hey.
04:19Can we get these fixed by the end of the day?
04:23Sure. I'll add that.
04:27Do you want me to send can we get these fixed by the end of day? Send it. Send it.
04:33Got it. Sending that. I'm sending it now.
04:41. You're literally Jarvis. Okay.
04:45So now let's test this new codecs voice feature, which I call Jarvis, in a new environment. So I'm gonna open it up in a new chat, and in the browser, I have Excalidraw open. And I'm just gonna call this Jarvis features, and we can start up Jarvis.
05:02Yo. What's good? Are you there?
05:08Pierre, what are you working on today?
05:12Okay. So that was really weird. So I if you have a browser open.
05:17Before I open up this voice mode, it'll, like, automatically close that browser tab, which is pretty interesting. Um, okay.
05:24Ignore that, Jarvis. Can you please look at the in app browser right now?
05:30Sure. Taking a look. Checking the current in app browser state.
05:38There he is.
05:40The in app browser is open to your Jarvis features Excalidraw board. Okay.
05:46You. Thank you. For context, I want you to know that I'm talking about the new GBT voice mode, which is the exact mode that we are using right now.
05:58And so this is a new Codex voice, and it's real time voice. Can you please just give me some three more things that I could add to what it can do?
06:07On it, I'll check the current capabilities against your board. Here are three you can add. One, it can look at your screen and act on what's there, like a doc, sheet, or diagram.
06:19Two, it can research something live while you talk and give you the answer conversationally. And three, it can build and edit real things in the background,
06:27like docs, slides, code, emails, or workflows. That is actually very interesting. I'm gonna start off doing research.
06:33So, uh, you know what we're talking about now. Right? We're talking about the new voice feature.
06:37I want you to go off, do some research, and I want you to create a new Notion doc. Just put this in your notes in Notion, um, and use the you know, the formatting I like with tabs in Notion. Um, and I like hyperlinks.
06:50Just do in-depth research on this, please. Sure. Actually, wait.
06:54Wait. I wanna I kinda wanna continue talking to you. So can you actually do this in a different chat?
07:01Yeah. I can handle that separately and keep this conversation open.
07:12Okay. So wait. To get this straight, did you just create a new chat so that we can just continue talking.
07:18Right?
07:19Yes. I spun that into a separate task so we can keep talking here.
07:24Wow. Okay. So it can open up new chats.
07:27Yeah. Okay. So we'll wait for that research to come back.
07:30Is there any limitations? Like, if you look at your environment, I noticed I can't switch the the model that you are. Is there is there any limitations on what it can't do?
07:39Checking the current limits now. Yes. This live voice chat is intentionally more constrained than a full task workspace.
07:47You can't swap my model from inside this live conversation. That's set by the task configuration, and I also can't silently take over apps, send messages, spend money, change permissions, or access private data unless you ask.
08:00For longer research or building docs or code, I'll often split it to a separate task. Really tap into that, like, um, you can't send messages unless I ask. What does that mean?
08:11It means I can draft a text, email, Slack message, or social post for you, but I won't actually send anything unless you explicitly tell me to. Same with No matter what. Okay.
08:22So it'll always make sure no matter what. Like, there's no way I can create a skill which bypasses that.
08:29Checking that. Right. A skill won't override those safeguards.
08:35You can set up an automation you explicitly authorize to send under your rules, but a skill can't give itself permission to send things or make consequential changes on its own. I see. That makes sense to me.
08:48Can you give me a progress report on that other chat session? And if it's done, I want to open up that Notion doc in a new tab here.
08:57Sure. I'll check the status and open it if it's ready. And by open it, I just I just mean I want it in the Codex browser.
09:04Got it. I'll open the Notion note here as soon as the task finishes.
09:09Hey. Um, using the browser use thing, can you add a use case to this board, and can you add, um, spinning up new sessions under use cases under background research?
09:21Adding that now. I'm adding spinning up new sessions under background research now.
09:39Hey. Can you also say, um, controlling Excalidraw?
09:46Yep. Adding that as well.
09:50The research note is done and verified, and I'm adding controlling XCalidraw now. Then I'll open the Notion note.
10:00Oh my god. It opened it.
10:05Look at this. It literally created this research report with the tabs, the exact style that I like. Here's the launch video.
10:14It found it on x. I can open it up inside Codex.
10:18Play my day list. One moment.
10:21Anything you'd like me to tweak or add to that note?
10:25Nah, dude. This is actually insane. You're like my .
10:31Hey. Um, can you use the build iOS app skill? And, um, I wanna create an iOS app.
10:37Um, can you please start create, uh, yeah. I want you to create a Swift app that is like an exact replica of the Apple Notes app, and I want you to run it in the in app simulator in Codex.
10:51Got it. I'll start by checking what's possible for an iOS simulator run. And you know what?
10:56While you're at it, can you please spin up two more chats? And I want one to be a Notion clone and then the other one to be a Trello clone. Spin off two more chats and do that for me, please.
11:08Sure. I'll branch those out too. Pipelines configured, and I'm spinning up two separate iOS build tasks, Notion style and Trello style.
11:19Both chats are set up with the same simulator target. Very cool. Now let's talk about this.
11:25What's your plan for this Apple Notes app?
11:29Sure. I'll start by mapping the core screens and data model.
11:33This is pretty cool. While that's working, do you think you can, like, open up, uh, the browser and and, like, search up, you know, in a few different tabs, like cool note UIs?
11:43I wanna get some inspiration, and I want you to look these up on Google Images just to give me some inspo while you work.
11:50Yep. I can pull those up. I'm opening a few visual references in new tabs now.
11:59Thank you for doing this. How's the notes app coming along? Wait.
12:02What did you what was that browser you just had open? Why did you close it? I'll check that.
12:08The browser was the in app simulator preview for Memo Canvas. It was closed by the browser session cleanup, not intentionally. Alright.
12:15Well, please close all the other tabs and then open that up again. Keep it open. Got it.
12:20I'll do that.
12:25Done. All other tabs are closed, and the memo Canvas simulator preview is open. Okay.
12:31Yeah. Can you show me? Just go ahead and use the app for a bit.
12:34Test out
12:35the features on this. I'll do that now.
12:48You know what? Um, let's continue to make some styling adjustments.
12:52I actually want the default to be light mode. Change it to light mode, and, um, please make the bottom tab, like, the liquid glass tab at the bottom higher quality, please.
13:02Got it. Making those updates now.
13:07Wow. This is just truly insane. Thank you, Codex.
13:11I'll pass that along. Done. Memo Canvas now defaults to light mode, and the bottom navigation uses a polished liquid glass capsule with clear spacing, depth, and a strong compose action.
13:23You're welcome. Okay. Check this out.
13:25OpenAI also announced right? They also announced that you can use your ChatGPT voice in Codex from an iOS app with paired remote access. Android support coming soon.
13:36So if we open up ChatGPT, right, we can also connect real time voice from our phone to this app. So let me see how this works.
13:44I've actually never done this before. This just came out. So hey, buddy.
13:52Um, so I'm using you for the first time. I'm using this on remote, and so apparently, you're working on my Mac.
14:00Can you is this automatically create a new session? What's going on here?
14:04Let me check how the session starts.
14:07You're in a new voice session, but it can still be connected to the same task in your Mac's workspace.
14:13I only create a separate task if you explicitly ask Okay. So let me create a new task real quick. I want you to create a new task, and I just want you to just give me a quick summary of the support emails.
14:25Just I just wanna see if this works.
14:28On it. I'll take a look. I'm creating a separate task for a concise summary of the support inbox using the Gmail work flow.
14:36The summary task is set up in a separate chat and checking the inbox now. What is the chat called? What is the chat called?
14:44Checking that now. It's called summarize support emails.
14:50Okay. So I see it. Nice.
14:53Okay. So now I'm I'm basically just talking to a voice agent that runs, but I can you can create new chats in the background, basically. Right?
15:00Is that how this works?
15:02Let me check how chats are created from here. Exactly. You can keep talking to me here, and I can create separate background tasks that show up as their own chats while I stay here and coordinate what comes back.
15:14Wow. Can you see my so you can, like, look at my other tasks. Like, can you check the status of my Memo Canvas UI Polish app that I was making earlier?
15:23Yep. Checking that now. That's the Memo Canvas app task you made earlier, and it's active.
15:28The latest light mode build with the liquid glass tab bar is still running in the simulator.
15:33Okay. Thank you. This is actually blowing my mind.
15:36Okay. So I'm absolutely blown away. So here's just a list of what we did.
15:41This was basically the first time I tested this. It can do it. It can fully control your computer.
15:45It can spin up chats in the background. You can use real time voice on your phone, and that will talk to your computer, and it can spin up chats, and you can fully just control your computer in real time voice directly from the ChatGPT iOS app.
15:59This is pretty insane. So, obviously, I've only spent, like, thirty minutes to forty five minutes using this, so I'll probably do a much more in-depth video once I kind of learn the best practices. Right?
16:09This was just me testing it, but I wanna learn actually how do I use this in my business, how do I use this to stay focused and be more effective. And so I'm just gonna test it for the next few days, and then I will make a more complete guide style video on exactly how I use this in my business, and I'll see you here for the next one.
The Hook
The bait, then the rug-pull.
OpenAI slipped a live, talk-back voice mode into Codex, and the presenter spends the video finding out just how far it runs on its own — triaging a real inbox, controlling a whiteboard, building an iOS app, and eventually talking to it from his phone while it works on his Mac.
Named ideas worth stealing.
08:44list
What Codex Voice Can / Can't Do
- Can: real-time voice, use my skills, open browser, look at screen, research something live, build and edit things in the background
- Can't: start with browser open, switch its own model, silently take over apps, silently send messages
A running whiteboard the presenter builds live by voice, splitting the feature's capabilities and hard limits into two columns as he tests each one.
Steal forA quick capability/limits cheat-sheet for any new AI agent feature before recommending it to a team
How they asked for the click.
VERBAL ASK
15:36next-video
“I've only spent, like, thirty minutes to forty five minutes using this, so I'll probably do a much more in-depth video once I kind of learn the best practices... I'll see you here for the next one.”
Soft close, no subscribe/link ask — just a promise of a deeper follow-up video once he's tested it further.
Visual structure at a glance.
