OpenAI's Realtime API Upgrades Siri with Cursor AI Integration
Let's become an AI prompt engineer📅 2024-10-07
Transcript
[00:00] so now we have a live Connection in our local server to open AI realtime API hey Chad jbt I wanted to show them today how to use real-time API to make a better Siri does that sound good hey that sounds like a fantastic idea leveraging a realtime API could really enhance Siri capabilities sounds good let's go ahead and jump into that in today's video where I walk you step by step from Step Zero to step done how to build out a real-time API application so we can get this no latency type of conversation
[00:30] ation like we see here does that sound good Chad gbt absolutely let's dive right into it if you have zero to no coding experience don't worry I'm making this as easy as possible we're going to be using cursor AI here we're going to use realtime API here let's go ahead and proceed some of youall right now are like Corbin why did you just show advanced mode by chbt that is because that is what realtime API is when leveraging realtime API here we have the ability to either do audio to text audio to audio with little to no latency through API this is extremely powerful therefore I have three major goals for this video first one I'm going to show
[01:01] you how to create a react based front end so we can just do start recording or start conversation with real time API the second one is I'm going to show you how to create a server but that is run on your machine so we don't have to deal with actually paying for a back end from Google or Amazon now of course everything I show you today can translate to using Google cloud or Amazon in the context of running this kind of application but I wanted to make this video as simple as possible so you can do it yourself so that means the third thing which is going to show you to the the limit of what we can do locally on our machine how to leverage
[01:33] realtime API sound good let's jump in let's go and get started here this is going to work with any IDE integrated development environment cursor AI repet vs code whatever your secret sa is whatever you like to do proceed the tutorial here is start from finish so we're going to start at the very beginning we need a project we need to be able to create this new Seri click this right here we're going to hit terminal let's begin now one thing I got to point out before we continue creating this directory if you're coming from the background of you've never coded before you've never used react before or you've never done this kind of development flow
[02:04] before I encourage you to check out this video I put in the description down below it's curs your AI beginner's guide it's around 30 minutes long and the reason I encourage this because one you can watch the video in its entirety learn more but two I have a Google doc here that is very important this Google doc gives you everything you need to know with the erors you can run into I give you a Chad gbt chat in this doc that gives you very specific answers for everything you're about to see here so to answer that question right off the bat I'm getting in air with mpm commands what's going on Corbin you didn't teach this that's because you didn't install
[02:35] no. JS which is all explained in this entire doc back to the video the commands you're about to see can be found in that Google doc as well but we're going to do them here together I'll make sure to try to put them in the description but they can be found in that Google doc actually honestly I'll link the Google Doc in the description down below but here we go so we're going to create a directory first is we're going to be storing all of our code let's go and give it a name this is how we're going to reference it we'll say Siri better enter so now we've created our repository we can go on and reference it to reference it we could use a CD command or alternatively because this is the first time we're launching it let's go into this opening
[03:05] folder with this opening folder here I'm going to go and scroll down and find it right here sir better open all right perfect so we are in here this is completely blank here I'm going to make sure that I can open up my terminal commands here so open like this real quick there we go let's open up a terminal window so simply search on your computer terminal in the shipa up here alternatively you can use it within the actual cursor AI platform itself or IDE you can simply come down here click this terminal I like opening it up cuz you're going to have multiple terminal Windows opened as you will see here so open up
[03:36] terminal Corbin you talk too fast this is too fast okay go to the little settings instead of one put me at0 75 if that's still too fast put me at 0.5 I get people telling me I'm too fast too slow pick one y'all real quickly I'm have to rename the project as I realize some commands have issues when there's capitalization within the actual project name don't ask me why I'm going to go Siri better better all right open again with this open let's go to a terminal here and point it towards this directory to do so and as outlined in that Google doc we're going to do CD Siri better enter once you're in Siri better okay
[04:09] I'm not I'm not I'm going to stop saying that don't worry we're going to go ahead and run a line here that's actually going to create a front end for us is going to install all the dependency that are relevant in this context it's going to be a react based front end I'm going to hit this line here and all I want you to think of when you hear react is no we're not reacting to something this isn't like top 10 reactions uh this is going to be a a web package we can use to build out most mostly the website that you see when it comes to user interface you go to YouTube that's a front end when you go to instagram.com if you go to the desktop version that's a front end that's a front end right so we're also going to be creating a
[04:39] backend in today's video as well but this is going to be a local backend EG we're going to be running the server in our machine so we don't have to deal with Google or Amazon when it comes to cost and to make this as simple as possible to be even more clear on that because of the fact of how the real-time API works this is a websocket this isn't a endpoint in the sense of how we've seen in the past where it's like reference the gbt 40 endpoint you send it like a web hook data and then you get data back because this is a websocket we need a constant server in order for the no latency to incur where we get the
[05:10] answers and responses fast as said here the real time API is a websocket that doesn't make sense put in a Chad gbt chat it'll make more sense maybe Chad gbt is a better teacher than me so now that we created our react front end here in the sense that that we had the files and we can access them source app.js is where we're going to be putting all of our code when it comes to what we're about to Showcase in today's video Let's create a folder that is dedicated for the backend and a local server to do so first off make sure that we're in the correct directory so Siri better and then on top of that we're going to create another folder within this directory that is dedicated for the back end so we're going to use these two
[05:41] commands mkdir again we'll call it backend and then we'll navigate to The backend enter there we go back end created here let's proceed next within the back end we're going to do npm init Dy enter there we go furthermore we're going to make sure we have everything that is relevant to what we're doing today with websockets installed as depend dependencies npm install Express ws. EnV enter we're going to do a simple line here just to create the file of ajs we could also right click in the section there and create it or we going do touch. server.js perfect click it blank
[06:12] file next we're going to create an EMV file here so that we can secure sensitive information this is going to be specific for the open AI key we'll use in today's video so again we're going to do touch. EMV if you run into any issues I've always encourag you to use 01 mini or 01 preview or just a chat gbt 4 chat or the chats that I provide personally that are specialized for this to help you out to learn more let's keep going purely the use case here is to provide very high sensitive information example this is going to be an API key from open AI we'll go ahead and format it like open AI API key which kind of makes sense and then go ahead and just
[06:42] paste your API key here so I'm going to go and do that once you do that command save make sure it's one line you don't have to add quotation marks or anything just one line it should start with SK proceed perfect let's go and create a another terminal window here cuz this one's going to be dedicated towards our local backend so I'm going to hit enter a couple times here you'll see what I mean here we're going to actually run this pretty soon here so I'm going to create another terminal window here that's going to be dedicated for our react front end these two processes are separate EG when you launch an application the rendering of the front end and the application processes in the
[07:14] back end are two separate runs in the context of local development as always make sure you're in the directory so for us it's Siri better and now we get to the code here which I'll be honest with y'all is very lengthy because when it comes to realtime API there is a just a ton right the this is going to be very complex for obvious reasons it's not like we can just create Siri with a simple web hook this requires a little bit more complex logic here so I'm going to walk you through step by step what I did in the code that I'm about to show you right now I'm going to see if there's a way I can just share it with you as well whether that's a get or
[07:45] whether that's a Google doc you can just kind of copy and paste over for now though let me show you the code let's begin right off the bat if you're using cursor AI you already know what you can do you can like command a command L learn about the code in the little chat but I'll go to explain it as best as possible from my expert what's incurring here therefore the idea here is that since it's a websocket which EG means that it's a constant flow of information within a server we have to make sure that there is something that is handling these operations this isn't just a post or a git in the context of a web hook we
[08:15] need to actually have a live server incurring here now what's great about this server.js is that you can run this locally on your machine you don't have to worry about Google or Amazon or any other server provider so the first major thing you need to know here is the port we're identifying this is important think of 4,000 as what we usually typically associate with functions or backend logic this is specific to if you've ever run Firebase emulator you'll know that's a 4,000 Port as well on top of that the one that's probably more obvious and easier to understand is like the Local Host 3000 that's typically the front end of a react app therefore we
[08:47] need to make sure that Local Host 3000 your front end isn't like Local Host 4,000 4,000 like there can't be conflict there so scrolling down here let's come down to the open AI logic here so right off the bat you'll notice process. env. open I key this is where we're referencing that EnV here this is important in any development not really important here I mean I guess it's kind of important so you don't see my API key but this is just important in general when deploying applications this protects you as the EnV typically isn't always pushed to your git isn't always pushed to full on production this is how we reference high-risk variables
[09:18] obviously the whole point is to give us the ability to access this endpoint which as you see here is gbt 40 realtime preview 2024 101 I will notate that some accounts don't have access to this endpoint yet some accounts do so you run into errors that may be your issue also it's important that we identify another tag here which is open AI beta real time V1 I mean this is early days y'all this is like extremely early days because we're getting the ability to create a Siri which is kind of crazy therefore the next thing that URS here is that we are taking the messages coming in from the user EGU and we're converting it and of
[09:49] course we have a bunch of just air logic right I put these in here just so you can get help if you run into airs we get some type of message air. message like you get some type of idea what's incurring here when you try to take this code past what I'm about to show you that's kind of the server side I could dive deeper into this but I just wanted to show you how to get this up and running as fast as possible so we're going to jump over to app.js which is your front end typically when you stru your front ends you'll have most of your variables that you will associate and later on code on the top of the fold EG Bowlings use States use refs Etc now
[10:22] just to give you a quick recap is recording set is recording this kind of logic is in the context of like I clicked the record button set recording to true I click it off set recording to false and then we can you know obviously cause the interactions on the actual app itself to change therefore the second most important thing that you need to identify when accessing realtime API is your backend URL so in this context because I want to do it all local it's going to be Local Host 4000 as we referenced earlier here ws- client that ws- client comes from the websocket server here that we set here on this path then I give us some nice little
[10:54] console logs here saying we're connected to the assistant Etc this right here is the meat of respon API which is the ability to select modalities voice instructions Now instructions is probably the most important in the context these can get very very lengthy so this one's from open AI here such as your voice and personality should be warm and engaging with a lively and playful tone Etc you can make the instructions more specific maybe you're Gordon Ramsay you know you're the best chef in the world you know how to cook the perfect steak I I keep talking about steak in these videos I don't know why I am cooking a riby pretty soon here that
[11:24] should be good put a little olive oil let the pan get on 10 get it get it real hot you let it sear in what and butter steak um and what's great is that within the code I provide there's this bunch of air logic to help you out and you can copy and paste into Chad gbt if you run into issues from this we are able to get the conversation we need therefore when setting up our front end logic here it's going to be like a button right it's going to be like start recording ask question and obviously in order to do that we need to be able to get access to the user's mic which we do in here next to the process audio logic here we structure the file so that it can be
[11:55] interpreted correctly within the real-time API which typically in this context is a b 64 audio which we then input into here therefore coming all the way down here we're going to have our button so be able to record stop record and proceed now there is one thing that I couldn't get it to work for some reason which is its ability to handle audio outputs let me explain let me first run this so in order to run obviously we're going to do in our react front and npm start and then for the back end in order to start that we're going to make sure that we are in the correct directory which is going to be backen that is going to be found within Siri better so you should just see
[12:26] something like this and enter no dat service DJs it's going to say the port it's listening to and then typically actually we're going to have to start the backend first and then run the front end so watch this npm start and here we go so now we have a live Connection in our local server to open AI realtime API if I come over here and I zoom in we can actually talk to it now when you first launch this it's going to ask can I get access to your microphone etc etc obviously say yes now it's saying in valid Json format set to server because
[12:57] we haven't sent anything yet watch this this is pretty cool Hello what is the best way to cook a steak is it in the oven on the grill or pan seared go so right now we have it set to where we'll have our audio file here then audio is sent to assistant for processing and then we get our response right here and I'm realizing we need to add some CSS okay with a little CSS added it looks a little bit better we got a little box here's our message here the best way to cook a steak really depends on your personal preference and the tools you're available if you love a smokey flavor grilling is fantastic for a beautiful
[13:27] here's the idea it was able to interpret that audio and give us a text output just to prove it to you again do you understand me yes or no I'm trying to prove whether or not I am talking to real time API stop recording audio is sent audio is processing yes I understand you how can I assist you further if this wasn't connected it would be like hi how can I help you this code and everything I just showed you up to this point should get you in the right direction towards any application development you'd want in the context of real time API now I'll be honest with you there is certain limitations to the local environment that I found personally EG we're not hearing any
[13:58] audio let me know in the comments down below if anyone that looks at the codee files and maybe finds a way around this cuz when I was kind of coding around I just was having issues I didn't know if it was a chors issue a security issue a browser issue or I was just conversating with the documentation wrong regardless though I know if I were to access this and maybe set up the server and a cloud run and then put maybe the data within a fire store something along the lines of using more of a actual backend I think I could solve that situation but at least in a local environment I was running into that issue other than that though make sure to leave a like it's completely free you learn how to use
[14:28] real time API you can download that code use it yourself check it out you can now start talking to open AI on your mic and I'll see you in the next video this tutorial was absolutely mind-blowing I feel like I've just hit the jackpot the insights the clarity okay okay okay okay those are two random videos that's my face and I'll see you in the next video