Understanding OpenAI's API Rate Limits
Let's become an AI prompt engineer📅 2023-10-18
Transcript
[00:00] welcome back to Corbin AI where I'm showing you daily how to start leveraging artificial intelligence in your personal and your business life today's video is a little bit more Niche we're going to be going over the topic of artificial intelligence integrated sasses and more specifically when it comes to open AI API and rate limiting now this is a important topic for you to understand as when you're building out your AI SAS this is going to come into play we're going to give you some of our past experiences when Consulting the documentation for rate limiting and some stuff to keep in mind as you go into a production level and actually launch
[00:30] your AI SAS in this context so let's go to jump in here and let's go ahead and first outline what even is rate limiting in the context of open AI API and what it is here is essentially how many requests per minute and how many tokens per minute you can use on their Ai and they do this for obvious reasons of exploitation but also excess usage that may incur when in the development process when you may cause an accidental Loop and I'm going to give you an example of that real quick but as you see here we'll see that there is token per minute and request per minute you
[01:02] are more likely to hit tokens per minute faster than you are request per minute because of the fact that as you'll see from our personal rate limiting and our personal tokens per minute in the past comparative to what it is now it's a little bit higher but you'll see essentially why you'd be more likely to hit tokens per minute how it works is that every single minute in the context of tokens per minute and every single rate or every single request per minute resets um after one minute like in the name but your specific allocated amount
[01:33] of tokens or request is specific to each underlying account so if we jumped over to our main account here you can see that our tokens per minute and request per minute is pretty high I will say in the de in the beginning of our development phase the tokens per minute that we were allocated was 160,000 for gbt 3.5 and around 4,000 um or sorry for gbt for is around 40 ,000 tokens per minute but within the last like one to two weeks I think
[02:04] because we've been getting increased usage of the underlying API as we're getting ready to launch ours went up drastically all the way up to a million tokens per minutes and 150,000 tokens per minute for gbt 4 let me go to explain a little bit more what that means and essentially what you can gr to grab from this so coming over to our whiteboard here when we were working with the 3.5 model and the form model in the building of our SAS we were allocated around 160,000 tokens per
[02:36] minute for 3.5 and 40,000 per minute for GPT 4 now that might sound like a lot but in reality it really isn't because of the fact that one of the things that we did um early on is that we accidentally created a loop and that Loop caused us to request enough 3.5 tokens for us to basically max out within like 10 seconds honestly probably 5 seconds so we hit 160,000 within 5 seconds so in the context of what we were doing every single request equated
[03:08] to around 3,000 to 4,000 tokens used I believe if I remember correctly so then I started doing the math and I started thinking okay well this was caused by an err but in theory if we were to quantify that it comes out to be go around 50 people so in theory if 50 people use our serves at the exact same time or within the exact same minute that could cause an issue with the platform and the platform could break because then what happens in the next minute we forget
[03:39] these users that tried to just request our API but it failed because you know we hit our token usage so After experiencing that I realized two main things the two main things I realized was first off okay we need to set up some type of system to handle excess amount of requests to our API so that's what they in this context that's called rate limiting and then the second thing that it came apparent to me was the fact that um it would be nice to
[04:10] have more tokens per minute and rate per minute so if you want to see this specific context for your open a dashboard all you got to do is come over to your just open a dashboard and come up to rate limits and you will see what's afforded to you I'll be honest we didn't start this much now we have access to a lot more more kind of making my original issue in long term this obviously you know we built an entire system to handle excess amount of API usage but in short term for us
[04:42] personally to hit a 1 million tokens per minute it would require the underlying use it require 400 users to click the same button at the same time so let me go ahead and explain that a little bit more in depth here so when we ran into the issue and we realized that okay so in theory if 50 people were to use our service within a minute we would run into a rate limit issue the way we kind of circumnavigated that is that we built out a global queue and the way the global queue works is it will take um when we realize that we're nearing a
[05:14] certain level of token usage we're going to initiate Global q and what the global Q does in this context is that it will store all the relevant data of those 50 users and when we have reset after the minute we're going to go ahead and push those users data back out to the underlying U each specific user ID and have it so that we have you know the underlying service provided therefore we're not losing the data This Global queue is on the uh top level surface therefore it's in the cloud and if an
[05:46] individual were to um submit their video they would be prompted with some type of screen telling them that they're in a global queue and then since it's a global queue we will know their position align whether it's three four or you know 20 and so on and then based off what we know about our processing we can kind of give them a gauge of well if you're 20th in line that's probably going to take around you know maybe 15 minutes and then you know it'll reset the queue and kind of go from there now I want to go ahead and give you some real inside uh knowledge here on how you
[06:18] would set up something like this and in the future we plan on doing more in depth tutorials on this topic but here's two major pointers that you need to understand when dealing with rate limits and when it comes to opening eyes API first thing you need to understand is you can't use the way that you would traditionally call it in the coding so what I mean by that is right now if you were to integrate open ai's API into a react or just visual code Studio and you would use the typical way of
[06:49] communicating with API which would be you know open AI chat and like you kind of just use their their more like it's basically their version of if they made an update on their back end it would automatically push towards your front end so you don't have to do a lot of wew work on the underlying prompt the problem is that if you go that route there is a certain data point you're not going to get access to that data point you're not going to get access to is these rate limit headers what these rate
[07:20] Li limit headers do is they give you context on specifically where your specific key and usage is about this might sound a little confusing but put it this way the way you access this is that you actually have to do an axio webp put call to this specific URL and then honestly as the models get newer you're going to have to update that URL which shouldn't be too big of an issue but the idea is that this is not well-known information because if you were to approach openi API in the sense
[07:52] of open ai. chat and you just import open aai you're not going to be able to get these specific rate limit headers because it actually requires you to do it as a post call to a specific URL why are these headers important these headers are important because they are what's going to allow you to gauge whether you're going over the amount of your tokens per minute or you're going under the amount of tokens per minute so what that means is the one of the more important headers here would be uh random spam call one of the most
[08:22] important headers here in this context would be rate limit remaining tokens so this is going to be your tokens per minute and how many are left or available in your um you know within the minute so let's say within a minute you've had three people use your service therefore we're down to like 130,000 tokens and we're nowhere near to zero so we don't have to really worry about doing anything on the front end we can still kind of proceed as normal because we know in a minute it's going to jump back up to 160,000 what that means
[08:52] though is that we need to keep track of this header so every single time you call upon open ai's API you using a web hook and using a specific API endpoint you're going to have to create logic where you're going to basically create your Global queue but you're going to have security measures where you are tracking every single time a message goes out through the open ai's API you're keeping track of that you know limit header right so how many tokens are left and the way the logic would
[09:24] work here is you are then going to take those the the repeat competing updating of that data so maybe your 3.5 tokens this is like going to be your metad data so you know if you have a thousand or you have 100 people all accessing or all using your service at once they're all going to be pushed towards the same open AI key so therefore you're going to have it's not going to be like oh these 10 users have a different uh tokens amount left or these users have a different tokens amount left no all 100 have the same amount of tokens left because they're all using your centralized key
[09:55] therefore we know that if we go through 5050 uh 30 users and we have within that minute let's say there is you know 30 seconds left but we know we've just hit now we only have 20,000 tokens left instead of taking the risk of letting the service continue it is better to have some type of cloud function set up where we say that if tokens uh 4 3.5 is found to be uh less than not greater
[10:26] than 20,000 then the logic we're going to proceed with is we're going to add all uh people's data into a global queue and then what we'll do is that we're going to create something called a pub sub and the pub sub is going to be able to check every you know every five minutes to check this underlying amount of tokens left so the pub sub here we're going to go ahead and set up we're going to say every 5 minutes we're going to go ahead and
[10:56] check our meta 3.5 token thata point and our meta four point or meta 4 token point the next question you might be asking yourself is how do we check that accurately if the only re if the only way we get access to this is through a message incurring that's the only way we're going to get access to this the only way we get access to the remaining tokens is if there is a message had incurred therefore we get a fresh data point therefore we get the new one minute where it's like okay we're back to 160,000 we're back to 40,000 tokens
[11:26] the way to do that is you set up that Pub sub I'm not too sure if that's called something different in different documentation I'm just referring to Firebase in this context and we set up so we will do this we're going to have a open AI prompt for 3.5 and we're going to have an open AI prompt for four but the purpose of these prompts is going to be you is going to be sending as little amount of data in that underlined message the idea here is that the message that we're sending is more of just a feeler message and what I mean by that is that in reality the expenditure found in this 3.5 message and the
[11:58] expenditure found found in this for message maybe he's only five tokens and maybe he's only 10 tokens and the reason we do that is when we enter that Global queue we're going to have logic within that Pub sub that essentially says that you know we're only spending five tokens so we'll be able to get that accurate number so when the five minutes hits and it realizes that the logic you would have inside that uh Pub sub would be um when the tokens so back to 20,000
[12:32] when the when the tokens found so 20,000 here when it is greater than 20,000 because what's the original way we entered in this entire logic is because we knew it was under but when it's greater than 20,000 because of that repeated check on the um limits of the t uh tokens per minute we've checked it every five minutes and we know that after a certain amount of time it's going to reset to 160 therefore when the tokens are higher we're going to go ahead and proceed with handling and pushing back the data that we had stored in the global queue back to its
[13:02] underlying users now on top of that uh there's going to be a meta bullying that's used that for the global queue you're going to have your data stored in this um array you're going to also use a booing which is called status the idea of this buling is is the status paused or is it not paused now if it is paused that is what incurs the data to be stor at the global Q then the uh logic of the pub sub to incur uh because it's basically kicking in it's like okay well we know we're paus so we got to keep checking for the tokens and making sure
[13:33] that we are over a certain amount in order to uh proceed with everyone else's process now that being said the reason this is important and I wanted to make today's video on this which is honestly probably the one of the most complex videos I've made on this channel so keep in mind when I do come out with that series building out AI SAS it's going to get very complex it's possible but it is very complex um I wanted to make this video because it's important to one
[14:04] understand this but two start getting some wiggle room with open AI API does that mean request a little bit more usage I would encourage to do so does that mean you know maybe do more processes using it I would encourage to do so because I'll be honest with you our original intention is that we were going to reach out to Open Eyes um support team and be like hey we're about to laun this SAS right now we're stuck at this amount you probably want a little bit more wiggle room within your backend but when we checked I just checked within you know the last two
[14:35] weeks I was like holy smokes when we started this entire process we were only afforded 160,000 tokens per minute but now we're afforded a million so that's well above the amount that we need at our current stage with our current processes and how our SAS Works a million tokens per minute for 16k would in our context would mean that a 400 users would have to hit the same button within the same minute for the global Q to incur what's important though is that if you don't have that much breath or that much room um that's why Global Q is
[15:08] important now global Q is going to be important for us regardless because why if we do hit 500 in a minute now we have that Global Q as insurance to Ure that the platform doesn't break and we can you know ensure that everyone's uh experience of the platform is good knowing this though Insurance maybe insurance is not the best word to use in this but having some type of insurance or some type of of security or something where you have to take into account for everything so we went with the logic that um are we going to look at tokens
[15:39] per minute or requests per minute for us tokens per minute was more relevant because in reality there's no way we would hit we there is no way we would over request per minute would never be higher than tokens per minute in our context that being said if you are using maybe it only cost one like 10 tokens then request per minute would be more pertinent in this context for us it's just tokens per minute as we're expending 3 to 4,000 per minute or 3 to 4,000 per process knowing that that
[16:10] concludes today's video um yes this was a very complex video if you are building out AI SAS and you're approaching open AI dashboard um and just the API in general I encourage you to check this video back out um when you get to this point and more specifically uh understanding rate limiting better I mean honestly this also ensures that you have some type of you know insurance that if you do blow up or if the amount of usage that you have access to is very low that you have a way to handle it and
[16:41] you don't you know basically can't scale because this is this is part of scaling this is a huge part of scaling scaling is having the ability to handle 500 Clicks in one minute in that context so if you enjoyed today's video make sure to leave a like uh for the value found today if you want more talks like this I will say this is a little more complex than I usually do on this channel um you can check out the playlist at the end here where I'm basically diving into more topics like this like Ai sasses and just digital content in general but without further Ado I'll see you in the next video thanks for tuning in and yes surprise I'm an AI Avatar make sure to
[17:13] explore more here at Corbin AI where we demystify AI for your personal and business life until next time