Assistant API with GPT-4 Turbo Vision: OpenAI's Complete Guide to Integration
Let's build with Zapier and AI (100+ videos)📅 2024-04-15
Transcript
[00:00] open AI released an update to gbt 4 Turbo more specifically now we have the ability to access the vision capability on this new endpoint gbt 4 Turbo 2024 049 now a major question I've been getting asked in the comments is okay this is super cool but how the heck do we leverage it in the assistance API so in today's video I'm going to show you how now to be clear this is a workaround so I'm going to give you a code way of approaching this logic but also I'm going to give you a no code way of approaching this logic let's jump in earlier this week I did an entire video
[00:30] on this tweet we got from openingi developers talking about this endpoint where specifically I ran a little scenario of like what the cost would be associated with this kind of app where you take a picture you get the calories so you can check out that video right there to get more context on that but after that video I got a ton of comments asking me how the heck do we leverage this in an assistant API first off let's see if it's even possible within the current assistant UI that we have access to to see an image using this endpoint so I'm going to go and name this image test and you're just going to have a simple instructions here we're going to say tell me what is in the
[01:03] image period we're going to go to the specific model that was released recently which is gb4 Turbo 2024 049 and this should be sufficient we're going to go ahead and go to the playground in this playground in theory based on what we've been told in the documentation this endpoint should have the ability to read images Let's test it though I'm going to go and attach an image the image I attached is going to be a fake invoice here and we're going to Simply ask this endpoint what is this inv voice what is this image I attached
[01:34] hit run but then we get that kind of response sorry I can't currently view images however I can help you understand analyze text or descriptions of the images you provide them this is no good though as a lot of people want to leverage this assistant API in the context of reading images so in this video Let's see the possible workaround also I want to point out if you want to see more in- depth videos on just how to leverage assistance without this new update check out this video right here I show you how to manage like over 50 files within assistant if you don't know what an assistant API is think of it as like an even lasered in version of these
[02:05] models but you can provide like you know different files functions code interpreter there's a bunch of like extra layering of the cake we can do with these kind of assistance apis to make it more specialized for my business I've done like three or four videos on this topic so either look through my Channel or type in Corbin assistance API therefore in order to leverage this kind of Technology let's go and create a new assistant let say create new assistant here and this assistant let's just say is going to be our invoice hand hander invoice Handler first way I'm going to show you is going to be a no code way we going to say invoice Handler we will
[02:36] provide you with invoice data I'm going to go to Upper model to gbt for Turbo the modern one we're going to say We'll provide you with invoice data based on this data please write an email summarizing it this is going to be really simple because I just want to show you how to do it obviously there's a ton of other stuff like I referenced earlier so if you want to see the other videos of adding files making this more laser in for a business adding content T adding all this other stuff go ahead and watch that we have our current assistant it's going to be invoice Handler we have our directions we have a model let's go
[03:07] and jump over to zapier in zapier let's go ahead and say create new zap now we have our new zap here let's go ahead and do a trigger of Google drive folder this is going to how we're going to provide the data that's going to be read and do an event of new file and folder we're going to do an account okay that should be fine do test data here perfect and let's go ahead and jump over to drive in our drive here we're going to goad and drag over that image I had earlier I like showing Drive in these kind of cases because of the fact that it's a free service so you can get access to it right away and you can test this
[03:38] yourself without paying anything we got our invoice paid PNG in here let's go ahead and first off basically find the data that's relevant in this invoice assistant API is our name here we're going to go ahead and test this trigger and we should see the relevant piece of data here which is invoice paid so basically every time we want to trigger this we would just drag in the file in that specific folder and then it would proceed everything that about to do today there we go we got our file here invoice paid PNG this is the workaround y'all we're going to do a Chad gbt block CH Chad gbt we're going to do an event
[04:08] of analyze image content with vision because here's the idea you have a lasered in an assistance API your assistance API is trained on your data is trained on everything you want it to be trained on you have it where it's very lasered in for the specific task you want it to do you can reference previous conversations with it etc etc but here's the one thing it can do read images but but we can still provide the information that would be necessary that we would feed the assistant API already EG if we were already going to do the
[04:38] assistance API in the context of reading an image what are we doing we're internalizing specific data from that image to feed it into the assistance API so a great workaround is let's just get the data that we would want anyways and then feed it into the assistance API with a midterm block this is actually a pretty common workflow in the context of building out software where you want to reformat data and pass it down to another open AI block or AI provider block in this context we're doing it spe specifically to feed it into the
[05:08] assistant API but a lot of times in software and development you could do this in other context where maybe you have a ton of data that you want to compress with a 3.5 block and then use a chat GPT 4 block of that compressed data and provide another Value Point that way so this is normal and this is standard we're going to apply to assistance API therefore let's go ahead and say continue we're going to choose our account continue we're going to do the message here we're going to just say provide all the relevant information from this invoice we'll do things such as build to build
[05:42] to comma descriptions comma total amount Etc so if you want to get very specific here I have a bunch of other stuff on this topic but the idea here is that I just want to show you let's grab the data and then play with it so we're going to expect basically the build to descriptions and see what else they see is relevant in this context we're going to continue here and test this step also the image is going to be file exists but not shown it's not going to be the title is that's a string of data just text this is the actual image itself and here's the relevant information so we
[06:12] got the bill to we got the from got the date got the description oh this went basically overboard so I said Etc so basically that took it as let's just provide all the relevant data found within that image to gut check this real quick let's find out build to Olivia Wilson build to Olivia Wilson final price is around 2,100 then we were looking at 2,100 perfect in four different Services revision social media templates local design graphic design all right that has all the relevant information that we want therefore from extracting
[06:44] this data from this image we can feed this into an assistant block so I'm going to go ahead and click this little plus button here and do another chat gbt do chat gbt here we do an events and we're going to do conversation with assistant and they continue here continue again and with the message since we've already proctored within our assistance dashboard what to do we actually don't have to worry too much here we can actually just provide the invoice data like this invoice data semicolon parenthesis and we should expect a response an email relevant to
[07:14] this information let's chose our assistant we got our insistant right here invoice Handler and everything else should be standard here everything else is kind of already preset in that dashboard that we saw earlier but continue here and I should be expecting an email or like a pseudo email or a draft email here we go so this is the data we fed to it I like using invoice um I like referencing the Val variable point because of the fact that we said that earlier here we said the invoice data here so we're going to reference that with what we provide here then we do parentheses in order to basically compartmentalize the data that we're providing here and then we're going to
[07:45] come down here and we should see an email and boom subject line invoice summary invoice 200015 from warier in here l Wilson we confirmed that earlier and we should see all the relevant information in our invoice 2100 and all the relevant information in regards to that image therefore the logic you just saw here can be translated in a codel likee manner all you need to understand in that context is we're going to be passing data around within an HTTP callable function or whatever you know maybe you want to use a Firebase database flow it might be a little bit
[08:16] more C heavy but the idea there basically is that you call that endpoint in the first one the first function right here and you you know say your max token set your temperature have your prompt provide the relevant data point of the image asking for an output and then the second part of that logic would be like the output from that first function or from that first callable point you would send it to the assistance API in the code and then you'd get your relevant information slata that you care about this is pretty common in the context of software development and just workflow in general where you kind of want to maybe you know
[08:47] restructure the data a little bit before you send it to a further workflow or further Chad gbt block or anthropic block or whatever you're using in that context what I will say is that probably in the future let's hope at least when referencing the assistant API they'll make it more native and integrated so you don't have to necessarily do that little middle block between if you feel like you learned something today's video make sure to leave a like it's completely free we'll leave another assistance video I did a couple weeks ago and showing you how to leverage it for your business at the end here I'm also going to leave a playlist showing you how to leverage Ai and
[09:19] automation this is the playlist I was referring to when it comes to Ai and Automation and how to leverage on your business that is an assistance video more specifically an assistance API video you can start learning how to leverage this kind of tech easier here and that's my face daily content subscribe bye-bye