Zapier & GPT-4V: ChatGPT Reads Images in Automations - Complete Setup Guide
Let's build with Zapier and AI (100+ videos)📅 2023-11-20
Transcript
[00:00] welcome back y' today we're going to be learning how to use chat gpt's ability to see and analyze images in the context of Zap here as you see from this video here you're able to actually look at images analyze images and provide value outputs based off this new openai API we're going to learn how to do that today so just so you can understand we're going to be using this documentation in order to create this API call I'll add this in the description down below it's going to be pretty simple preview here of how to basically leverage this new model over to zap you're going to hit create new zap we're going to do an event of when I drag an image to a a Google drive folder
[00:31] just to show you how to do this your use case might be entirely different all you need to know is that you're going to learn how to basically send this request and get a you know response back so therefore you can really start leveraging it in any context you may have in your business for me though I'm going do new file in folder continue here I'm going to choose our courses account here at webcafe AI continue here again and we're going to go ahead and choose the specific folder so for me it's a YouTube zaps folder so I got the folder called vision test here I'm going to continue here and then I'm going to
[01:02] go ahead and come over to our Google Drive I'm going to drag in a photo real quick so the photo we're using today is uh three Australian shepherds we're going to see if gbt can understand that these are puppies I will say this is pretty low resolution but still let's see if they can understand that we got three little dogs here so I'm going to go ahead and drag this into vision test here so we can start leveraging it perfect so we got our image here let's go back over to zpp here we'll call it Vision gbt 4 and we're going to do test trigger here and we should see our relevant file here I'm going to be completely
[01:33] transparent y'all I haven't even tested this myself if this works I'm just going to go based off documentation and my expertise and see what we can come up with here so we got our file here let's go ahead and add our action which is going to be that open Ai call so open Ai and just so you have more reference here the only time You' use uh anything alternative to open AI in this block is when you want to use gbt 4 model in the context of chat gbt and like more of the conversational aspect for the custom request though you want to use the open AI block so we're going to do API request beta we're going to hit continue
[02:05] here and because we're requesting uh the back end of openi to do something for us we're going to go ahead and do an event of post because we're pushing data towards them and then on top of that we're going to go ahead and make sure we do the right URL here so for us to understand when it comes to gbt 4 and specifically in the context of conversation model we're going to be doing the completions URL here and then we're going to add a couple more parameters here so we're going to add the Json payload here which is going to be the body that has our relevant information here and I believe we need to add an additional request header for the sake of dotting my eyes and crossing
[02:37] my te's I'm going to add this just in case where the type of information we're receiving and the structuring of that information is in Json so we're going to do content type here application SL Json and then from here let's go ahead and add our body here so I went ahead and marked one up already this should work in theory although I may need to work on the syntax on this body here I'm going to bring this in here real quick and let's see if everything looks good to here all right so I'm going to try this y'all if it doesn't work then I'll need to restructure the syntax but first off
[03:07] let's just go ahead and walk through it a little so we're going to be using the gbt 4 Vision preview that's going to be our model as you see here the prompt we're going to use when we ask like what's going on with the image we're going to say what's in this image and then finally we're going to provide the underline URL we're using today so I'm going to try the file exist but not shown but to be honest with you it really depends on the context when it comes to Google Drive which one I'm supposed to use but I'm going to try this one first let's see if we getting any errors and kind of troubleshoot together all right so I am debugging still but as you'll see here the air that I'm getting right now is that the
[03:38] type of files that it can read right now are PNG jpeg gift and webp and the current image I'm using is a JPEG but gpg so I'm going to convert this into a PNG but I wanted to point this out as so you know basically the types of files that it can actually read in this context so now I have it set up so it is a PNG let's jump back over to gb4 here let's go ahead and retest this trigger and find the new record where it's a PNG file this is important for you to understand so you don't find yourself running into issues here uh because of the fact that you're just using a wrong file type so I'm going to type in PNG
[04:09] here make sure cut the right one perfect as you see here we got mime type to continuous selected record here going to keep the bug in here one other thing I realized during the debug phase is that we can't actually identify the amount of tokens that or the max tokens this isn't too pertinent so I went ahead and remove that as well and then I'll go ahead and explain this once we get to that point first let me just see if it works so after further debugging and looking through it a little bit more I went ahead and uploaded a another image here but it seems like it probably would have worked with the older image as well this is just a dog JPEG and as you see here the way we need to format our URLs is going to be basically finding the direct
[04:40] image source but there's another thing I got to point out here if we go over to my folder here you have to ensure that basically anyone with the link is an editor and gets access to it this is very important this is going to allow the API to even see the image to begin with coming back over to gbt or zapier we're going to go ahead and format our Google Drive Link like this we're going to do HT gtps Drive goole.com equals ID in order to find the ID we're going to Simply go to the uh you know Google Drive new file here and type in ID and it's going to be this one right here put
[05:12] that right here so that therefore any new image you add here it will automatically give us an interpretation of it I'm going to hit continue here and I will retest this step again so you can kind of see the output and therefore the output in this context is going to be a golden retriever so if I come down here as you see we are going to come all the way down here and the content is going to be response here the image show a happy looking Golden Retriever with its mouth mouth open and if we come back over to our file here I think that's pretty spot on but just to show you how this works as an automation what I'm going to do is I'm going to add a new file here and we can go ahead and see
[05:44] you know basically gut checking and make sure it works that only another image here as it is a part of the files that can accept this is a webp as you see here I also made it so the name is like really random so we can really gut check it and make sure it doesn't like read the name for interpretation as well so coming back over here I'm going to go ahead and test this trigger again say find new record I know this is W uh WP here so here we go continous selected record come over to here and I'm going to go ahead and say continue retest this step and let's see if it comes up okay
[06:14] here we go you're looking at an image of a cat with a displeased or grumpy so this does work okay so we've officially learned how to access open AI API when it comes to Vision really cool stuff now you know how to leverage it when it comes to autom on however contexts you may want to do in that realm if you feel you learn something up to this point make sure to leave a like it's completely free if you like this kind of content you can check out a playlist at the end here as we're diving into zap and all five 5,000 apps and seeing how we can automate every single one without further Ado I'll see you in the next
[06:44] video thanks for tuning in and yes surprise I'm an AI Avatar make sure to explore more here at Corbin AI where we demystify AI for your personal and business life until next time time