Automate Data Entry From Images and PDFs with Zapier and OpenAI Vision API
Let's build with Zapier and AI (100+ videos)📅 2024-06-17
Transcript
[00:00] in today's video we're going to be using artificial intelligence and automation softwares to automatically extract data specifically in this context I got requested by a taxi company on how to extract data from a image of a license so in this video we're going to learn how to extract data from any type of image we're going to go ahead and leverage zapier as the automation software we'll be using open AI as our artificial intelligence let's go ahead and jump in today's video Welcome Back y' on today's video we're going to be learning how to extract data in emails this can be applied to any other type of Contex specifically when extracting data
[00:31] from PDFs or image based files now you might be asking yourself Corbin how did you get this extremely Niche use case for a driver's license and that comes down to web Cafe software if you come to this site if you go to any of the pages so where for example go to packages and you scroll down here if you have a specific automation you want done simply request the automation click it fill out this Google form and as we keep going here I'm going to slowly push out videos that go over these different automations that different businesses are requesting so I'm going to go ahead and leave a link to this down below it's going to be AI Services web CAF sf.com check it out
[01:04] an example of me doing this past is a 43 minute video talking about how to automate your onboarding process this went over six different tasks that were traditionally done manually and completely automated them let's learn how to do today's video the way we're structuring today's video is we're going to be assuming that an individual had filled out a form and attached their driver's license through email so let's going to begin we're hit create zap obviously you can mix and match use the blocks how you want to use them in this context we'll just say the trigger is a new email I'm going go over to Gmail here Corbin I connected my Gmail it's not working you need to have a workspace
[01:34] account $7 USD alternatively if you use Outlook this will work as well so just connect your outlook I'm going to go ahead and choose an event here and you have a couple different options here when it comes to how to identify said email a really good video that shows you how to identify emails using AI but also alternatively filter method is this video right here so check that out if you want further information on to make sure that we're not getting triggered for every type of email this one we're just going to assume that within the email itself the title will say like application new email matching search continue choose the relevant Gmail
[02:04] continue and the search string will go ahead and add a little bit extra bazaz here we're going to say the subject line so we're going to use this little bit of dictation this is important as this identifies that it if this shows up in the subject line everything else gets triggered so what is say application as of now I have nothing sent to this specific email so I'm going to go ahead and send a dummy email for dummy data went ahead and S a dummy email for my corporate account here to my training account click it for now this is purely just for use case and data we're going to assume that you'll have obviously
[02:35] additional information within the body here but then we get the attached driver's license now the way you get access to this driver's license can be alternative methods you may use a different form submission when it comes to analyzing data you receive whether it's a form submission a different software whatever it may be the tech I'm about to show you right now can still be applied okay Alexander let's get your data when I continue here and there we go so we got the body that I sent earlier and we should have the attached image as well we continue now that we have selected record here we can actually start leveraging open AI right away so I'm going use Chad gbt while
[03:06] using Chad gbt we're going to do an event of analyze image content with vision now here's a big one y'all if you really want to get this lasered in where it knows exactly your context it knows what it's looking for and you don't have to Proctor as much I implore you to use assistance API assistance API allows us to preload an entire chatbot with context and data so it knows how to operate in these kind of automation form now I've done a ton of videos when it comes to assistance API Cad gbt and zappier so you could probably just type Corbin Brown assistance API on top of that I'll goad and make sure I leave a
[03:38] relevant assistance API video you can check out yourself that is trained on your business data so you can use it in whatever your operation may be that kind of sounded like a promo ad that's not a promo ad that's just like me telling you a good good option in this context but for this one what'll to say analyze content image content for vision right continue we're going to choose our account continue again and here we go so really simple y'all so this is actually I mean back in the day this would have been a big deal but now with AI and all its new features it's coming really simple to do this kind of stuff so we're going to add our image I'm going to Simply type in jpeg here but don't worry
[04:10] anytime you do this you're going to be able to find it anyways and because I do all my videos live I may need to do some formatting restructuring and get the correct image file but for now let's go see if this works I'm do attachment one file name let's just go and see if it works we're going to say extract the name on the driver's license I'm just testing if this is the correct way to access said image file continue test step and there you go this is all live and that is not the correct way to access the image so let's find out how second time is a charm so what you need to do with this action is we're going to
[04:40] actually use all attachments exist but not shown typically in zap year when it can't show the data or use those parentheses you notice this as well file IDs and Google Docs and there's ways to go about this and I've done videos on this in the past as of now though this is how we're going to extract the relevant information so now that we have confirmed and just to confirm to yall coming down here here's the output want to drag this over here now that we've confirmed we can actually look at the image let's go and structure a prompt that's going to work at scale context we received a driver's license extract the
[05:10] following information in this format name semicolon date of birth semicolon xir semicolon DL semicolon and the list goes on so whatever seems relevant here that you'd want to add addition height weight sex pair eyes everything from here let's go and see the structured output here when it comes to tokens if you're having issues raise it right now 250 seems a little bit low oh don't want to add that 250 seems a little low typically I like starting at 500 and we're going to hit continue retest this
[05:42] step also if you're having issues with outputs that's where assistance API comes in when dealing with making sure it's consistent my first output actually didn't work and what I realized is that sometimes with chat gbt you need to be very specific so it understands what you're looking for so I added because we already added the context that we received the driver's license went asking for the the name do o a DL add the term drivers to it so if you ever run into issues specify a little bit further for variables that are requested but from this output we got our relevant information let's check it out so
[06:12] obviously he messed up a little because it has sample here but we got Alexander Joseph Alexander Joseph we got a date of birth of 831 1977 831 1977 we got a driver's expiration of 83122 2014 831 2014 and finally the DL which is d112 34562 that's not correct let's see why because that's not correct let's go ahead and specify a little bit further here so we're going to say account actions drivers license number see if
[06:43] that's sufficient enough continue test up and there we go so we got a pretty solid output here driver's name d exp and the license number let's go and proceed to The Next Step here where we're going to format this data so we can actually extract it and push it to wherever to do so we're going to add another step here and this is going to be a formatter by zap here now what's created by zap here recently is that they made it so all these like formatting steps or zapier specific steps don't actually cost you money these are just L Fair free to use to be fair it's all just code so it makes sense whereas we're going to do text continue transform we're going to split
[07:15] it we're going to split text our input is going to be the output here which is going to be the image analysis from here we're going to do a separator which based off what we've seen is we're structuring output which is off new line Corbin how do you come to New Line click this link it's a little help article you can use it however you will for us we're going to do new line here separator new line and segment Index this is very important we're going to choose all as separate Fields this going to allow us to choose basically whatever data point we want so we test this step
[07:45] alternatively if we try to use this data elsewhere it's going to be like all one big block which we don't want so for this example real quick we'll just assume we're using zap your tables you could use Google Sheets you could use Microsoft Excel let say create a record oh Tech hire this is from that hire video I showed earlier instead of using otch high as I realized we actually need to provide the correct columns for the data we're providing we're just going to create a new table here this could be a Google sheet this could be Excel as well let's say blank table here we're going to say uh data extract create table now within this table we're going to go
[08:15] ahead and just add our Fields so we'll go start DL name do and we'll add one extra field here which would be exp not experience all right great here we go coming back to our tables here let's go and choose the correct table so not Tech higher data extract and here we go so what's perfect is because of the fact that we used the formatter block previously we can make sure there's no issue here so if we just did this for example we would have it all as one block so we couldn't separate the data which obviously is not really that
[08:46] efficient now that we have the format block though we can just put in the relevant information here so we can do the DL license number the name here we go do o and finally the EXP with all this information we hit continue here test this step and this should all be pushed to our table here boom that was good timing now one thing you might be saying to yourself is Corbin this is good but I don't like the driver license number driver's name driver's do driver's exp I don't want I don't want to actually use the name real quick let me show you real quick trick using 3.5
[09:17] specifically in the context we're going to be using gbt 3.5 or add a new block here if you feel like you learn something up this point make sure you like it's completely free maybe I got to do one of those Car Sales man was like $20,000 here $25,000 here because sometimes people say I speak too fast use 75 speed J gbt we're going to do an event of conversation we'll go ahead and choose gbt 3.5 turbo the cheapest model as of now within gbt's ecosystem there's cheaper but they're not as good we're going to use a user message here we're going to say based on this data remove all text before I have no clue this is
[09:50] going to work this might work we'll see as I said before this is all live so we'll go ahead and do this we going to do data one and then kind of proceed in this manner data two now mind you this step would have to basically here's the more effective way to do the step actually me move this up one block boom move step and we're just going to do data we're going to do that original block here so we don't have to add extra work here so we're going do data he mentioned analysis this was our original data block then we used the formatter
[10:21] but for now we're just going to go ahead and remove that information we don't like and I'm going to do semic here because that's how you specify data points with impr prompts and add a memory key consistent outputs at scale continue and test step so this should remove this this this and this test step and it didn't work so let's go ahead and use a higher model here typically if we could probably keep prompting this but to make my life easier we'll use a higher model using something like gbt 4 Turbo is always a good bet it's the most
[10:52] comprehensive model in the market right now we hit test step again there we go so we got our output here and we do notice that this is still in the which I don't like so coming over here I can just say generate just the data no text before after this is a very very famous line of mine I've used this for the last two years and it's been very reliable refresh the memory key add of one new slot retest step and we got a perfect output so using this output we can go ahead and grab that information put it through the text format again instead of
[11:22] image analysis we'll be using the conversation here the reply continue and instead of that we get this there we go y'all we have successfully together created a automation flow that allows us to extract data this specific context was a driver's license for the taxi company that made that request at web cafes software.com so if you want to make a request check out the website description down below if you feel like learned something make sure you leave a like is completely free and I'll see you in the next video this video right here
[11:53] is automating your entire hiring process 43 minutes long you'll learn a ton of stuff that maybe you can use in other part of your business that's a random video that's my face and I'll see you in the next video