https://discord.gg/botpress logo
#🤝help
GPT-Vision or Google-Vision API possible?
# 🤝help
c

clever-horse-5574

01/26/2024, 8:54 PM
Hey everyone 🙂 has anyone already implemented the GPT-Vision API or Google-Vision API to analyze images directly in the bot? If so, I would be happy to hear more about how you implemented it!
q

quick-musician-29561

01/27/2024, 6:05 AM
@clever-horse-5574 Recommended YouTube video related to that

https://www.youtube.com/watch?v=NO1LD4kF4Sk

f

fresh-fireman-491

01/27/2024, 7:28 AM
I can share the code with you in an hour
These are my repositories on github https://github.com/DecayAI?tab=repositories You have the option to use an S3 bucket from AWS or use Base64. The bucket can be a bit of a pain to set up but this video:

https://www.youtube.com/watch?v=t-lhgq7Nfpc&list=WL&index=6&t=275s

from 4 min 35 seconds explains it really well and is not outdated. To get the best results I would use the bucket as you would have to resize the image if you usage the Base64 as the string from it will be too long to send in a payload to Botpress if the image is in a good quality.
All of the code are in the repositories. I don't think anyone except me have tested this tutorial so you will be the first to do that so if you have any questions about it or if you become stuck please let me know and I will be more than happy to assist you.
c

clever-horse-5574

01/28/2024, 7:46 PM
I really appreciate that 🙏 I'll sit down with it this week and see how the implementation goes, then I'll be happy to come back to you guys 🙂
f

fresh-fireman-491

01/28/2024, 8:02 PM
Sounds great and let me know if you stumble upon any issues
c

clever-horse-5574

01/30/2024, 12:42 PM
The video and the bot template are great @fresh-fireman-491 ! I tried to implement the imageURL workflow first. Unfortunately, the code didn't quite work for me for whatever reason 😴 But now that I've changed it a bit, it works perfectly for me too. Here is the code I am using now: const openai = axios.create({ baseURL: "https://api.openai.com/v1", headers: { "Content-Type": "application/json", "Authorization": "Bearer OPENAI_API_KEY", }, }); async function createChatCompletion() { try { const response = await openai.post('/chat/completions', { model: "gpt-4-vision-preview", messages: [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: workflow.prompt }, { role: "assistant", content: [ { type: "image", "image_url": workflow.imageUrl, }, ], }, ], max_tokens: 500, }); return response.data.choices; } catch (error) { console.error("Error during the chat completion: ", error.response ? error.response.data : error.message); return null; } } async function main() { const choices = await createChatCompletion(); if (choices) { workflow.response = choices[0].message.content; console.log("Response:", workflow.response); } } await main();
▶️ Do you have any ideas on how to best implement this with base64 or S3 Bucket so that images uploaded via Telegram or WhatsApp can be used as FileUpload?
f

fresh-fireman-491

01/30/2024, 2:15 PM
I actually have a template for both
Its just waiting to get uploaded to the hub which is hopefully this week
What did you change? I used the code yesterday that I gave you and it worked for me
c

clever-horse-5574

02/18/2024, 10:12 PM
Hey @fresh-fireman-491 🙌 I've just sat down with it again and your workflow is great! I would now like to use it to send an uploaded image from WhatsApp to OpenAI. Unfortunately, it doesn't quite work. In the picture is the workflow and the code to capture the WhatsApp image. The other two codes are exactly the same as in your example. Am I doing something wrong when receiving or processing the image?
f

fresh-fireman-491

02/19/2024, 5:09 AM
Hey there. I would create a separate post about getting WhatsApp images as I unfortunately don't really know anything about it.