GPT-Vision or Google-Vision API possible?
# 🤝helpc
clever-horse-5574
01/26/2024, 8:54 PMHey everyone 🙂
has anyone already implemented the GPT-Vision API or Google-Vision API to analyze images directly in the bot? If so, I would be happy to hear more about how you implemented it!
f
fresh-fireman-491
01/27/2024, 7:28 AMI can share the code with you in an hour
fresh-fireman-491
01/27/2024, 11:05 AMThese are my repositories on github https://github.com/DecayAI?tab=repositories
You have the option to use an S3 bucket from AWS or use Base64. The bucket can be a bit of a pain to set up but this video: from 4 min 35 seconds explains it really well and is not outdated. To get the best results I would use the bucket as you would have to resize the image if you usage the Base64 as the string from it will be too long to send in a payload to Botpress if the image is in a good quality.
fresh-fireman-491
01/27/2024, 11:06 AMAll of the code are in the repositories. I don't think anyone except me have tested this tutorial so you will be the first to do that so if you have any questions about it or if you become stuck please let me know and I will be more than happy to assist you.
c
clever-horse-5574
01/28/2024, 7:46 PMI really appreciate that 🙏 I'll sit down with it this week and see how the implementation goes, then I'll be happy to come back to you guys 🙂
f
fresh-fireman-491
01/28/2024, 8:02 PMSounds great and let me know if you stumble upon any issues
c
clever-horse-5574
01/30/2024, 12:42 PMThe video and the bot template are great @fresh-fireman-491 ! I tried to implement the imageURL workflow first. Unfortunately, the code didn't quite work for me for whatever reason 😴 But now that I've changed it a bit, it works perfectly for me too. Here is the code I am using now: const openai = axios.create({
baseURL: "https://api.openai.com/v1",
headers: {
"Content-Type": "application/json",
"Authorization": "Bearer OPENAI_API_KEY",
},
});
async function createChatCompletion() {
try {
const response = await openai.post('/chat/completions', {
model: "gpt-4-vision-preview",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: workflow.prompt },
{
role: "assistant",
content: [
{
type: "image",
"image_url": workflow.imageUrl,
},
],
},
],
max_tokens: 500,
});
return response.data.choices;
} catch (error) {
console.error("Error during the chat completion: ", error.response ? error.response.data : error.message);
return null;
}
}
async function main() {
const choices = await createChatCompletion();
if (choices) {
workflow.response = choices[0].message.content;
console.log("Response:", workflow.response);
}
}
await main();
clever-horse-5574
01/30/2024, 12:47 PM▶️ Do you have any ideas on how to best implement this with base64 or S3 Bucket so that images uploaded via Telegram or WhatsApp can be used as FileUpload?
f
fresh-fireman-491
01/30/2024, 2:15 PMI actually have a template for both
fresh-fireman-491
01/30/2024, 2:15 PMIts just waiting to get uploaded to the hub which is hopefully this week
fresh-fireman-491
01/30/2024, 2:15 PMWhat did you change?
I used the code yesterday that I gave you and it worked for me
c
clever-horse-5574
02/18/2024, 10:12 PMHey @fresh-fireman-491 🙌 I've just sat down with it again and your workflow is great!
I would now like to use it to send an uploaded image from WhatsApp to OpenAI. Unfortunately, it doesn't quite work. In the picture is the workflow and the code to capture the WhatsApp image. The other two codes are exactly the same as in your example. Am I doing something wrong when receiving or processing the image?
clever-horse-5574
02/18/2024, 10:12 PMf
fresh-fireman-491
02/19/2024, 5:09 AMHey there.
I would create a separate post about getting WhatsApp images as I unfortunately don't really know anything about it.