Hi, I'm dealing with an issue of latency in responses from the OpenAI API. I've observed wait times of up to 30 seconds, even for simple prompts. I'm happy to share workflow screenshots and any other detailed information, but need to do so privately if possible, due to the fact that this is for a corporate environment. Thank you!

you can always take a gander at this:

Thanks for the quick reply - this is an issue lasting several weeks

hm - when you say OpenAI API, how are you deployed on Botpress?

We're connecting to an OpenAI Assistant. I'm new to the team, but I believe they set it up using this template: https://botpress.com/templates/deploy-openai-assistants

ah gotcha

That makes sense, I figured there'd be a certain amount of latency expected. I guess I'm just trying to determine if the amount we're seeing is normal or there's something wrong. Is waiting more than 10 seconds per reply expected behavior, even for an always-alive bot?

always alive keeps the instance alive for first requests, but won't take effect on subsequent ones throughout a conversation

gpt-4-turbo

10 seconds for a 4t request doesn't sound unexpected from my end - the gpt4 family works much slower than 3.5 and 3.5turbo

Gotcha, I figured this might be the answer. Thanks so much!

\o/

Actually sorry I wanna add on to the question real quick. We still do see outliers sometimes of 25-30 seconds for a response. Not incredibly uncommon. Is that also expected for the gpt4 family?