Not sure, what's going on here. I can see on the conversations tab that the user entered her email address at this time: 15:38:08. but I can't see this email and time on the logs. So the customer entered her email and then nothing else happened. Just suddenly stopped. Not the first incident as well. This is on a webchat. But the same thing has also happened on our slack integrations. Need some help please. Our client is not very happy. And now we need to check every single conversation (a lot of them as well over the weekend) to check if this has happened to others. reported this problem: report_01J0K1DAR5DAP2XRP94TXDP7BX https://cdn.discordapp.com/attachments/1252204483954479136/1252204484738809946/1.jpg?ex=66715dcd&is=66700c4d&hm=b392fc1c14c0b78a73df9378bf46af38c40de0e0f3150f955735250af6f085d6& https://cdn.discordapp.com/attachments/1252204483954479136/1252204485049061386/Untitled-1.jpg?ex=66715dce&is=66700c4e&hm=1464ef5672e158f7201b535360f485182e4b00ad6ef8e1ea475c3c57c52d2797&

Hi Jojo, will you get any error if you give it another try?

Hi @big-market-58184 , I'm Jojo's partner. I have just used a transcript from one of the failed conversations to recreate a chat and it failed again albeit even earlier this time. The event log shows another internal server error.

using your own email 🤔 ?

Yes

hey there - thanks for flagging this. I took a look but, like you, I can't identify any reason this might be happening, and this isn't an issue we've seen with other users

Hi @bumpy-butcher-41910 thanks for the reply. I'm trying to export the bot to my own companies workspace now. What do you mean "simply a fresh bot file and re-deploying it"? Like start over from scratch?

instead of a new workspace, just importing it into a brand new bot in the same workspace

@bumpy-butcher-41910 I've tried to import the bot half a dozen times now into another workspace & it keeps throwing an error & freezing. Upon page refresh the flows do appear, but given that it causes an error I don't have much confidence that everything is imported correctly.

interesting. let me know~

We've been having the same issues and we import fresh versions of the bot every day or two. report_01J1B42DZNGVTNKTGYS31KDFY0

@bumpy-butcher-41910 & @big-market-58184 , ok I finally managed to get around to setting up an alternate version of our bot. On the first attempt of testing it out the internal server error happened (report_01J1SPWDZ64RDRY3TMVDETV4HW), again. It's not good to hear that someone else is having the same issue (see prev comment), though it does suggest it's not an issue specific to our account. Here's some background info on our situation so that someone can hopefully make sense of what is going wrong: ***Our bot was functioning perfectly for months, we had moved onto building a Slack Bot for our client (although this newer bot is actually suffering from the same issue now also...). We know it was working perfectly because we check through the conversation logs every day & cross check them against any leads that come in. We then needed to update the bot with a small prompt update on an AI task. Upon publishing this update we were quickly made aware of the bot being broken by reports from our clients website visitors (embarrassing 🤦🏼). We checked it and learned that all of the Generative Text cards were broken & needed to be fixed. It took us a few days to find & fix them all. This is when we began noticing the internal server errors. During our daily conversation checks we saw that chats were cutting off mid-way through. We cross checked these 'blips' against the server errors & sure enough they all matched up. The errors don't happen at the same point during the chat. Sometimes they happen early, sometimes towards the end. There seem to be no rhyme or reason to it. On your advice we set up the bot on a fresh template in the BP studio - but as prev mentioned the server error happened on the first attempt.* We did wonder if this was caused by Webchat, but the same happens in Slack so it must be caused by BP.

Thanks for the fast response @bumpy-butcher-41910! I'll post here since it seems to be the same problem Ryan is facing. It happened twice today. This is what happened in the user events, and nothing on the logs for a while after the internal server error. https://cdn.discordapp.com/attachments/1252204483954479136/1257704618096463945/image.png?ex=66856033&is=66840eb3&hm=836a2540e47ff8f932476c9a96b967b4a59a8d216d7c69376f1de3fcc6050648& https://cdn.discordapp.com/attachments/1252204483954479136/1257704618327015475/image.png?ex=66856034&is=66840eb4&hm=8e837365e1ef533c8c9f694dea295810d26183e26e57235334ae88a98382b255&

thanks for sharing gang 🫡

Hey all – Here's what we know so far about this error. The error is caused by an execution timeout. Basically, each message has to be processed within 60s by the bot, otherwise an error is thrown. (we will improve the error message so it says that). We are still investigating what can be the cause for these timeouts. We think that in the vast majority of the intermittent failures, the cause is OpenAI being slow or down, but we're looking into other possible causes. We're also working on having backup LLMs for when OpenAI is down or slow, which should ship within the new few weeks. If you could answer these questions (and anyone else having these errors) that would help us troubleshoot further: - Are you making API calls to external systems that may be slow? - Are you chaining multiple AI Tasks back-to-back in the same workflow, such that executing the whole chain may take more than 60 seconds? - Are you storing large payloads in the variables? - Do you have loops or manual promises (like using setTimeout) inside your workflows that can run for a while?

@fast-printer-67716 @limited-orange-3544 ^ 🫡

Thanks for the quick response you guys! I'm using an external API to send/save the information the user tells me, but the problem never happened during the API call. It only happened when I'm saving some variables (asking them to the user). We're not using AI directly, the user should only answer/choose some options from a dropdown button, there are some exceptions where they type some info like their name and I save it. No large payloads, the biggest ones are usually strings less than 30 char. No loops, only the regular bot timeout that I changed to 15 min.

@fast-printer-67716 thanks for all the answers, that really helps! would you mind creating a problem report, this will give us access to a copy of your bot for further troubleshooting? @square-energy-41150

# Sending a Problem Report Here's how to send the Botpress Team a problem report: 1. In the Botpress Cloud Studio, locate the "Help" icon in the bottom left corner of the screen. 2. Click "Report a problem" and follow the steps on the screen. 3. Copy the Report ID to your clipboard and provide it to a member of the Botpress Team on Discord.

Surely, thanks for the help. Here is the report ID: report_01J1W4XZ01AMREV4BE0953V4GY

Thanks @rich-battery-69172 . My wife @calm-cricket-17313 is going through your questions now & composing some answers.

Hi @rich-battery-69172. We indeed have many back to back chains of AI Tasks in the same workflow. Many of the errors occur during those AI tasks but then that could also be just because we have many of them because they also occur during a simple yes or no question which does not involve AI tasks. I've been looking at the logs and events and conversations just now and I found this conversation that actually had a 500 bad gateway error on the logs. In this one, the user just needed to select yes or no. He selected no and nothing else happened. That message from the user didn't even reflect on the logs. It showed as failed on the events with a 500 gateway error on the logs too. Although most of the time, there's no error on the logs at all. just nothing happens. https://cdn.discordapp.com/attachments/1252204483954479136/1258061983308320778/badgateway-error.jpg?ex=6686ad06&is=66855b86&hm=92b4b5e248c69f33f50636182a0312b463e35b58e1e8cd0b7c5105619d3db68f& https://cdn.discordapp.com/attachments/1252204483954479136/1258061983639539822/conversation-stopped.jpg?ex=6686ad06&is=66855b86&hm=ed8bf134a1c269fd66ed4a53551dec7eb8a2df2620ef15749df983b595e94367&

@rich-battery-69172 @bumpy-butcher-41910 @big-market-58184 It's been a week since I last checked in here. We're still getting the internal server errors causing the conversations to abruptly end. Have we any response to @calm-cricket-17313 's comments above?

Same here. However, if the user send another message after the error, the conversation resumes as if nothing happened. I instructed them to do so for now

We get that ocassionally, but more often the chat freezes and never recovers. Our bot is public facing so we can't really tell people to just keep typing messages if it freezes. If we did it'd look quite unprofessional!

I get it, my boss wasn't too happy with my "solution" either, but he accepted it for now. My bot records occurrences/failures in the public lighting service in 15 small towns in rural São Paulo. So we thought that it would be better for them to keep trying instead of just getting no answers.

We have more work to do now on this bot as well. makes me wonder if it's worth developing it not knowing the status of this issue. such a shame because everything was going quite well until we published it again for a very small update. then all the generative texts broke and then this internal server error. Has to be one of their updates that broke it.

@bumpy-butcher-41910 would our client be able to get this resolved if he were on the Team plan instead of PAYG? I see you get Live Chat as part of the package.

we're looking into this issue but - as you can imagine - there isn't an easy diagnosis + fix (if there was, it would have been resolved by now) - and since this only affects a small percentage of conversations it's difficult to keep track of

Sure thing! Thanks!

Thanks @bumpy-butcher-41910 . We monitor our bots constantly anyway. I'll have to ask my client to switch his ads back on so that we can give it a thorough testing. We'll let you kow what happens.

great - of course, the fix will remain in place even after 4pm EST tomorrow. we applied a hotfix to a small selection of accounts and want to determine whether to apply that hotfix elsewhere

Hi @bumpy-butcher-41910. 51 mins ago we had another server error that caused a users conversation to abruptly end. Do we need to re-deploy/publish our bot to see the benefit of the hotfix you rolled out? If not, then it would appear the fix hasn't worked for us 😣

Things are going smoothly over here, had just 9 users since the hotfix. None got the problem until now

This helps Ryan, we saw that you seem to be facing a different issue than @fast-printer-67716 & @calm-cricket-17313 In your case the problem is the state is too big to be persisted to our State API. There's a lack or clear error which we will fix to make this problem easier to troubleshoot in the future. It's unclear looking at your bot where and why this happens, we're going to add more detailed logs to help find the root cause. Will keep you posted on the updates here.

progress is progress is progress, thanks for the updates everyone!

@calm-cricket-17313 is my wife - our issue is the same issue. To be fair this is the only error we've had today. Jojo had to roll back an update earlier & I wonder if the last screen grabs I posted were related to that. We'll continue to minitor & stay hopeful your hotfix worked as it has for junkrs

happy monday bot builders - do we have any updates for me? 😎 🫡

Hi @bumpy-butcher-41910 Happy Monday indeed! Our client hasn't yet switched on his Ads so the bot hasn't had a proper test. Of the few users who found the website/bot organically today one of them had 2 internal server errors in the space of 3 minutes - however the chat wasn't interrupted & the goal of the interaction was met (they were sent an invoice). So it's looking positive. Once we have more data to look at we'll report back here. Thanks for checking in. Much appreciated.

@limited-orange-3544 we can apply the hotfix too, but just curious when the issue was first experienced by your client on slack?

Today is the first time this specific issue happened, but there have been internal server errors over the past few weeks that have caused the bot to freeze (similar to our website bot) - although these errors have been escapable. I was intent on asking you to apply the hotfix to the Slack bot eventually, once we'd managed to test the website one sufficiently. This error was much worse than those previously which prompted me to ask you to apply it sooner.

Sorry for the late response, I've been busy with some issues. I have not seen the bot freezing since them, things seem to be working fine now, I'll let you know if anything happens on this matter. Thanks for the help!

@rich-battery-69172 @bumpy-butcher-41910 sorry guys, now my client has switched back on his ads and the bot is taking more enquiries it's freezing still. It did appear that the issue was gone, but now it appears not. We've done a little work on the bot, unsure if that's managed to break your hotfix? Here's a new report ID - report_01J32P94E7PY298PZME6KBV82A We also have another issue. We've started to see duplicate responses from the bot sometimes too. An issue we saw on another post (https://discord.com/channels/1108396290624213082/1263401378231881759) Having looked through the Help topics this morning I can see there are many more users experiencing similar issues!

@rich-battery-69172 @bumpy-butcher-41910 Just another thing I want to add that I noticed. Yesterday, I exported one of the bots that I was using as a stage and imported it in our live bot (where you applied the hotfix). After I published the live bot, the errors came in one after another. so I immediately rolled it back to the old version (I exported the live bot before importing the stage so I just had to import the previous version back). I made some changes in it and published it again. Since publish, the errors still happen and the conversations still suddenly stop but not as crazy as after I imported and published that stage version from another bot (where the hotfix was not applied).

Good morning @bumpy-butcher-41910! My bot froze 2 times this weekend, even after the user tried to message it again there was no response. In events all there was is "internal server error" again. Should I create a problem report again?

@fast-printer-67716 please

Here it is: report_01J3DF007RDCP26BGT25MTW8FV Thanks!

@limited-orange-3544 @fast-printer-67716 thanks, the team is checking both your issues. we have a good idea of what the source might be

@rich-battery-69172 @bumpy-butcher-41910 unfortunately we have another of the same incident. much less frequent but happened again. made another report report_01J3MNXH93FDDD4CDW0KT6F70C..

@calm-cricket-17313 thanks for providing this. if you look at the end of the error in the logs, can you screenshot this too? thanks

@rich-battery-69172 here's the screenshot of the end of the error in the logs... https://cdn.discordapp.com/attachments/1252204483954479136/1266338730717745175/error4.jpg?ex=66a4c958&is=66a377d8&hm=7c34e6fc8cd079d39ad797a4788a83b72b4c5ffc7f08e1b9033adb21e42a014f&

@bumpy-butcher-41910 @rich-battery-69172 We are experiencing the same issues—flow errors and conversation state problems—with several of our bots. I will create a report for a few of them, but please let me know if you need any additional information. I reviewed the entire thread, and the reasons appear to be quite similar. Additionally, is there a limit to the size of a conversation state?

Thanks all for your patience and reporting on this. We're rolling out a few fixes this week that should address those.

Thanks for addressing the issue guys

Thank you Sly & your team. Your efforts are much appreciated. We'll let you know if we have a relapse!

Thanks Ryan, will have the team look into it

I am having an issue with integrating Slack. I guess it is related tot his bug.