I don't really know how it could be implemented so feel free to drop your ideas below ! I feel like it could be interesting and really useful to be able to test the bot with different pre-defined scenarios Also to be able to run test (especially for autonomous agents) for comparing outputs from different LLMs (instead of doing it by hand, with multiple different tests etc...)

Enhance the iteration workflow card by adding the option to choose the LLM for all ANs in that specific instance of the workflow This way, we would also have the ability to change the LLM for some ANs, as someone has already requested, putting it in a workflow.

When we were creating a new bot to replace the old one I took some conversations from the old one and ran them through the new one automat via the Chat integration and then we compared the answers. You could define any scenario here. We were also thinking about letting an AI have the conversation with our bot, but we ended up with the conversations instead.

Good idea, I like it! In this way, I would need to create a bot to compare the behaviors of different LLMs However, if I only need to compare the behavior of an Autonomous Node (AN) with different LLMs, I would prefer the option to have a workflow with the same nodes and select the LLM in the workflow card options. With this feature, I can have different LLMs in different workflows within the same bot, too.

yes i think i made a thread about that indeed 😅