You can stop it answering instantly by turning on Manual Answering and then using {{turn.KnowledgeAgent.answer}}. Would this help your use case?