Website Crawling Fails
# 🤝help
g
Hej. I have difficulties in adding a website to my knowledge base. after entering the URL https://digitaldruck-esser.de and clicking "Discover Pages", there quickly appear two messages "Browsing website" and "Finding robots.txt" - but then the dialog reverts to its initial state, the "Add pages" button remaining disabled. I already checked the robots.txt - it contains no entries prohibiting website crawling. what goes wrong here? and what can I do...?