Website is not getting indexed correctly
# 🤝helpp
plain-car-17885
05/02/2024, 2:14 PMHey, I am working on a Chatbot and I wanted to add a website to the KB, but somehow it is not taking the content from the website itself but from some popups you can reach from the header. After investigating the website a bit more, I found out that some genius put everything, including the header into the body. But now I am asking myself: it is possible only to index the relevant content on the page? Like selecting a single div on the page to index or something similar?
Thanks 🙂
b
bumpy-butcher-41910
05/02/2024, 2:15 PMcurrently you can't get that granular on the indexing
bumpy-butcher-41910
05/02/2024, 2:15 PMan alternative workaround would be to just save those pages to a document, like a pdf or plain text, and uploading that
bumpy-butcher-41910
05/02/2024, 2:15 PMsince indexing a site only takes a snapshot of your website at a moment in time, this should result in the same KB
p
plain-car-17885
05/02/2024, 2:17 PMalso thought about this solution, kind of unsatisfying but as you mention, results are the same.. Thanks 🙂
b
bumpy-butcher-41910
05/02/2024, 2:17 PMit's also worth noting that if we're having trouble indexing your content so will search engines
bumpy-butcher-41910
05/02/2024, 2:17 PMso if you run a business, it's probably in your best interest to fix this... 😛
p
plain-car-17885
05/02/2024, 2:18 PMit's not my business, I am building for a client😂 but yeah, probably I should mention this...
b
bumpy-butcher-41910
05/02/2024, 2:32 PMhahaha
bumpy-butcher-41910
05/02/2024, 2:32 PMyeah gotcha
p
plain-car-17885
05/02/2024, 2:32 PM@bumpy-butcher-41910 could I contact you via dm? I have a quick question about this and don't want to share the clients website here..
b
bumpy-butcher-41910
05/02/2024, 2:33 PMI can't offer support via DMs, sorry!
p
plain-car-17885
05/02/2024, 2:35 PMokay... can you explain why botpress is only indexing a part of the website? I found out that it is not even looking at the other parts of the site but only at a specific subpart, even if the link shows every part of the website
b
bumpy-butcher-41910
05/02/2024, 2:42 PMdo you know if the part of the website not being indexed by botpress is being indexed by a search engine, for example?
p
plain-car-17885
05/02/2024, 2:43 PMYes it is
b
bumpy-butcher-41910
05/02/2024, 2:44 PMhmm
bumpy-butcher-41910
05/02/2024, 2:44 PMI'm not sure I have a reliable answer to your question, then
p
plain-car-17885
05/02/2024, 2:48 PMokay...
plain-car-17885
05/02/2024, 2:57 PMI will open a new post, maybe someone knows fix
b
bumpy-butcher-41910
05/02/2024, 2:59 PMyup! you might find it helpful to include some screenshots of what's happening when you try to index the site too
p
plain-car-17885
05/02/2024, 3:22 PM@bumpy-butcher-41910 you know which crawler Botpress is using? What I found is that the website I tried to index has a robots.txt that looks like this:
https://www.contiss.de/robots.txt
Is it possible that this website is blocking the Botpress crawler? And only indexing the Popup, because the popup is under a special url: https://www.contiss.de/#preisrechner
145 Views