https://discord.gg/botpress logo
Join Discord
Powered by
Web Search is not working
# 🤝help
e
Hi, I've just published my website and i want to use botpress as a chatbot that could scrape all of the information provided in my website. To do that, i tried to use the web search feature in knowledge base but it doesn't work. I've ask few questions to my bot but it doesn't return anything. I've verified that my website's URL is accurate and accessible, but somehow my bot can't read the information from my website. Any ideas or solution toward this problem? Thankyou
j
hey man
can you gve me your website url
e
ikn.presn.cloud
this is the link feel free to check it and contact me if you have any questions
j
ok so
do this
ikn.presn.cloud/sitemap.xml
put that into a site map in your KB
e
it shows this pop up when i try to ask something to my bot
j
hmmmm
wier
let me get botdassedors
@User
b
Make sure you do not use "http://" or "https://" in the URL when adding it to Web Search.
e
i didnt use "http://" or "https://", it still doesnt work
and is botpress only can web scrap in english only?
or it can web scrap in other language?
j
ive got the same problem
i did the same as you but does not wokr
even with a correct url
e
right? hahaha
@User is there anything that you guys could help with this? i've already use "http://" or "https:// it still doesnt work. Also, can the bots do web scraping in languages other than English?
d
When scraping from your site, you tried using the url exactly like this? https://ikn.presn.cloud/
This is the error I'm receiving when I run try it myself.
e
can you run it using this url https://ikn.presn.cloud/informasi
i mean, why does this happen? any explanation?
Also, yes i already tried using the url exactly like that
@dazzling-army-23945
b
Your website has no content, until it's build by javascript:
Copy code
<body>
    <app-root></app-root>
  <script src="runtime.93acb3ee7f708311.js" type="module"></script><script src="polyfills.a7ab35367558375f.js" type="module"></script><script src="main.cddddc494b6a99d8.js" type="module"></script>
</body>
The Botpress indexing sees an empty page and gives the error: "no text could be extracted".
m
don't add to websearch, add to webpages and select sitemap
oops not @bitter-magazine-8114, I meant @early-waiter-80933
e
sorry for the late reply, thanks for the advice sir. But, webpages can't read data in real time right?
Hello guys @User, Can you guys please check out this link "ikn.go.id", i've been trying to use that link in web search feature but it also doesn't work. Does anyone know the reason? thankyou
b
Hi, remote host does not work
e
try this instead, https://www.ikn.go.id
m
Can you translate that error message in English?
f
m
It’s not the host end possibly?
f
did you find the issue?
e
not yet
f
im using a website with a bunch of dropdown content
and its picking up nothing
maybe this is a lead?
have u tried using vectors?
e
yes same here, what framework did use for your website?
f
how do i check? im making this for a client
e
yes, but it's also doesn't work
oh, what language did you use for your project
reactJS for example or angularJS
f
i think this seems to be angular
e
oh u facing the same issue? because im using angular too
f
yes
do u think its bc its how the website is built
e
yes i think so, many said to me that it's because of the javascript
but i don't really understand. And i think this too
f
Hi @early-waiter-80933 and @future-potato-92297 have you guys thought about using an external service to scrape your website and then upload that information into the KB?
f
that
is the plan now
e
okay, what external service will you use ?
f

https://www.youtube.com/watch?v=B89Cf4pLNds▾

i just did a simple search of what i can potentially use
maybe link the api to botpress and somehow get it working from there
e
oh, how would you link the api to the botpress?
f
hahaha im new to this my man i would probably use the exec code block. managed to get the gpt assistant that way
f
I would just scrape the website in to a text doc.
Organize that text doc so its not messy but instead it should be precise and organized.
l
is there any tutorial about implementing scrape functions yet?
@early-waiter-80933 / @future-potato-92297 did you solved it 🙂 ?
just came in here for the same question ^^ Puppeteer Stealth could be a solution
f
i just externally scraped it using beautifulsoup on python
put it into a txt file
l
ok thanks got it!
e
yea, by using server side rendering bro. It works haha
266 Views
PreviousNext
Powered by