"NPM packages, short for Node Package Manager packages, are essential tools for developers working with Node.js. These packages are repositories of code that provide additional functionality or utilities to a project without the need to write everything from scratch. NPM serves as a platform where developers can publish, share, and manage open-source projects. By using NPM packages, developers can save time, enhance reusability, benefit from community support, manage dependencies efficiently, and access a wide selection of libraries and tools for various functionalities like server-side rendering, data validation, and more. In essence, NPM packages can be considered external libraries. They are external in the sense that they are not part of the core codebase of a project but are instead external resources that developers can integrate into their applications to extend their functionality. These packages are maintained by the community and can be easily included in projects to streamline development, improve code quality, and leverage existing solutions without reinventing the wheel." https://cdn.discordapp.com/attachments/1223911670041149590/1223911972752326686/Screenshot_from_2024-03-31_11-22-15.png?ex=661b93d9&is=66091ed9&hm=fae17ecead9944c781217dbaf459559374684ce7733359f7b2485c61275e6581&

In this solution, we are using serverless functions and then directing the Axios call from the Botpress chatbot to that function. I'm using Netlify serverless functions, which are free; there are other options available for more professional use. The Netlify free version has a 10-second timeout limit, so it doesn't work with large code that takes a lot of time to execute, but it's super useful for testing and building stuff. And we don't want to use large code that takes minutes to execute in our chatbots anyway. Code to use Puppeteer and scrape webpages, inside a serverless function

js
const chromium = require("@sparticuz/chromium");
const puppeteer = require('puppeteer-core');

exports.handler = async function(event, context) {
  const data = JSON.parse(event.body);
  const url = data.url;

  if (!url) {
    return {
      statusCode: 400,
      body: JSON.stringify({ error: 'No URL provided in the request body' })
    };
  }

  const browser = await puppeteer.launch({
    args: chromium.args,
    executablePath: await chromium.executablePath(),
    headless: chromium.headless,
    ignoreHTTPSErrors: true,
  });

  const page = await browser.newPage();

  await page.goto(url, { waitUntil: 'domcontentloaded' });

  const headingsAndTexts = await page.evaluate(() => {
    const headings = Array.from(document.querySelectorAll('h1, h2, h3, h4, h5, h6'));
    const paragraphs = Array.from(document.querySelectorAll('p'));
    const headingsTexts = headings.map(h => ({ tag: h.tagName, text: h.innerText }));
    const paragraphsTexts = paragraphs.map(p => ({ tag: p.tagName, text: p.innerText }));
    return [...headingsTexts, ...paragraphsTexts];
  });

  await browser.close();

  return {
    statusCode: 200,
    body: JSON.stringify({
      status: 'Ok',
      content: headingsAndTexts
    })
  };
}

Botpress Execute Code card with Axios call (I set it up to try again after 10 seconds if something goes wrong, but after testing, it normally took 3-5 seconds to scrape the provided webpage):

js
const fetchUrl = workflow.webpage

async function fetchPageContentWithRetry(url, retryDelay = 10000, maxRetries = 1) {
  let attempts = 0;

  const attemptFetchingData = async () => {
    try {
      const data = { url };
      const response = await axios.post('https://botpress-puppeteer.netlify.app/.netlify/functions/meta', data);
      console.log(response.data);
      workflow.scrapeData = JSON.stringify(response.data);
      return true;
    } catch (error) {
      console.error(`Attempt ${attempts + 1}: Error fetching page content:`, error);
      
      if (attempts < maxRetries) {
        console.log(`Waiting for ${retryDelay / 1000} seconds before retrying...`);
        await new Promise(resolve => setTimeout(resolve, retryDelay));
        attempts++;
        return attemptFetchingData();
      } else {
        console.log(`Failed to fetch page content after ${maxRetries} attempts.`);
        return false;
      }
    }
  };

  return attemptFetchingData();
}

await fetchPageContentWithRetry(fetchUrl);

After asking the user to provide a webpage link and storing it in 'workflow.webpage', send that URL in an Axios call directed to your serverless function, here called 'https://botpress-puppeteer.netlify.app/.netlify/functions/meta'. Wait for the response and store it in 'workflow.scrapeData', then use that as an AI Task input and ask the AI to summarize the webpage content. https://cdn.discordapp.com/attachments/1223911670041149590/1223916142050152488/image.png?ex=661b97bb&is=660922bb&hm=514b455bc5e9b68e7024905f3485757c34223760693d39eef7158686e82fde43&

⬆️ These projects requires that you know how to use serverless functions (Netlify, AWS Lambda, Appwrite etc.). Another option is to wait for these community builds, which everyone can connect to their own Botpress chatbots, using only the provided chatbot code in Execute code cards. And this Netlify that I'm using is free and allows 125k function calls per site per month. The third option is to wait for even better and more user-friendly solutions.

bot file https://cdn.discordapp.com/attachments/1223911670041149590/1223931042918830192/puppeteer-summarize_-_2024_Mar_31.bpz?ex=661ba59c&is=6609309c&hm=c16c540aa847156ec02277845b6829eb230a28da5112842be3f84ec760fa0288&

This is so advanced, i fall to the side trying to process all of this information

Both these examples use an NPM package called Puppeteer, and this is one option for using external libraries that aren't pre-installed in Botpress Cloud. Those who don't want to build their own serverless functions can use these that I and soon others are building. Download the chatbot files and all required code is there already, pointed to the correct serverless function, using the correct libraries. If you want to build your own serverless functions, all the code to get started is also provided here in these tutorials.

After this, I'm moving away from Puppeteer and starting to build other examples using different NPM packages and external libraries with Botpress. Some good reasons to use Node libraries like Puppeteer with Botpress: 1. We don't need any external libraries to build really good chatbots for many real use cases that companies and clients need. But some use cases might need more functionality, and then we can use NPM packages and leverage the work of 17 million developers to build even more advanced Botpress chatbots. 2. You have total control of the info you need to provide the users from the website. You can decide what information you need to get from websites; in the first example, I used headings (titles) and paragraphs (text) and then the chatbot summarizes those to the user. In the second example using YouTube search, I took the video title, channel name, image URL, and link to the video, then displayed those to the user. Also, you have control when the info is scraped from the webpage, for example, every time when someone uses the chatbot (so the information is not hours or days old). 3. You can let users add webpages to the Knowledge Base. You can ask users to provide a webpage, then scrape the webpage data and store the result in the KB table. Then ask the user if they want to add more webpages (maybe news, research, etc.) or if they want to start asking questions from the new info (not only summarize it but ask all questions related to that). 4. You can do many of the same things with APIs also, but many times they cost money. Web scraping and the usage of other external Node libraries are free with serverless functions (125k function calls and 100 hours of execute time a month). If your users exceeds those limits, then you'll have enough money to pay for the serverless functions PRO version (2M function calls, 1000 hours of execute time a month, $19 on Netlify).

⬆️ Chatbot file, chatbot code, and serverless function code can be found in the #share-your-bots link above.

https://discord.com/channels/1108396290624213082/1225025706376036393 A simple chatbot (bot file + code in that link) that creates a random UUID and stores it to a 'workflow.randomUuid' variable. It generates unique identifiers for use in your chatbots and use cases. All projects follow the same scenario: We have a task in our chatbot that requires the use of an external library. The chatbot connects to that library using Axios, gets back the response, and then we display the result to the chatbot user or store it to a variable. Netlify (and other similar companies but this is my favorite) allows us to build and deploy serverless functions meaning we can really easily create API endpoints where we can hit that serverless function and perform the tasks we need. While this might not seem like a lot, what we can do with these is to create a serverless function which we can hit as an API endpoint inside of our Botpress chatbots, Make.com scenarios or really from anywhere.

Next projects we are going to try and build using external libraries in Botpress: **UUID: **Generates unique identifiers for use in your applications, ensuring each ID is distinct ✅ **date-fns: **Provides a comprehensive set of date utilities, allowing for easy manipulation, parsing, and formatting of dates. jsonwebtoken: Facilitates the creation, decoding, and verification of JSON Web Tokens (JWTs) for secure data transmission 🛠️ socket.io: Enables real-time, bidirectional communication between web clients and servers. We can see that bot builders already have needs and real use cases for UUID and jsonwebtoken. With date-fns, we can build our own customized calendars (modern, functional, good-looking user interface) for example with React. Date-fns excels for its simplicity, efficiency, and modern JavaScript compatibility, ideal for developers seeking a light but powerful date library. And I know that some bot builders here have used WebSocket for building a real-time connection between Botpress and Slack. WebSocket is a low-level protocol that enables two-way communication over the web, and socket.io is a library built on top of WebSocket (and other technologies) to provide an easier and more robust solution for building real-time web applications. So maybe socket.io helps us in some pain points with those projects.

I write down everything that I have learned and include it in those chatbot links. Sometimes, I collect the important parts to this thread also to keep everything related to this in one tutorial thread💡 UUID is done, so I guess next is JSON Web Token.

Three chatbot ideas using JSON Web Token 1. Chatbot asks for login (username, password). After the user provides those, it creates a JSON Web Token and stores that in a user variable. JWT expires after 30 minutes. If the user ends the chat and comes back before 30 minutes, they don't need to provide login credentials again but have access to the chatbot using that JSON Web Token. If they come back after 30 minutes, JWT has expired, and users need to log in again with username and password, and then a new JSON Web Token is created for the next 30 minutes. You can change it to expire after 5 minutes, 1 hour etc. 2. Chatbot with two Knowledge Bases, which both require a login (username, password). When the user goes to one of those workflows using that Knowledge Base, the chatbot asks to provide the login credentials and then it creates a JSON Web Token. When the user goes to another workflow which has the second Knowledge Base, it first checks if the JSON Web Token is valid and then doesn't require logging in, but if it's not valid or has expired, it asks to log in again. It doesn't matter which KB they try to access first, without JWT they first need to log in, and with a valid JWT they can continue asking questions without login. 3. Similar to the second chatbot, but instead of accessing two Knowledge Bases with login or JWT, it tries to make an Axios call to two different servers (or whichever applications or platforms your chatbot is connected to). The first chatbot checks if there is a valid JWT, then they can continue, and if there isn't, then the users need to first log in (username, password), then a new JWT is created and they can come back or connect to the other servers/applications without login until JWT has expired. You can build advanced chatbot use cases for both authentication (by verifying user identity with login credentials) and authorization (by determining what actions an authenticated user is allowed to perform with JSON Web Tokens).

after login, token is valid https://cdn.discordapp.com/attachments/1223911670041149590/1225503913516728350/Screenshot_from_2024-04-04_20-49-41.png?ex=66215e75&is=660ee975&hm=186245411c5d3dcffeed5eabf942ecd1f783b2c1ad55a935a213b873ddc14f80&

after it expires, user can't continue without login again. https://cdn.discordapp.com/attachments/1223911670041149590/1225504075714400267/Screenshot_from_2024-04-04_20-51-42.png?ex=66215e9c&is=660ee99c&hm=5e1c804bef99d53035db676d6dc44932939a1327deadbf68cda960eb725d780f&

Mr. Postman was also happy earlier 🛠️ https://cdn.discordapp.com/attachments/1223911670041149590/1225505071727640657/image.png?ex=66215f89&is=660eea89&hm=c37bd24fd73e47de7936d789bf4746d2ac2c097816d89d19e15483e8101b90c3&

https://discord.com/channels/1108396290624213082/1225823810062651402/1225823810062651402 Bot file, code and instructions how to use it.

Are we supposed to add something outside of the main function file if we also add stuff to the function file?

We're quite perplexed as to why we can't seem to get anything new added. We certainly don't expect you to fix our code for us, but could we ask you to give us a clue as to why we can't seem to add anything new? Did you build your function in a non-standard way?

It doesn't look so different from the examples in the Puppeteer docs.