https://discord.gg/botpress logo
Join Discord
Powered by
HUGE BILLING bug. Seems like a scum
# 🤝help
g
I will ad explanation in topick
Guys, I have a huge problem with billing
Specifically the Open ai payment. I am charged very large sums for dialogue. A small dialogue with the bot costs me 0.4 $
After posting my bot in telegram there were a few dialogs, and it costed me 4.4 dollars! If I have 10,000 usual dialogs, I need to pay 4k$ only to open ai
Also I notice that after changing model in ai task there is no difference in token pricing. I made some calculations
It was per tocken price test for: gpt 3.5 turbo version 1 and 2 And gpt 4 turbo versions 1 and 2
LOGS AVIDANCE THAT THERE IS SOME BUG In this case I asked small question that is not in my KB. So ai task performed (gpt 3.5 turbo and gpt 4 turbo have thesame price results) Gpt 3.5 test - Question was: "How are you". - Answer was: I'm doing great, thanks for asking! How can I help you today? Gpt 4 test - Question was: "How are you". - Answer was: I always enjoy chatting with you guys! I'm fine, thanks for asking 😊. Is there anything else I can help you with or do you have any other questions? LOOK in logs!!! I pay 95% of my money to kb, that didn't perform in this case
Also look how much tokens has been calculated to this little query
SECOND CASE In this case I asked question that is in my KB. And here is results
I decided to ask my lovely chat gpt to calculate how much tokens in given answer from kb. There was only 107 tokens in answer.
Only summary, personality and translator agents tacked 2,347 tokens that is 22 times bigger than tokens in answer
f
How big is your KB?
g
Little
.
f
In RAG, the user query is sent to a vectorial database, where the most similar piece of information is retrieved and sent to the LLM along with the query. The number of tokens sent to the LLM can be significant depending on how many chunks of text are retrieved from the database. A lot of redundant text in your KB will ramp up the costs. Try and go thru you KB and delete everything that is redundant. If you are lazy and smart you will give your documents to ChatGPT and prompt it to optimize the text for RAG and delete all redundant text.
g
If the problem is with database optimisation, why is the price many times less than in cases where the database is not used? And why is there no difference in pricing of different gpt versions?
f
There is a big difference between the pricing between GPT-3.5 and GPT-4. GPT4 is ≈20 times more expensive that GPT-3.5
You can see that the most expensive part is the KB agent
g
Please, can you review my messages again?
There is no difference in two versions
f
Even if you query the KB and it is not able to answer it will still have a cost
g
And here, where was no answer in database I was charged for 0,024$ for second database quary
In this case it was only 0,0032
f
Have you selected Hybrid in your KB agent?
g
Yes
f
It used GPT-3.5 first which is the first cost, then it used GPT-4 which is the second cost
GPT-4 is so much more expensive that GPT-3.5
g
Thanks for response
But why it charges 2,347 tokens for summary, personality and translator agents if text is only 107 tokens?
Thanks for this advice
f
I can't tell you exactly what is going on behind the scenes but my guess is that the all of the agents tokens costs works like this: Receives a message of x tokens, uses a prompt of y tokens and outputs z tokens. Total token cost is there for x + y + z and you can only see the total cost. It is kind of like a black box for us, since we can't really see what is going on except the input and output. This post is sent to the team and when they are back in office on Monday that can take a look and hopefully give you some more information about this than me.
g
Thank you very much for the reply, I look forward to hearing back from the team
Sorry for another twitch, but after your advice on optimising skew gpt the database started using 1k more tokens for analysis
3500 tokens instead 2450
f
Do you think that it was optimised? Did it reduce the amount of text in the KB?
g
Yes
But in any case my bot has a very small database, but the spend is very large. What happens when I have a bot with a big database?
My database is only 500kb from 100mb
f
It used more tokens with the exactly same query?
g
Yes
I tried different queries
I will give you my document and optimized one
f
I might be thinking about to different posts but didn't you say that your KB was only 0.5 MB earlier? Or am I thinking of another help post?
Yea here
g
Yes
It is only 0.5 mb that is 500kb
f
So it is not 500kb from being 100mb it is just 500kb?
g
yes my database is only 500kb
And it takes 2500 tokens to review it with gpt 3.5
And as i understand, if my KB will be 50mb each call to my database will take 250r tokens, that is minimum 2 dollars
@fresh-fireman-491
f
No. Here is a really simplified picture of how RAG works.
The query will remain the same amount of tokens. The amount of context given by the database will also be the same. The amount of tokens used will not increase a lot since the LLM will roughly process the same amount of tokens. It will have to search in a larger DB but it shouldn't ramp up the costs by 100x
g
Thanks for your professional explanation
f
You are very welcome, I am testing it right no to make sure I am not wrong!
g
Please, provide me with test results
f
With a KB of just 587 bytes.
With a KB of 59.336.422 bytes. 59.3MB
So about 4 times more in token usage
And the KB is 101084,1942 times larger
So as stated, there will be an increase in the token usage the larger the KB is but it will not go up by 100x just because the KB size does
With a KB of 81.054.142 bytes. 81MB
Basically the same amount of tokens used between 59MB and 81MB KB even though the new KB is around 1.37 times bigger.
g
Thanks
Can i ask you for a kb template?
In order to figure out why my kb is si weight
and in your experience, approximately how much money is made for an average conversation between a client and a bot
f
Hey I won't be able to share a template with you until much later today. I would recommend you to watch a YouTube video about it instead. It is 6 months old so somethings has changed but the principles are the same.

https://youtu.be/lWihOWJRMaM?si=Mp_RQhodGE9d74v5▾

The amount of money earned differentiates a lot. Some bots doesn't even directly earn money they are just there to make the customer happier while some might earn a lot. I am not the best at giving business advice so I would recommend you to ask those types of questions in #1174812513636987042
g
Thanks for your response. You really helped me. However, if it is possible I really would like to look at your bot with 50 mb knowledge base
Appreciate your time
f
I just used botpress.com as the kb
g
Thank you so much
f
You are very welcome
g
Guys, it’s an actual question
f
Hey Could you give a recap of what you are missing an answer to?
g
How can I decrease amount of tokens of kb query
r
Wow, I see. I'm not alone for this fast consumption of AI service problem 😉 You can check out what we discuss in this post https://discord.com/channels/1108396290624213082/1204990801265037362
f
Make you KB more concise
h
I also have the same problem with GPT4. I created a test bot with a single 37-page 1.22Mb pdf. The cost for each single response is $0.16. I can't understand how this cost can be brought down considering that I can't optimize the PDF in any way.
17 Views
PreviousNext
Powered by