Instead of
to limit to 5000 characters sent (5000 characters is between 714 words and 1250 words according to google) we should probably use
GPT3Tokenizer to really limit the tokens sent to 4000 or a configurable value.
I found out about it while reading this article
https://supabase.com/blog/openai-embeddings-postgres-vector
Also
https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them says
Token Limits: Depending on the model used (GPT-3), requests can use up to 4097 tokens shared between prompt and completion. If your prompt is 4000 tokens, your completion can be 97 tokens at most.
Instead of
alchemy-worker/index.js
Line 55 in 2d69713
to limit to 5000 characters sent (5000 characters is between 714 words and 1250 words according to google) we should probably use GPT3Tokenizer to really limit the tokens sent to 4000 or a configurable value.
I found out about it while reading this article https://supabase.com/blog/openai-embeddings-postgres-vector
Also https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them says
Token Limits: Depending on the model used (GPT-3), requests can use up to 4097 tokens shared between prompt and completion. If your prompt is 4000 tokens, your completion can be 97 tokens at most.