I wish to tokenize the text, before submitting to the model, does this model provide API to tokenize?

#42
by JLouisBiz - opened

I wish to tokenize the text, before submitting to the model, does this model provide API to tokenize?

Or should I use LLM to tokenize?

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment