A tool that allow you to run SQL-like query on local files
→
T
TokenizerMicrosoft Corporation
Microsoft.Tokenizerv1.3.3
Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.
