1 winget apps indexed. Showing results for tags:Microsoft.ML.Tokenizers.
tags:Microsoft.ML.Tokenizers
Microsoft.Tokenizer
Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.