Tokenizer
About Tokenizer
Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.
Information
| Publisher | Microsoft Corporation |
|---|---|
| Version | 1.3.3 |
| Package ID | Microsoft.Tokenizer |
| Website | View Site |
| Source | View Source |