Tokenizers.DotNet 1.4.1

• Tokenizers.DotNet
• Nuget Package list
• Requirements
• Supported functionalities
• How to use
  • (1) Install the packages
  • (2) Write the code
• How to build
  • (Recommended) Cross-platform build
  • Building with local machine system
• Release guide

Tokenizers.DotNet

.NET wrapper of HuggingFace Tokenizers library

👁 Build (Windows-x64)
👁 Build (Multi-platform)

Nuget Package list

Requirements

• .NET 6 / .NET Standard 2.0 or above
• (Build) Latest Rust
• Tokenizers.DotNet managed assembly is strong-name signed. This changes the assembly identity from earlier unsigned releases, so upgrading consumers may need to rebuild.

Supported functionalities

• Download tokenizer files from Hugginface Hub
• Load tokenizer file(.json) from local
• Encode string to tokens
• Decode tokens to string

How to use

(1) Install the packages

1. From the NuGet, install Tokenizers.DotNet package
2. And then, install Tokenizers.DotNet.runtime.<OS>-<ARCH> package too (e.a win-x64 or linux-arm64, check Nuget package list above).

(2) Write the code

Check following example code:

using Tokenizers.DotNet;

// Download skt/kogpt2-base-v2/tokenizer.json from the hub
var hubName = "skt/kogpt2-base-v2";
var filePath = "tokenizer.json";
var fileFullPath = await HuggingFace.GetFileFromHub(hubName, filePath, "deps");
Console.WriteLine($"Downloaded {fileFullPath}");

// Create a tokenizer instance
Tokenizer tokenizer;
try
{
 tokenizer = new Tokenizer(vocabPath: fileFullPath);
}
catch (TokenizerException e)
{
	Console.WriteLine(e.Message);
 return;
}
try
{
 var text = "음, 이제 식사도 해볼까요";
 Console.WriteLine($"Input text: {text}");
 var tokens = tokenizer.Encode(text);
 Console.WriteLine($"Encoded: {string.Join(", ", tokens)}");
 var decoded = tokenizer.Decode(tokens);
 Console.WriteLine($"Decoded: {decoded}");
}
catch (TokenizerException e)
{
 Console.WriteLine(e.Message);
 return;
}
Console.WriteLine($"Version of Tokenizers.DotNet.runtime.win: {tokenizer.GetVersion()}");
Console.WriteLine("--------------------------------------------------");

//// Download openai-community/gpt2 from the hub
hubName = "openai-community/gpt2";
filePath = "tokenizer.json";
fileFullPath = await HuggingFace.GetFileFromHub(hubName, filePath, "deps");

// Create a tokenizer instance
Tokenizer tokenizer2;
try
{
 tokenizer2 = new Tokenizer(vocabPath: fileFullPath);
}
catch (TokenizerException e)
{
 Console.WriteLine(e.Message);
 return;
}
try
{
 var text2 = "i was nervous before the exam, and i had a fever.";
 Console.WriteLine($"Input text: {text2}");
 var tokens2 = tokenizer2.Encode(text2);
 Console.WriteLine($"Encoded: {string.Join(", ", tokens2)}");
 var decoded2 = tokenizer2.Decode(tokens2);
 Console.WriteLine($"Decoded: {decoded2}");
}
catch (TokenizerException e)
{
 Console.WriteLine(e.Message);
 return;
}

Console.WriteLine($"Version of Tokenizers.DotNet.runtime.win: {tokenizer2.GetVersion()}");
Console.ReadKey();

How to build

(Recommended) Cross-platform build

You can use Docker to compile this library for Windows x64/arm64 and Linux x64/arm64

Run update_version.ps1 before running Docker to update the package version.

Windows:

PS > docker build -f Dockerfile -t ghcr.io/sappho192/tokenizers.dotnet:latest .
PS > docker run -v .\nuget:/out --rm ghcr.io/sappho192/tokenizers.dotnet:latest

Linux/MacOS:

$ docker build -f Dockerfile -t ghcr.io/sappho192/tokenizers.dotnet:latest .
$ docker run -v ./nuget:/out --rm ghcr.io/sappho192/tokenizers.dotnet:latest

Built packages will be in the nuget folder.

Building with local machine system

(Note that this has been confirmed only in Windows machine)

1. Prepare following stuff:
   1. Rust build system (cargo)
   2. .NET build system (dotnet 6.0 or above)
   3. PowerShell (Recommend 7.4.2 or above)
2. Bump the version number in NATIVE_LIB_VERSION.txt
3. Run build_all_clean.ps1
   1. To build Tokenizers.DotNet.runtime.<OS> only, run build_rust.ps1
   2. To build Tokenizers.DotNet only, run build_dotnet.ps1

Each build artifacts will be in nuget directory.

Release guide

0. Change the version number in NATIVE_LIB_VERSION.txt
1. Run update_version.ps1 to update the version number
2. Commit with a new version number and add a tag. New tag should be end with -RC0, -RC1, ... until final check. BE SURE to check following files are committed:

• NATIVE_LIB_VERSION.txt
• Cargo.toml
• Tokenizers.DotNet.nuspec
• Tokenizers.DotNet.runtime.win-x64.nuspec
• Tokenizers.DotNet.runtime.win-arm64.nuspec
• Tokenizers.DotNet.runtime.linux-x64.nuspec
• Tokenizers.DotNet.runtime.linux-arm64.nuspec

3. Update README.md and name a real version tag (not ending with -RC0, ...)