Testing Framework

This page documents the LLama.Unittest project: its test model acquisition strategy, shared constants, CI integration, and the structure of each major test class. For details on how the CI workflow triggers and filters tests across platforms, see CI/CD Pipeline.

---

Project Overview

LLama.Unittest is an xUnit-based test project targeting net8.0. It exercises the full LLamaSharp stack and references the core library, the Semantic Kernel integration, and the Kernel Memory integration LLama.Unittest/LLama.Unittest.csproj4-128

NuGet packages used:

Package	Version	Purpose
xunit	2.9.3	Core test framework LLama.Unittest/LLama.Unittest.csproj18
xunit.runner.visualstudio	3.1.5	Visual Studio / CLI test runner LLama.Unittest/LLama.Unittest.csproj19
Xunit.SkippableFact	1.5.61	[SkippableFact] for runtime skip LLama.Unittest/LLama.Unittest.csproj27
Microsoft.NET.Test.Sdk	18.3.0	SDK test host LLama.Unittest/LLama.Unittest.csproj16
Moq	4.20.72	Mocking LLama.Unittest/LLama.Unittest.csproj17
coverlet.collector	10.0.0	Code coverage LLama.Unittest/LLama.Unittest.csproj23

The project also imports LLamaSharp.Runtime.targets to pick up native backend binaries at test time LLama.Unittest/LLama.Unittest.csproj2

Sources: LLama.Unittest/LLama.Unittest.csproj1-28

---

Model File Acquisition

Tests require real GGUF model files. These are downloaded automatically during the MSBuild BeforeBuild phase via a custom MSBuild target system embedded in the project file LLama.Unittest/LLama.Unittest.csproj121

Download Item Declaration

Each model is declared as a DownloadFileItem element with three metadata fields LLama.Unittest/LLama.Unittest.csproj30-34:

Metadata field	Purpose
SourceUrl	Hugging Face URL to fetch from
DestinationFolder	Local subdirectory (always Models)
LocalFileName	Final filename after download

Declared models (LLama.Unittest/LLama.Unittest.csproj36-71):

Item name	File	Source model
Llama-3.2-1B-Instruct-Q4_0	Llama-3.2-1B-Instruct-Q4_0.gguf	bartowski/Llama-3.2-1B-Instruct-GGUF
smollm-360m-instruct-add-basics-q8_0	smollm-360m-instruct-add-basics-q8_0.gguf	HuggingFaceTB/smollm-360M-instruct-v0.2-Q8_0-GGUF
jina-reranker-v1-tiny-en-FP16	jina-reranker-v1-tiny-en-FP16.gguf	gpustack/jina-reranker-v1-tiny-en-GGUF
gemma-3-4b-it-Q4_K_M	gemma-3-4b-it-Q4_K_M.gguf	ggml-org/gemma-3-4b-it-GGUF
gemma-mmproj-model-f16	gemma-mmproj-model-f16.gguf	ggml-org/gemma-3-4b-it-GGUF (mmproj)
all-MiniLM-L12-v2	all-MiniLM-L12-v2.Q8_0.gguf	leliuga/all-MiniLM-L12-v2-GGUF

MSBuild Target Flow

Diagram: Model Download MSBuild Target Chain

Sources: LLama.Unittest/LLama.Unittest.csproj74-123

The DownloadAllFiles target runs BeforeTargets="DispatchToInnerBuilds;BeforeBuild" LLama.Unittest/LLama.Unittest.csproj121 Downloads are skipped if the file already exists LLama.Unittest/LLama.Unittest.csproj99-100 In the CI environment, the Models/ directory is cached using actions/cache to avoid re-downloading across runs.

---

Constants and Model Paths

All model file paths and environment-sensitive configuration live in Constants LLama.Unittest/Constants.cs5-15:

CIGpuLayerCount

CIGpuLayerCount is a computed property that returns a GPU layer count appropriate for the build configuration LLama.Unittest/Constants.cs20-34:

Build config	Return value	Effect
DEBUG	20	Uses GPU/Metal acceleration
RELEASE	0	Disables GPU; runs CPU-only on CI

This prevents tests from failing on CI runners that lack GPU/Metal support without requiring test-level skipping.

Sources: LLama.Unittest/Constants.cs1-36

---

CI Test Filtering

Tests are executed in CI using the --filter Category!=NoCI flag to exclude computationally prohibitive tests or those requiring specific local environments. Individual tests that are known to be slow are skipped using the Skip property on the Fact attribute LLama.Unittest/StatelessExecutorTest.cs59

Sources: LLama.Unittest/StatelessExecutorTest.cs59-61

---

Test Class Structure

Diagram: Test Class to Code Entity Mapping

Sources: LLama.Unittest/BasicTest.cs6-11 LLama.Unittest/LLamaContextTests.cs6-11 LLama.Unittest/LLamaContextWithCustomLoggerTests.cs7-28 LLama.Unittest/StatelessExecutorTest.cs8-14 LLama.Unittest/TokenTests.cs7-12 LLama.Unittest/EncodingExtensionsTests.cs6-7 LLama.Unittest/IEnumerableExtensionsTests.cs4-5 LLama.Unittest/DictionaryExtensionsTests.cs5-6

---

BasicTest

Uses Constants.GenerativeModelPath2 (SmolLM-360M) LLama.Unittest/BasicTest.cs16

Test method	What it checks
AdvancedModelProperties	Reads LLamaWeights.Metadata and asserts known key/value pairs from the GGUF header (architecture, embedding length, head count, etc.) LLama.Unittest/BasicTest.cs30-62

Sources: LLama.Unittest/BasicTest.cs16 LLama.Unittest/BasicTest.cs30-62

---

LLamaContextTests

Implements IDisposable LLama.Unittest/LLamaContextTests.cs7 Creates a shared LLamaWeights + LLamaContext pair in the constructor using Constants.GenerativeModelPath2 LLama.Unittest/LLamaContextTests.cs13-25

Test method	What it checks
CheckProperties	ContextSize, EmbeddingSize, Vocab.Count against expected values LLama.Unittest/LLamaContextTests.cs33-39
Tokenize	Exact token IDs for "The quick brown fox" LLama.Unittest/LLamaContextTests.cs41-47
TokenizeNewline	Newline tokenizes to token ID 198 LLama.Unittest/LLamaContextTests.cs49-55
TokenizeRoundtripSpecialStrings	Tab, newline, null, and combinations round-trip through StreamingTokenDecoder LLama.Unittest/LLamaContextTests.cs57-78
SaveLoadState	LLamaContext.GetState() → State.Save(stream) → State.Load(stream) preserves Size LLama.Unittest/LLamaContextTests.cs89-106
SaveLoadStateAsync	Async variant of state persistence LLama.Unittest/LLamaContextTests.cs108-126

Sources: LLama.Unittest/LLamaContextTests.cs7 LLama.Unittest/LLamaContextTests.cs13-25 LLama.Unittest/LLamaContextTests.cs33-39 LLama.Unittest/LLamaContextTests.cs41-47 LLama.Unittest/LLamaContextTests.cs49-55 LLama.Unittest/LLamaContextTests.cs57-78 LLama.Unittest/LLamaContextTests.cs89-106 LLama.Unittest/LLamaContextTests.cs108-126

---

LLamaContextWithCustomLoggerTests

Validates that setting a custom ILogger via NativeLogConfig does not cause crashes during model loading LLama.Unittest/LLamaContextWithCustomLoggerTests.cs40-44 This addresses a specific issue where a naive logger could cause crashes LLama.Unittest/LLamaContextWithCustomLoggerTests.cs38-44

Test method	What it checks
CheckProperties	Verifies context properties still load correctly after setting a custom logger LLama.Unittest/LLamaContextWithCustomLoggerTests.cs56-62

Sources: LLama.Unittest/LLamaContextWithCustomLoggerTests.cs38-44 LLama.Unittest/LLamaContextWithCustomLoggerTests.cs56-62

---

TokenTests

Focuses on tokenization, decoding, and string processing logic LLama.Unittest/TokenTests.cs7

Test method	What it checks
TokensEndWith	Checks if a sequence of tokens ends with a specific string using TokensEndsWithAnyString LLama.Unittest/TokenTests.cs29-40
TokensEndSubstring	Checks if a sequence of tokens ends with a substring using TokensEndsWithAnyString LLama.Unittest/TokenTests.cs43-52
TokensNotEndWith	Checks if a sequence of tokens does not end with a specific string LLama.Unittest/TokenTests.cs55-66
TokensNotEndWithNothing	Checks behavior when antiprompt list is empty LLama.Unittest/TokenTests.cs69-75
TokensEndWith2	Similar to TokensEndWith, but using AntipromptProcessor LLama.Unittest/TokenTests.cs78-93
TokensEndSubstring2	Similar to TokensEndSubstring, but using AntipromptProcessor LLama.Unittest/TokenTests.cs96-107
TokensNotEndWith2	Similar to TokensNotEndWith, but using AntipromptProcessor LLama.Unittest/TokenTests.cs110-125
TokensNotEndWithNothing2	Similar to TokensNotEndWithNothing, but using AntipromptProcessor LLama.Unittest/TokenTests.cs128-141
RoundTrip	Verifies that strings containing UTF-8 characters (Korean, Emojis) round-trip through tokenize/decode LLama.Unittest/TokenTests.cs144-169
StreamingDecoderRoundTrip	Tests StreamingTokenDecoder with incremental token additions LLama.Unittest/TokenTests.cs172-200

Sources: LLama.Unittest/TokenTests.cs7 LLama.Unittest/TokenTests.cs29-40 LLama.Unittest/TokenTests.cs43-52 LLama.Unittest/TokenTests.cs55-66 LLama.Unittest/TokenTests.cs69-75 LLama.Unittest/TokenTests.cs78-93 LLama.Unittest/TokenTests.cs96-107 LLama.Unittest/TokenTests.cs110-125 LLama.Unittest/TokenTests.cs128-141 LLama.Unittest/TokenTests.cs144-169 LLama.Unittest/TokenTests.cs172-200

---

StatelessExecutorTest

Creates LLamaWeights with a small context (ContextSize=60) to test that StatelessExecutor is truly stateless LLama.Unittest/StatelessExecutorTest.cs15-25

Test method	Category	What it checks
Stateless	[Fact]	Two identical InferAsync calls with the same prompt produce identical output (determinism) LLama.Unittest/StatelessExecutorTest.cs32-57
OutOfContext_WithTruncateStrategy_SuccessfullyGenerates	[Fact(Skip="...")]	Inference beyond ContextSize still completes coherently using TruncateAndReprefill LLama.Unittest/StatelessExecutorTest.cs59-81
OutOfContext_WithDefaultStrategy_ThrowsException	[Fact]	Verifies ContextOverflowException is thrown when context limit is reached without a truncation strategy LLama.Unittest/StatelessExecutorTest.cs86-127
OutOfContext_WithDefaultStrategy_2_ThrowsException	[Fact]	Verifies ContextOverflowException is thrown for InstructExecutor when context limit is reached LLama.Unittest/StatelessExecutorTest.cs129-162

Sources: LLama.Unittest/StatelessExecutorTest.cs15-25 LLama.Unittest/StatelessExecutorTest.cs32-57 LLama.Unittest/StatelessExecutorTest.cs59-81 LLama.Unittest/StatelessExecutorTest.cs86-127 LLama.Unittest/StatelessExecutorTest.cs129-162

---

Extension Method Tests

Several test classes are dedicated to verifying the functionality of extension methods:

• EncodingExtensionsTests: Tests LLama.Extensions.EncodingExtensions for GetCharsImpl and GetCharCountImpl with various string types, including empty, English, and Chinese characters LLama.Unittest/EncodingExtensionsTests.cs6-63
• IEnumerableExtensionsTests: Tests LLama.Extensions.IEnumerableExtensions for TakeLastImpl with empty, full, and partial collections LLama.Unittest/IEnumerableExtensionsTests.cs4-35
• DictionaryExtensionsTests: Tests LLama.Extensions.DictionaryExtensions for GetValueOrDefaultImpl with empty dictionaries, missing keys, and existing keys LLama.Unittest/DictionaryExtensionsTests.cs5-37
• KeyValuePairExtensionsTests: (Not shown in full, but follows similar pattern for LLama.Extensions.KeyValuePairExtensions).
• IReadOnlyListExtensionsTests: (Not shown in full, but follows similar pattern for LLama.Extensions.IReadOnlyListExtensions).

Sources: LLama.Unittest/EncodingExtensionsTests.cs6-63 LLama.Unittest/IEnumerableExtensionsTests.cs4-35 LLama.Unittest/DictionaryExtensionsTests.cs5-37

---

Fixture Lifecycle Pattern

All test classes that use model weights follow the same construct/dispose pattern to ensure native resources are freed:

Diagram: Test Fixture Lifecycle

Sources: LLama.Unittest/LLamaContextTests.cs13-31 LLama.Unittest/BasicTest.cs13-27 LLama.Unittest/StatelessExecutorTest.cs15-30 LLama.Unittest/TokenTests.cs13-26