Custom Samplers

This page explains how to create custom sampling logic by implementing the ICustomSampler interface and integrating it into a sampling pipeline. Custom samplers allow you to modify token probabilities or implement novel selection strategies beyond the built-in samplers provided by llama.cpp.

For information about the overall sampling pipeline architecture, see Sampling Pipeline Overview For details on the standard built-in samplers, see DefaultSamplingPipeline

---

The ICustomSampler Interface

Custom samplers must implement the ICustomSampler interface, which defines the contract for interacting with llama.cpp's native sampling chain. While the interface definition is primarily used to bridge managed code to llama.cpp, its implementation allows for direct manipulation of token candidates.

Based on the CustomSampler example and native interop requirements, a custom sampler must implement the following methods:

Method	Purpose	When Called
Name	Returns a string identifier for the sampler LLama.Examples/Examples/CustomSampler.cs72	During initialization and introspection
Apply(ref LLamaTokenDataArrayNative)	Modifies the logit distribution LLama.Examples/Examples/CustomSampler.cs74	Before each token is selected
Accept(LLamaToken)	Updates internal state after a token is chosen LLama.Examples/Examples/CustomSampler.cs96	After each token is sampled
Reset()	Resets the sampler's internal state LLama.Examples/Examples/CustomSampler.cs100	When the conversation context is reset
Clone()	Creates a deep copy of the sampler LLama.Examples/Examples/CustomSampler.cs104	When the sampling chain is cloned

Sources: LLama.Examples/Examples/CustomSampler.cs69-112

---

Custom Sampler Lifecycle and Memory Management

Custom samplers are integrated into the sampling pipeline via the BaseSamplingPipeline. When a pipeline is executed, it creates a native sampler chain that can include both built-in llama.cpp samplers and managed custom samplers.

Data Flow and Code Entity Mapping

The following diagram bridges the natural language concept of "Sampling" to the specific code entities in LLamaSharp.

Sampling Data Flow

Sources: LLama.Examples/Examples/CustomSampler.cs49-67 LLama/Sampling/BaseSamplingPipeline.cs7-23 LLama/Sampling/ISamplingPipeline.cs9-37

---

Working with Token Data Structures

Custom samplers manipulate token probabilities through the LLamaTokenDataArrayNative structure. This structure is a C# equivalent of llama_token_data_array and provides access to the underlying candidates.

LLamaTokenDataArrayNative Structure

The native structure must align exactly with the llama.cpp memory layout to ensure safe P/Invoke calls. It is used to pass candidate token data between the managed custom sampler and the native llama.cpp sampling logic.

Property	Type	Description
_data	LLamaTokenData*	Pointer to the array of LLamaTokenData LLama/Native/LLamaTokenDataArray.cs142
Size	ulong	Number of valid candidates in the array LLama/Native/LLamaTokenDataArray.cs191
Sorted	bool	Flag indicating if candidates are sorted by logit descending LLama/Native/LLamaTokenDataArray.cs173
Selected	long	The index of the selected token (not the token ID) LLama/Native/LLamaTokenDataArray.cs183

Implementation Details: Sorting and Modifying

When implementing Apply, it is critical to manage the Sorted state. If your sampler modifies logits such that the previous order is invalidated, you must set Sorted = false LLama.Examples/Examples/CustomSampler.cs93

Sources: LLama/Native/LLamaTokenDataArray.cs136-202 LLama.Examples/Examples/CustomSampler.cs74-94

---

Integration Example: Custom Pipeline

To use a custom sampler, you typically extend BaseSamplingPipeline and override CreateChain. This method is responsible for initializing a SafeLLamaSamplerChainHandle and appending the desired sampler stages.

Custom Pipeline Integration

Implementation Pattern:

1. Define the Sampler: Implement ICustomSampler LLama.Examples/Examples/CustomSampler.cs69-112
2. Define the Pipeline: Inherit from BaseSamplingPipeline and add your sampler to the chain using chain.AddCustom() LLama.Examples/Examples/CustomSampler.cs49-67
3. Assign to Inference: Set the SamplingPipeline property in InferenceParams LLama.Examples/Examples/CustomSampler.cs27-31

Sources: LLama.Examples/Examples/CustomSampler.cs8-67 LLama/Sampling/BaseSamplingPipeline.cs23-48

---

Use Cases and Performance

Custom samplers are powerful for enforcing specific constraints that standard Top-K or Top-P sampling cannot handle.

Common Use Cases

• Banning Tokens: Setting specific LLamaToken logits to float.NegativeInfinity LLama.Examples/Examples/CustomSampler.cs86
• Dynamic Biasing: Adjusting logits based on external state or complex heuristics.
• Stateful Selection: Using the Accept and Reset methods to track generation history and avoid repetitive patterns or specific forbidden sequences LLama.Examples/Examples/CustomSampler.cs96-102
• Grammar Constraints: While built-in, grammars are also integrated into chains via AddGrammar LLama/Sampling/DefaultSamplingPipeline.cs166

Performance Considerations

• Memory Layout: The LLamaTokenDataArrayNative layout is strictly defined to match the native llama.cpp implementation LLama/Native/LLamaTokenDataArray.cs136-155
• Native Chain Ownership: When you add a custom sampler to a SafeLLamaSamplerChainHandle, the native side manages the execution. Calling Reset() or Accept() on the chain handle propagates these calls to the native logic LLama/Sampling/BaseSamplingPipeline.cs62-71
• Cloning: The Clone() method is vital for ensuring each execution context has its own independent sampler state, especially when reusing pipeline configurations LLama.Examples/Examples/CustomSampler.cs104-107

Sources: LLama/Native/LLamaTokenDataArray.cs136-202 LLama/Sampling/BaseSamplingPipeline.cs35-71 LLama.Examples/Examples/CustomSampler.cs96-112