👁 Image
ElBruno.LocalLLMs.Rag 0.18.0

ElBruno.LocalLLMs

👁 NuGet
👁 NuGet Downloads
👁 Build Status
👁 HuggingFace
👁 .NET
👁 GitHub stars
👁 Twitter Follow

Run local LLMs in .NET through IChatClient 🧠

Run local LLMs in .NET through IChatClient — the same interface you'd use for Azure OpenAI, Ollama, or any other provider. Powered by ONNX Runtime GenAI and BitNet.

What's New

• ✅ Gemma 4 support path is now active (E2B, E4B, 12B Unified, 26B-A4B, 31B) via conversion workflows.
• 🔄 Gemma 4 status moved from pending to convert across the model tables and guides.
• ⬆️ ONNX Runtime GenAI upgraded to 0.14.1 across library, tests, samples, and benchmarks.

Features

• 🔌 IChatClient implementation — seamless integration with Microsoft.Extensions.AI
• 📦 Automatic model download — models are fetched from HuggingFace on first use
• 🚀 Zero friction — works out of the box with sensible defaults (Phi-3.5 mini)
• 🖥️ Multi-hardware — CPU, CUDA, and DirectML execution providers
• 💉 DI-friendly — register with AddLocalLLMs() or AddBitNetChatClient() in ASP.NET Core
• 🔄 Streaming — token-by-token streaming via GetStreamingResponseAsync
• 📊 Multi-model — switch between Phi-3.5, Phi-4, Qwen2.5, Llama 3.2, and more
• 🎯 Fine-tuned models — pre-trained Qwen2.5 variants for tool calling and RAG ()
• ⚡ BitNet support — run 1.58-bit ternary models via bitnet.cpp with extreme efficiency ()

Packages

Installation

dotnet add package ElBruno.LocalLLMs

Then add one runtime package depending on your target hardware:

# 🖥️ CPU (works everywhere — required for CPU-only apps):
dotnet add package Microsoft.ML.OnnxRuntimeGenAI

# 🟢 NVIDIA GPU (CUDA):
dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda

# 🔵 Any Windows GPU — AMD, Intel, NVIDIA (DirectML):
dotnet add package Microsoft.ML.OnnxRuntimeGenAI.DirectML

⚠️ Add exactly one runtime package. Do not reference both Microsoft.ML.OnnxRuntimeGenAI and Microsoft.ML.OnnxRuntimeGenAI.Cuda simultaneously — the native binaries conflict and GPU support will silently fail.

🚀 The library defaults to ExecutionProvider.Auto — it tries GPU first and falls back to CPU automatically. No code changes needed.

Quick Start

using ElBruno.LocalLLMs;
using Microsoft.Extensions.AI;

// Create a local chat client (downloads Phi-3.5 mini on first run)
using var client = await LocalChatClient.CreateAsync();

var response = await client.GetResponseAsync([
 new(ChatRole.User, "What is the capital of France?")
]);

Console.WriteLine(response.Text);

First Run

The first time you create a LocalChatClient, the model is downloaded from HuggingFace to your local cache directory (~2-4 GB). This typically takes 30-60 seconds depending on your internet connection.

Track download progress:

using var client = await LocalChatClient.CreateAsync(
 new LocalLLMsOptions { Model = KnownModels.Phi35MiniInstruct },
 progress: new Progress<ModelDownloadProgress>(p =>
 {
 var percent = (p.BytesDownloaded * 100) / p.TotalBytes;
 Console.WriteLine($"{p.FileName}: {percent:F1}%");
 })
);

Subsequent runs load instantly from cache (%LOCALAPPDATA%/ElBruno/LocalLLMs/models).

Skip auto-download if using a pre-downloaded model:

var options = new LocalLLMsOptions
{
 Model = KnownModels.Phi35MiniInstruct,
 ModelPath = "/path/to/local/model",
 EnsureModelDownloaded = false
};
using var client = await LocalChatClient.CreateAsync(options);

Streaming

using ElBruno.LocalLLMs;
using Microsoft.Extensions.AI;

using var client = await LocalChatClient.CreateAsync(new LocalLLMsOptions
{
 Model = KnownModels.Phi35MiniInstruct
});

await foreach (var update in client.GetStreamingResponseAsync([
 new(ChatRole.System, "You are a helpful assistant."),
 new(ChatRole.User, "Explain quantum computing in simple terms.")
]))
{
 Console.Write(update.Text);
}

GPU Acceleration

By default, ExecutionProvider.Auto tries GPU first (CUDA → DirectML) and falls back to CPU automatically:

// Use explicit GPU provider (fails if CUDA not installed; use Auto to fallback to CPU)
var options = new LocalLLMsOptions
{
 ExecutionProvider = ExecutionProvider.Cuda
};

// Multi-GPU systems: select device ID
var options2 = new LocalLLMsOptions
{
 ExecutionProvider = ExecutionProvider.Cuda,
 GpuDeviceId = 1 // Use second GPU
};

Auto fallback behavior:

• CUDA available → uses NVIDIA GPU
• CUDA unavailable, DirectML available → uses AMD/Intel Arc GPU
• GPU unavailable → falls back to CPU (no errors, just slower)

See for debugging GPU issues.

Model Metadata

Inspect model capabilities at runtime — context window size, model name, and vocabulary:

using var client = await LocalChatClient.CreateAsync();

var metadata = client.ModelInfo;
Console.WriteLine($"Model: {metadata?.ModelName}");
Console.WriteLine($"Context window: {metadata?.MaxSequenceLength}");
Console.WriteLine($"Vocab size: {metadata?.VocabSize}");

This is useful for prompt-length validation, adaptive chunking, and model selection logic.

Dependency Injection

builder.Services.AddLocalLLMs(options =>
{
 options.Model = KnownModels.Phi35MiniInstruct;
 options.ExecutionProvider = ExecutionProvider.DirectML;
});

// Inject IChatClient anywhere
public class MyService(IChatClient chatClient) { ... }

Error Handling

The library provides structured exception types for graceful error handling:

using ElBruno.LocalLLMs;
using Microsoft.Extensions.AI;

try
{
 using var client = await LocalChatClient.CreateAsync();
 var response = await client.GetResponseAsync([
 new(ChatRole.User, "Your question here")
 ]);
}
catch (ExecutionProviderException ex)
{
 // GPU/provider-specific error (no CUDA, DirectML not available, etc.)
 Console.WriteLine($"Provider error: {ex.Message}");
}
catch (ModelCapacityExceededException ex)
{
 // Prompt/response too long for model's context window
 Console.WriteLine($"Capacity error: {ex.Message}");
 // Solution: use a larger model or truncate the prompt
}
catch (InvalidOperationException ex)
{
 // General operation error (model not found, download failed, etc.)
 Console.WriteLine($"Operation error: {ex.Message}");
}

Troubleshooting

GPU not working? Use ExecutionProvider.Cpu explicitly. See .

Out of memory? Try a smaller model:

var options = new LocalLLMsOptions
{
 Model = KnownModels.Qwen25_05BInstruct // 0.5B instead of 3.8B
};

Model download fails?

• Check your internet connection
• For private HuggingFace models, set the HF_TOKEN environment variable

For detailed troubleshooting, see .

Supported Models

🔄 Convert = Use the conversion scripts in scripts/ to export ONNX locally before running the model.

Fine-Tuned Models

Pre-trained variants optimized for specific tasks. A fine-tuned 0.5B model often matches or exceeds a base 1.5B on its specialized task.

See the for detailed model cards, performance benchmarks, and selection guidance.

Samples

Requirements

• .NET 8.0 or .NET 10.0
• CPU (default), NVIDIA GPU (CUDA), or Windows GPU (DirectML)
• ~2-8 GB disk space per model (depending on size and quantization)

Building from Source

git clone https://github.com/elbruno/ElBruno.LocalLLMs.git
cd ElBruno.LocalLLMs
dotnet restore ElBruno.LocalLLMs.slnx
dotnet build ElBruno.LocalLLMs.slnx
dotnet test ElBruno.LocalLLMs.slnx --framework net8.0

Documentation

• — installation, first steps, configuration
• — full model reference with tiers, specs, decision tree
• — setup and usage of 1.58-bit BitNet models
• — design decisions and internal structure
• — walkthrough of each sample application
• — how to run and interpret performance benchmarks
• — using and training fine-tuned models
• — converting HuggingFace models to ONNX format
• — NuGet package publishing with OIDC
• — how to contribute
• — version history

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License — see the file for details.

👋 About the Author

Made with ❤️ by Bruno Capuano (ElBruno)

• 📝 Blog: elbruno.com
• 📺 YouTube: youtube.com/elbruno
• 🔗 LinkedIn: linkedin.com/in/elbruno
• 𝕏 Twitter: twitter.com/elbruno
• 🎙️ Podcast: notienenombre.com

🙏 Acknowledgments

• ONNX Runtime GenAI — inference engine
• BitNet / bitnet.cpp — 1.58-bit ternary model inference
• Microsoft.Extensions.AI — IChatClient interface
• Hugging Face — model hosting and community