Package Architecture

Purpose and Scope

This document explains the modular package structure of LLamaSharp and how the different NuGet packages are organized and distributed. It covers the core library package, backend packages for different hardware platforms, integration packages for Microsoft AI frameworks, and experimental mobile/auto-download support.

For information about installing these packages and setting up your development environment, see 1.2 Installation and Setup For details about the runtime architecture and how these packages interact with native libraries at runtime, see 2.1 Native Library Integration

Package Structure Overview

LLamaSharp follows a modular distribution strategy that separates concerns into distinct NuGet packages. This design allows users to install only the components they need, reducing package size and avoiding unnecessary dependencies.

The package ecosystem consists of four primary categories:

Package Category	Purpose	Target Users
Core Library	Provides the fundamental API and abstractions for LLM inference	All users
Backend Packages	Contains platform-specific native binaries compiled from llama.cpp	Users select one based on hardware
Integration Packages	Extends LLamaSharp to work with Microsoft AI frameworks	Users of Semantic Kernel or Kernel Memory
Experimental/Mobile	Advanced features (auto-download) and MAUI mobile support	Advanced users and mobile developers

Sources: README.md87-108 LLama.Experimental/LLama.Experimental.csproj19-22 Llama.Mobile/Llama.Mobile.csproj15-31

NuGet Package Distribution

The following diagram shows the complete NuGet package ecosystem and how packages relate to each other:

LLamaSharp Ecosystem Relationship

Sources: README.md5-11 LLama/LLamaSharp.csproj30 LLama.SemanticKernel/LLamaSharp.SemanticKernel.csproj31 LLama.KernelMemory/LLamaSharp.KernelMemory.csproj25 LLama.Experimental/LLama.Experimental.csproj29 Llama.Mobile/Llama.Mobile.csproj79

Core Library Package

The LLamaSharp package is the primary library that provides all high-level APIs for working with LLMs. It contains the managed .NET code but does not include native binaries.

Package Metadata

Property	Value
PackageId	LLamaSharp
Version	0.27.0
Target Frameworks	netstandard2.0, net8.0
Platforms	AnyCPU, x64, Arm64

Sources: LLama/LLamaSharp.csproj3-30

Core Dependencies

The core library has minimal external dependencies to maximize compatibility, including the Microsoft.Extensions.AI.Abstractions for standardized AI interfaces.

Core Package Dependencies

Sources: LLama/LLamaSharp.csproj44-59

Binary Distribution Mechanism

The core package includes MSBuild targets that automatically download and extract native binaries during build for development environments. This mechanism uses a BinaryReleaseId to track versions of llama.cpp dependencies.

MSBuild Binary Retrieval Flow

Sources: LLama/LLamaSharp.csproj61-100

Backend Packages

Backend packages contain platform-specific native binaries compiled from llama.cpp. These packages use .props files to ensure native libraries are copied to the output directory and correctly linked for various platforms, including Android.

Available Backend Packages

Package Name	Platforms	Hardware Support
LLamaSharp.Backend.Cpu	Windows, Linux, macOS	CPU inference; Metal GPU on macOS
LLamaSharp.Backend.Cuda11	Windows, Linux	NVIDIA GPU with CUDA 11.x
LLamaSharp.Backend.Cuda12	Windows, Linux	NVIDIA GPU with CUDA 12.x
LLamaSharp.Backend.Vulkan	Windows, Linux	Vulkan-compatible GPUs

Sources: README.md99-105 LLama/runtimes/build/LLamaSharp.Backend.Cuda12.Windows.nuspec4-12 LLama/runtimes/build/LLamaSharp.Backend.Vulkan.Windows.nuspec4-12

Native Library Mapping (Android Example)

The backend packages for mobile use AndroidNativeLibrary items to map native .so files (like libllama.so, libggml.so, and libmtmd.so) to specific ABIs.

Native Library	ABI	Source Path
libllama.so	arm64-v8a	runtimes/deps/android-arm64-v8a/libllama.so
libggml.so	x86_64	runtimes/deps/android-x86_64/libggml.so

Sources: LLama/runtimes/build/LLamaSharpBackend.props57-84 LLama/runtimes/build/LLamaSharp.Backend.Cpu.Android.nuspec21-31

Integration Packages

Integration packages extend LLamaSharp to work seamlessly with Microsoft's AI ecosystem.

Semantic Kernel Integration

The LLamaSharp.semantic-kernel package provides integration with Microsoft Semantic Kernel, enabling LLamaSharp to act as a local LLM backend for SK-based applications.

Metadata:

• PackageId: LLamaSharp.semantic-kernel LLama.SemanticKernel/LLamaSharp.SemanticKernel.csproj31
• Target Frameworks: netstandard2.0, net8.0 LLama.SemanticKernel/LLamaSharp.SemanticKernel.csproj4
• Dependencies: Microsoft.SemanticKernel.Abstractions (v1.64.0) LLama.SemanticKernel/LLamaSharp.SemanticKernel.csproj37

Kernel Memory Integration

The LLamaSharp.kernel-memory package provides integration with Microsoft Kernel Memory for Retrieval-Augmented Generation (RAG) scenarios.

Metadata:

• PackageId: LLamaSharp.kernel-memory LLama.KernelMemory/LLamaSharp.KernelMemory.csproj25
• Target Frameworks: net8.0 LLama.KernelMemory/LLamaSharp.KernelMemory.csproj4
• Dependencies: Microsoft.KernelMemory.Abstractions (v0.98.250508.3) LLama.KernelMemory/LLamaSharp.KernelMemory.csproj30

Experimental and Mobile Support

Experimental Auto-Download

The LLama.Experimental package includes features for automatic native library downloading from Hugging Face via the NativeLibraryDownloader class. This allows applications to fetch the required native binaries at runtime based on environment detection.

• PackageId: LLamaSharp.Experimental LLama.Experimental/LLama.Experimental.csproj29
• Dependencies: HuggingfaceHub (v0.1.3) LLama.Experimental/LLama.Experimental.csproj35
• Key Components: NativeLibraryDownloadSettings provides configuration for endpoints, repo IDs, and tags for the download process LLama.Experimental/Native/NativeLibraryDownloader.cs60-115

Mobile Support (MAUI)

The Llama.Mobile project demonstrates LLamaSharp running on mobile platforms using .NET MAUI.

• Platforms: Supports net8.0-android, net8.0-ios, net8.0-maccatalyst Llama.Mobile/Llama.Mobile.csproj15
• Integration: References the core LLamaSharp project directly Llama.Mobile/Llama.Mobile.csproj79

Sources: LLama.Experimental/LLama.Experimental.csproj1-42 Llama.Mobile/Llama.Mobile.csproj1-82 LLama.Experimental/Native/NativeLibraryDownloader.cs1-115

Examples and Web Projects

The repository includes several demonstration projects:

• LLama.Examples: A console application for feature demonstration LLamaSharp.sln8
• LLama.Web: An ASP.NET Core application for SignalR-based chat LLamaSharp.sln14
• LLama.WebAPI: A REST API implementation LLamaSharp.sln12

Sources: LLamaSharp.sln8-14

Version Synchronization

All LLamaSharp packages maintain synchronized version numbers to ensure compatibility across the ecosystem.

Package	Current Version
LLamaSharp	0.27.0
LLamaSharp.semantic-kernel	0.27.0
LLamaSharp.kernel-memory	0.27.0

Sources: LLama/LLamaSharp.csproj10 LLama.SemanticKernel/LLamaSharp.SemanticKernel.csproj13 LLama.KernelMemory/LLamaSharp.KernelMemory.csproj7