👁 Image
SiLA2.Database.NoSQL 10.2.4

SiLA2.Database.NoSQL

Lightweight NoSQL Document Database for SiLA2 Applications

NuGet Package	SiLA2.Database.NoSQL on NuGet.org
Repository	https://gitlab.com/SiLA2/sila_csharp
SiLA Standard	https://sila-standard.com
License	MIT

Overview

SiLA2.Database.NoSQL is an optional component of the sila_csharp implementation that provides lightweight, embedded NoSQL document database capabilities using LiteDB. This module offers a repository pattern abstraction for document-oriented data storage, eliminating the need for external database servers while maintaining flexibility for complex data structures.

Key Value Proposition

Traditional SQL Approach:

Entity Framework → SQL Server/SQLite → Relational Tables → Join Queries

NoSQL Approach:

Repository Pattern → LiteDB → BSON Documents → Direct Object Storage

This package is ideal for:

• AnIML scientific data storage - Store experimental results in flexible document format
• Configuration and settings - Persist complex configuration objects without schema migrations
• Audit logs and event storage - Store heterogeneous log data with varying structures
• Embedded applications - No external database server required, single-file deployment
• Rapid prototyping - Schema-less design allows quick iteration without migrations

When to Use NoSQL vs SQL in SiLA2 Applications

Scenario	Use SiLA2.Database.NoSQL	Use SiLA2.Database.SQL
Storing AnIML experimental data	✅ Yes (recommended)	❌ No (poor fit for hierarchical data)
Configuration and settings	✅ Yes	⚠️ Optional
User authentication/authorization	❌ No	✅ Yes (better relational integrity)
Audit logs with flexible schemas	✅ Yes	⚠️ Optional
Complex relational queries	❌ No	✅ Yes (SQL is better)
Large datasets (>100GB)	❌ No (use SQL or specialized DB)	✅ Yes
Schema-less or evolving data models	✅ Yes	❌ No (requires migrations)
Single-file deployment requirement	✅ Yes (embedded)	⚠️ Partial (SQLite only)

Installation

Install via NuGet Package Manager:

dotnet add package SiLA2.Database.NoSQL

Or via Package Manager Console:

Install-Package SiLA2.Database.NoSQL

Requirements

• .NET 10.0+
• LiteDB 5.0.21+ (automatically installed as dependency)
• Microsoft.Extensions.DependencyInjection 10.0.2+ (for ASP.NET Core integration)

Core Concepts

1. LiteDB - Embedded NoSQL Database

LiteDB is a serverless, embedded NoSQL database for .NET, similar to SQLite but designed for document storage rather than relational data.

Key Characteristics:

• Single-file database - Entire database stored in one .db file
• BSON document format - Binary JSON for efficient serialization
• No server required - Runs in-process with your application
• ACID transactions - Supports atomic operations and rollback
• Indexed queries - Create indexes on document properties for fast lookups
• Thread-safe - Concurrent read/write with proper locking
• Small footprint - ~450KB assembly size

Comparison to Other Embedded Databases:

Feature	LiteDB	SQLite	RavenDB Embedded
Data Model	Document (BSON)	Relational (SQL)	Document (JSON)
Schema	Schema-less	Fixed schema	Schema-less
Query Language	LINQ-like	SQL	RQL/LINQ
File Size	Single file	Single file	Multiple files
Performance	Good for documents	Excellent for relational	Excellent (commercial)
License	MIT (free)	Public domain	AGPL/Commercial
Best For	SiLA2 NoSQL needs	SiLA2 SQL needs	Enterprise applications

2. Repository Pattern

The repository pattern abstracts data access logic, providing a clean separation between business logic and data persistence. This module implements a generic repository interface that works with any entity type.

Benefits:

• Testability - Easy to mock repositories for unit testing
• Consistency - Standard CRUD interface across all entity types
• Flexibility - Switch database implementations without changing business logic
• Encapsulation - Hide LiteDB-specific code from application layer

Pattern Structure:

IBaseRepository<T> (Interface)
 ↓
BaseRepository<T> (Abstract Base Class)
 ↓
MyEntityRepository (Concrete Implementation)

3. BSON Document Storage

BSON (Binary JSON) is the native storage format for LiteDB. Entities are automatically serialized to BSON when saved.

Supported Data Types:

• Primitive types: int, long, double, decimal, bool, string
• Date/time: DateTime, DateTimeOffset
• Identifiers: ObjectId, Guid, int, long
• Collections: List<T>, Dictionary<TKey, TValue>, arrays
• Nested objects: Complex types and object graphs
• Binary data: byte[] for blobs

Attributes for Customization:

using LiteDB;

public class MyEntity
{
 [BsonId] // Marks the identifier property
 public ObjectId Id { get; set; }

 [BsonField("name")] // Custom field name in BSON
 public string DisplayName { get; set; }

 [BsonIgnore] // Exclude from serialization
 public string InternalCache { get; set; }

 [BsonCtor] // Mark constructor for deserialization
 public MyEntity(string name) { ... }
}

4. When to Use NoSQL vs SQL in SiLA2 Applications

Use NoSQL (SiLA2.Database.NoSQL) when:

• Data has complex, nested, or hierarchical structures (AnIML documents)
• Schema changes frequently or is unknown at design time
• You need embedded database with no server setup
• Data is naturally document-oriented (JSON/XML equivalents)
• You want simple CRUD operations without complex queries

Use SQL (SiLA2.Database.SQL) when:

• Data has strong relational integrity requirements (foreign keys, cascades)
• You need complex queries with joins, aggregations, and reporting
• Multi-user access with high concurrency and ACID guarantees
• Data volume exceeds 100GB or requires scalability
• You need mature tooling and database administration features

Can Use Both: Many SiLA2 servers use both modules:

• SQL for user authentication, feature configuration, device state
• NoSQL for AnIML experimental data, audit logs, temporary results

Architecture & Components

Component Overview

┌─────────────────────────────────────────────────────────────┐
│ SiLA2 Application Layer │
│ (Feature Services, Business Logic) │
└───────────────────────────┬─────────────────────────────────┘
 │
 ▼
┌─────────────────────────────────────────────────────────────┐
│ MyEntityRepository : BaseRepository<MyEntity> │
│ (Custom repository with domain-specific methods) │
└───────────────────────────┬─────────────────────────────────┘
 │
 ▼
┌─────────────────────────────────────────────────────────────┐
│ BaseRepository<T> : IBaseRepository<T> │
│ - Create(T) / Create(T, out object) │
│ - All() → IEnumerable<T> │
│ - FindById(object) → T │
│ - Update(T) │
│ - Delete(object) → bool │
│ + Collection → ILiteCollection<T> (direct access) │
│ + DB → ILiteDatabase (advanced operations) │
└───────────────────────────┬─────────────────────────────────┘
 │
 ▼
┌─────────────────────────────────────────────────────────────┐
│ ILiteDatabase │
│ (LiteDB instance - manages collections, transactions) │
└───────────────────────────┬─────────────────────────────────┘
 │
 ▼
┌─────────────────────────────────────────────────────────────┐
│ database.db (BSON file on disk) │
└─────────────────────────────────────────────────────────────┘

IBaseRepository<T>

Purpose: Defines the standard CRUD interface for all entity repositories.

Interface Definition:

public interface IBaseRepository<T>
{
 T Create(T data, out object id); // Create with ID output
 T Create(T data); // Create without ID output
 IEnumerable<T> All(); // Get all documents
 T FindById(object id); // Get by ID
 void Update(T entity); // Update or insert (upsert)
 bool Delete(object id); // Delete by ID
}

Design Philosophy:

• Generic type parameter - Works with any entity class
• Minimal surface area - Only essential CRUD operations
• Extensible - Inherit and add domain-specific methods
• Flexible ID types - Supports ObjectId, Guid, int, long, etc.

BaseRepository<T>

Purpose: Provides concrete implementation of IBaseRepository<T> using LiteDB as the backing store.

Key Features:

• Automatic collection management - Collection created on first access
• Virtual methods - All methods can be overridden in derived classes
• Direct collection access - Collection property for advanced queries
• Database access - DB property for transactions and multi-collection operations

Protected Members:

public abstract class BaseRepository<T> : IBaseRepository<T>
{
 public ILiteDatabase DB { get; } // Access to database
 public ILiteCollection<T> Collection { get; } // Access to collection

 protected BaseRepository(ILiteDatabase db) // Constructor
}

Extension Points: Derived classes can:

1. Override virtual methods to customize behavior
2. Use Collection for advanced LINQ queries
3. Use DB for transactions spanning multiple collections
4. Add domain-specific query methods

DocumentDbServiceCollectionExtensions

Purpose: Simplifies dependency injection setup for LiteDB in ASP.NET Core applications.

Extension Methods:

public static class DocumentDbServiceCollectionExtensions
{
 // Register ILiteDatabase as singleton
 void RegisterDocumentDatabase(
 this IServiceCollection services,
 string path,
 out ILiteDatabase db);

 // Register ILiteCollection<T> as singleton
 void RegisterDocumentDatabaseTypes<T>(
 this IServiceCollection services,
 ILiteDatabase db);
}

Usage Pattern:

// In Program.cs or Startup.cs
services.RegisterDocumentDatabase("myapp.db", out var db);
services.RegisterDocumentDatabaseTypes<Customer>(db);
services.RegisterDocumentDatabaseTypes<Order>(db);

Usage Examples

Setup

All examples assume you have registered the database in your dependency injection container. See the "Dependency Injection Setup" example below.

Example 1: Basic Entity and Repository

Step 1: Define Your Entity Class

using LiteDB;

public class TemperatureReading
{
 [BsonId]
 public ObjectId Id { get; set; }

 public DateTime Timestamp { get; set; }

 public double Temperature { get; set; }

 public string SensorId { get; set; }

 public string Unit { get; set; } = "Celsius";
}

Step 2: Create a Repository

using SiLA2.Database.NoSQL;
using LiteDB;

public interface ITemperatureReadingRepository : IBaseRepository<TemperatureReading>
{
 // Add domain-specific query methods here
 IEnumerable<TemperatureReading> GetReadingsBySensor(string sensorId);
 IEnumerable<TemperatureReading> GetReadingsInRange(DateTime start, DateTime end);
}

public class TemperatureReadingRepository : BaseRepository<TemperatureReading>,
 ITemperatureReadingRepository
{
 public TemperatureReadingRepository(ILiteDatabase db) : base(db)
 {
 // Optional: Create indexes for better query performance
 Collection.EnsureIndex(x => x.SensorId);
 Collection.EnsureIndex(x => x.Timestamp);
 }

 public IEnumerable<TemperatureReading> GetReadingsBySensor(string sensorId)
 {
 return Collection.Query()
 .Where(x => x.SensorId == sensorId)
 .OrderByDescending(x => x.Timestamp)
 .ToList();
 }

 public IEnumerable<TemperatureReading> GetReadingsInRange(DateTime start, DateTime end)
 {
 return Collection.Query()
 .Where(x => x.Timestamp >= start && x.Timestamp <= end)
 .ToList();
 }
}

Example 2: Dependency Injection Setup

In Program.cs (ASP.NET Core):

using SiLA2.Database.NoSQL;
using LiteDB;

var builder = WebApplication.CreateBuilder(args);

// Method 1: Using extension methods (recommended)
builder.Services.RegisterDocumentDatabase(
 "Data/MyAppDatabase.db",
 out ILiteDatabase db);

// Register collections for specific types
builder.Services.RegisterDocumentDatabaseTypes<TemperatureReading>(db);
builder.Services.RegisterDocumentDatabaseTypes<AnIMLType>(db);

// Register repositories
builder.Services.AddSingleton<ITemperatureReadingRepository, TemperatureReadingRepository>();

// Method 2: Manual registration with connection string options
builder.Services.AddSingleton<ILiteDatabase>(sp =>
 new LiteDatabase("filename=Data/MyApp.db;password=mySecretPassword"));

var app = builder.Build();

Connection String Options:

LiteDB supports various connection string parameters:

// Encrypted database
"filename=data.db;password=myPassword"

// Read-only mode
"filename=data.db;readonly=true"

// Upgrade from older LiteDB version
"filename=data.db;upgrade=true"

// Custom timeout (milliseconds)
"filename=data.db;timeout=30000"

// Shared mode for multi-process access (Windows only)
"filename=data.db;mode=Shared"

// Complete example
"filename=C:\\Data\\myapp.db;password=secret;timeout=60000;upgrade=true"

Example 3: CRUD Operations

using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/[controller]")]
public class TemperatureController : ControllerBase
{
 private readonly ITemperatureReadingRepository _repository;

 public TemperatureController(ITemperatureReadingRepository repository)
 {
 _repository = repository;
 }

 // CREATE - Add new reading
 [HttpPost]
 public IActionResult CreateReading([FromBody] TemperatureReading reading)
 {
 reading.Timestamp = DateTime.UtcNow;
 var created = _repository.Create(reading, out object newId);

 return CreatedAtAction(
 nameof(GetReading),
 new { id = newId.ToString() },
 created);
 }

 // READ - Get all readings
 [HttpGet]
 public IActionResult GetAllReadings()
 {
 var readings = _repository.All();
 return Ok(readings);
 }

 // READ - Get by ID
 [HttpGet("{id}")]
 public IActionResult GetReading(string id)
 {
 var reading = _repository.FindById(new ObjectId(id));
 if (reading == null)
 return NotFound();

 return Ok(reading);
 }

 // UPDATE - Modify existing reading
 [HttpPut("{id}")]
 public IActionResult UpdateReading(string id, [FromBody] TemperatureReading reading)
 {
 reading.Id = new ObjectId(id);
 _repository.Update(reading); // Upsert operation
 return NoContent();
 }

 // DELETE - Remove reading
 [HttpDelete("{id}")]
 public IActionResult DeleteReading(string id)
 {
 bool deleted = _repository.Delete(new ObjectId(id));
 if (!deleted)
 return NotFound();

 return NoContent();
 }

 // Custom query method
 [HttpGet("sensor/{sensorId}")]
 public IActionResult GetReadingsBySensor(string sensorId)
 {
 var readings = _repository.GetReadingsBySensor(sensorId);
 return Ok(readings);
 }
}

Example 4: Advanced Queries Using Collection Property

The Collection property provides direct access to LiteDB's powerful query API:

public class TemperatureReadingRepository : BaseRepository<TemperatureReading>,
 ITemperatureReadingRepository
{
 public TemperatureReadingRepository(ILiteDatabase db) : base(db) { }

 // Query with multiple conditions
 public IEnumerable<TemperatureReading> GetHighTemperatureReadings(
 string sensorId,
 double threshold)
 {
 return Collection.Query()
 .Where(x => x.SensorId == sensorId && x.Temperature > threshold)
 .OrderByDescending(x => x.Temperature)
 .Limit(100)
 .ToList();
 }

 // Pagination
 public IEnumerable<TemperatureReading> GetReadingsPage(int page, int pageSize)
 {
 return Collection.Query()
 .OrderByDescending(x => x.Timestamp)
 .Skip((page - 1) * pageSize)
 .Limit(pageSize)
 .ToList();
 }

 // Aggregation (count)
 public int GetReadingCountBySensor(string sensorId)
 {
 return Collection.Count(x => x.SensorId == sensorId);
 }

 // Full-text search (requires index)
 public IEnumerable<TemperatureReading> SearchBySensorName(string searchTerm)
 {
 return Collection.Query()
 .Where(x => x.SensorId.Contains(searchTerm))
 .ToList();
 }

 // Exists check
 public bool HasReadingsInRange(DateTime start, DateTime end)
 {
 return Collection.Exists(x => x.Timestamp >= start && x.Timestamp <= end);
 }

 // Delete multiple documents
 public int DeleteOldReadings(DateTime cutoffDate)
 {
 return Collection.DeleteMany(x => x.Timestamp < cutoffDate);
 }

 // Update multiple documents
 public int UpdateSensorName(string oldName, string newName)
 {
 var readings = Collection.Query()
 .Where(x => x.SensorId == oldName)
 .ToList();

 foreach (var reading in readings)
 {
 reading.SensorId = newName;
 Collection.Update(reading);
 }

 return readings.Count;
 }
}

Example 5: Using with AnIML Data (SiLA2.AnIML Integration)

This is the primary use case for the NoSQL module in SiLA2 applications.

using AnIMLCore;
using SiLA2.Database.NoSQL;
using SiLA2.AnIML.Services;
using LiteDB;

// Entity model for AnIML documents
public class AnIMLRepository : BaseRepository<AnIMLType>, IAnIMLRepository
{
 public AnIMLRepository(ILiteDatabase db) : base(db)
 {
 // AnIML documents are complex hierarchical structures
 // NoSQL is ideal for this use case
 }
}

// In a SiLA2 feature service
public class DataStorageFeatureService : DataStorageFeature.DataStorageFeatureBase
{
 private readonly IAnIMLRepository _anIMLRepository;
 private readonly ISeriesTypeBuilder _builder;
 private readonly ISeriesTypeProvider _provider;

 public DataStorageFeatureService(
 IAnIMLRepository anIMLRepository,
 ISeriesTypeBuilder builder,
 ISeriesTypeProvider provider)
 {
 _anIMLRepository = anIMLRepository;
 _builder = builder;
 _provider = provider;
 }

 public override Task<StoreData_Responses> StoreData(
 StoreData_Parameters request,
 ServerCallContext context)
 {
 // Build AnIML document from experiment data
 var animl = _builder.Build(new[] { request.SampleName.Value });
 animl.CreateAnIMLSeries(numberOfSeries: 2);

 // Add time series data
 var timestamps = request.Timestamps.Select(t => t.ToDateTime()).ToArray();
 var values = request.Values.Select(v => v.Value).ToArray();

 var seriesSet = animl.ExperimentStepSet.ExperimentStep[0].Result[0].SeriesSet;
 seriesSet.Series[0] = _provider.GetSeriesType(timestamps, DependencyType.independent);
 seriesSet.Series[1] = _provider.GetSeriesType(values, DependencyType.dependent);

 // Store in NoSQL database
 var created = _anIMLRepository.Create(animl, out object documentId);

 return Task.FromResult(new StoreData_Responses
 {
 DocumentId = new String { Value = documentId.ToString() }
 });
 }

 public override Task<RetrieveData_Responses> RetrieveData(
 RetrieveData_Parameters request,
 ServerCallContext context)
 {
 // Retrieve AnIML document from NoSQL database
 var animl = _anIMLRepository.FindById(new ObjectId(request.DocumentId.Value));

 if (animl == null)
 throw new RpcException(new Status(StatusCode.NotFound, "Document not found"));

 // Extract data from complex hierarchical structure
 var series = animl.GetAnIMLSeries(experimentStep: 0, result: 0, focusedSeries: 1);

 return Task.FromResult(new RetrieveData_Responses
 {
 Data = new String { Value = animl.GetAnIMLXml() }
 });
 }
}

Example 6: Complex Entity with Nested Objects

using LiteDB;

public class ExperimentConfiguration
{
 [BsonId]
 public ObjectId Id { get; set; }

 public string ConfigurationName { get; set; }

 public DateTime CreatedAt { get; set; }

 // Nested object
 public TemperatureSettings TemperatureSettings { get; set; }

 // List of complex objects
 public List<ProcessStep> ProcessSteps { get; set; }

 // Dictionary
 public Dictionary<string, string> Metadata { get; set; }
}

public class TemperatureSettings
{
 public double TargetTemperature { get; set; }
 public double Tolerance { get; set; }
 public string Unit { get; set; }
}

public class ProcessStep
{
 public int StepNumber { get; set; }
 public string Action { get; set; }
 public TimeSpan Duration { get; set; }
 public Dictionary<string, object> Parameters { get; set; }
}

// Repository with complex queries
public class ExperimentConfigurationRepository : BaseRepository<ExperimentConfiguration>
{
 public ExperimentConfigurationRepository(ILiteDatabase db) : base(db)
 {
 // Index on nested property
 Collection.EnsureIndex(x => x.TemperatureSettings.TargetTemperature);
 }

 // Query nested properties
 public IEnumerable<ExperimentConfiguration> GetByTargetTemperature(double temp)
 {
 return Collection.Query()
 .Where(x => x.TemperatureSettings.TargetTemperature == temp)
 .ToList();
 }

 // Query list elements
 public IEnumerable<ExperimentConfiguration> GetByStepAction(string action)
 {
 return Collection.Query()
 .Where("$.ProcessSteps[*].Action ANY = @0", action)
 .ToList();
 }
}

// Usage
var config = new ExperimentConfiguration
{
 ConfigurationName = "Standard Heating Protocol",
 CreatedAt = DateTime.UtcNow,
 TemperatureSettings = new TemperatureSettings
 {
 TargetTemperature = 80.0,
 Tolerance = 2.0,
 Unit = "Celsius"
 },
 ProcessSteps = new List<ProcessStep>
 {
 new ProcessStep
 {
 StepNumber = 1,
 Action = "Heat",
 Duration = TimeSpan.FromMinutes(30),
 Parameters = new Dictionary<string, object>
 {
 { "RampRate", 5.0 },
 { "HoldTime", 60 }
 }
 }
 },
 Metadata = new Dictionary<string, string>
 {
 { "Author", "Lab Technician" },
 { "Version", "1.0" }
 }
};

repository.Create(config);

Example 7: Identifier Strategies

LiteDB supports multiple identifier types. Choose the one that fits your use case:

using LiteDB;

// Option 1: ObjectId (MongoDB-style, recommended for new projects)
public class EntityWithObjectId
{
 [BsonId]
 public ObjectId Id { get; set; } // Auto-generated, globally unique
}

// Option 2: Guid (Windows-friendly, globally unique)
public class EntityWithGuid
{
 [BsonId]
 public Guid Id { get; set; }
}

// Usage:
var entity = new EntityWithGuid { Id = Guid.NewGuid() };
repository.Create(entity);

// Option 3: Auto-increment integer (simple, sequential)
public class EntityWithAutoIncrement
{
 [BsonId(autoId: true)]
 public int Id { get; set; } // Auto-incremented by LiteDB
}

// Option 4: String ID (custom identifiers)
public class EntityWithStringId
{
 [BsonId]
 public string Id { get; set; }
}

// Usage:
var entity = new EntityWithStringId { Id = "TEMP-2024-001" };
repository.Create(entity);

// Option 5: Composite key (custom logic)
public class EntityWithCompositeKey
{
 [BsonId]
 public string CompositeId => $"{SensorId}_{Timestamp:yyyyMMddHHmmss}";

 [BsonIgnore]
 public string SensorId { get; set; }

 [BsonIgnore]
 public DateTime Timestamp { get; set; }
}

Identifier Type Comparison:

Type	Auto-Generated	Globally Unique	Human-Readable	Best For
ObjectId	✅ Yes	✅ Yes	❌ No	General purpose, distributed systems
Guid	⚠️ Manual	✅ Yes	❌ No	Integration with .NET/Windows systems
int (auto)	✅ Yes	❌ No	✅ Yes	Single database, sequential IDs
string	❌ No	⚠️ Depends	✅ Yes	Custom business identifiers
Composite	❌ No	⚠️ Depends	✅ Yes	Natural keys, multi-part identifiers

Advanced Topics

Transactions

LiteDB supports ACID transactions for atomic operations across multiple collections:

public class OrderService
{
 private readonly ILiteDatabase _db;
 private readonly ILiteCollection<Order> _orders;
 private readonly ILiteCollection<Inventory> _inventory;

 public OrderService(ILiteDatabase db)
 {
 _db = db;
 _orders = db.GetCollection<Order>();
 _inventory = db.GetCollection<Inventory>();
 }

 public void PlaceOrder(Order order)
 {
 // Begin transaction
 _db.BeginTrans();

 try
 {
 // Insert order
 _orders.Insert(order);

 // Update inventory
 foreach (var item in order.Items)
 {
 var inventoryItem = _inventory.FindById(item.ProductId);
 if (inventoryItem.Quantity < item.Quantity)
 throw new InvalidOperationException("Insufficient inventory");

 inventoryItem.Quantity -= item.Quantity;
 _inventory.Update(inventoryItem);
 }

 // Commit transaction
 _db.Commit();
 }
 catch
 {
 // Rollback on error
 _db.Rollback();
 throw;
 }
 }
}

Indexes for Performance

Create indexes on frequently-queried properties:

public class TemperatureReadingRepository : BaseRepository<TemperatureReading>
{
 public TemperatureReadingRepository(ILiteDatabase db) : base(db)
 {
 // Simple index on single property
 Collection.EnsureIndex(x => x.SensorId);
 Collection.EnsureIndex(x => x.Timestamp);

 // Unique index (enforce uniqueness)
 Collection.EnsureIndex(x => x.SensorId, unique: true);

 // Compound expression (index on calculated value)
 Collection.EnsureIndex("TemperatureRounded", "ROUND($.Temperature, 0)");
 }
}

Index Best Practices:

• Index properties used in Where() clauses
• Index properties used in OrderBy() clauses
• Don't over-index (affects write performance)
• Use unique indexes to enforce data integrity
• Indexes are automatically maintained by LiteDB

File Storage (GridFS-style)

LiteDB supports file storage for large binary data:

public class FileStorageService
{
 private readonly ILiteDatabase _db;

 public FileStorageService(ILiteDatabase db)
 {
 _db = db;
 }

 // Store file
 public string StoreFile(string filename, Stream stream)
 {
 var fileInfo = _db.FileStorage.Upload(filename, filename, stream);
 return fileInfo.Id;
 }

 // Retrieve file
 public Stream RetrieveFile(string fileId)
 {
 var memoryStream = new MemoryStream();
 _db.FileStorage.Download(fileId, memoryStream);
 memoryStream.Position = 0;
 return memoryStream;
 }

 // Delete file
 public bool DeleteFile(string fileId)
 {
 return _db.FileStorage.Delete(fileId);
 }

 // List all files
 public IEnumerable<LiteFileInfo> ListFiles()
 {
 return _db.FileStorage.FindAll();
 }
}

File Storage Use Cases:

• Storing experimental images (PNG, JPEG)
• Archiving raw instrument output files
• Storing PDF reports
• Caching large binary data

Thread Safety

LiteDB provides thread-safe operations through connection-level locking:

// Singleton database instance (thread-safe)
services.AddSingleton<ILiteDatabase>(sp =>
 new LiteDatabase("myapp.db"));

// Concurrent reads and writes are automatically synchronized
// LiteDB uses a single-writer, multiple-reader lock strategy

Thread Safety Guarantees:

• ✅ Multiple threads can read simultaneously
• ✅ Single thread can write (others block)
• ✅ ACID transactions are atomic
• ⚠️ Long-running queries can block writers

Performance Tips for Multi-Threaded Scenarios:

// Use connection pooling for high concurrency (not built-in)
public class LiteDatabasePool
{
 private readonly ConcurrentBag<ILiteDatabase> _pool = new();
 private readonly string _connectionString;

 public LiteDatabasePool(string connectionString)
 {
 _connectionString = connectionString;
 }

 public ILiteDatabase GetDatabase()
 {
 if (_pool.TryTake(out var db))
 return db;

 return new LiteDatabase(_connectionString);
 }

 public void ReturnDatabase(ILiteDatabase db)
 {
 _pool.Add(db);
 }
}

Database Backup and Maintenance

public class DatabaseMaintenanceService
{
 private readonly ILiteDatabase _db;

 public DatabaseMaintenanceService(ILiteDatabase db)
 {
 _db = db;
 }

 // Backup database to file
 public void BackupDatabase(string backupPath)
 {
 using var backup = new LiteDatabase($"filename={backupPath}");

 foreach (var collectionName in _db.GetCollectionNames())
 {
 var sourceCollection = _db.GetCollection(collectionName);
 var targetCollection = backup.GetCollection(collectionName);

 foreach (var doc in sourceCollection.FindAll())
 {
 targetCollection.Insert(doc);
 }
 }
 }

 // Shrink database (reclaim space from deleted documents)
 public long ShrinkDatabase()
 {
 return _db.Rebuild();
 }

 // Check database integrity
 public bool CheckIntegrity()
 {
 try
 {
 return _db.CheckIntegrity();
 }
 catch
 {
 return false;
 }
 }

 // Export collection to JSON
 public string ExportToJson<T>(ILiteCollection<T> collection)
 {
 var documents = collection.FindAll();
 return JsonSerializer.Serialize(documents);
 }
}

Custom BsonMapper Configuration

Customize how objects are serialized to BSON:

var mapper = BsonMapper.Global;

// Map enum to string instead of int
mapper.EnumAsInteger = false;

// Custom serialization for specific type
mapper.RegisterType<MyCustomType>(
 serialize: (obj) => new BsonDocument { ["Value"] = obj.ToString() },
 deserialize: (bson) => MyCustomType.Parse(bson["Value"].AsString)
);

// Ignore specific properties globally
mapper.Entity<TemperatureReading>()
 .Ignore(x => x.InternalCache);

// Custom field mapping
mapper.Entity<TemperatureReading>()
 .DbRef(x => x.Sensor, "sensors"); // Foreign key reference

var db = new LiteDatabase("myapp.db", mapper);

API Reference Summary

IBaseRepository<T>

Standard CRUD interface for all repositories.

public interface IBaseRepository<T>
{
 /// Creates a new document and returns the generated ID
 T Create(T data, out object id);

 /// Creates a new document without returning the ID
 T Create(T data);

 /// Retrieves all documents in the collection
 IEnumerable<T> All();

 /// Finds a document by its unique identifier
 T FindById(object id);

 /// Updates an existing document or inserts if not found (upsert)
 void Update(T entity);

 /// Deletes a document by ID, returns true if deleted
 bool Delete(object id);
}

BaseRepository<T>

Abstract base class providing LiteDB implementation.

public abstract class BaseRepository<T> : IBaseRepository<T>
{
 /// Access to the underlying LiteDB database
 public ILiteDatabase DB { get; }

 /// Direct access to the collection for advanced queries
 public ILiteCollection<T> Collection { get; }

 /// Constructor requiring ILiteDatabase instance
 protected BaseRepository(ILiteDatabase db);

 // All IBaseRepository<T> methods implemented as virtual
 // (can be overridden in derived classes)
}

DocumentDbServiceCollectionExtensions

Dependency injection extension methods.

public static class DocumentDbServiceCollectionExtensions
{
 /// Registers ILiteDatabase as singleton
 /// path: File path or connection string
 /// db: Output parameter receiving the created database instance
 void RegisterDocumentDatabase(
 this IServiceCollection services,
 string path,
 out ILiteDatabase db);

 /// Registers ILiteCollection<T> as singleton for specific entity type
 void RegisterDocumentDatabaseTypes<T>(
 this IServiceCollection services,
 ILiteDatabase db);
}

Best Practices

Entity Design Recommendations

1. Always Use [BsonId] Attribute

public class MyEntity
{
 [BsonId] // Explicitly mark the ID property
 public ObjectId Id { get; set; }
}

2. Use Meaningful Property Names

// Good
public class TemperatureReading
{
 public DateTime MeasurementTime { get; set; }
 public double TemperatureCelsius { get; set; }
}

// Avoid
public class TemperatureReading
{
 public DateTime dt { get; set; } // Unclear abbreviation
 public double temp { get; set; } // Missing unit
}

3. Avoid Deep Nesting (>3 levels)

// Acceptable
public class Experiment
{
 public ExperimentSettings Settings { get; set; } // Level 1
}

public class ExperimentSettings
{
 public TemperatureSettings Temperature { get; set; } // Level 2
}

public class TemperatureSettings
{
 public double Target { get; set; } // Level 3
}

// Too deep - consider flattening or splitting into separate collections
public class Experiment
{
 public Level1 { Level2 { Level3 { Level4 { Level5 } } } } // Hard to query
}

4. Use [BsonIgnore] for Computed Properties

public class TemperatureReading
{
 public double TemperatureCelsius { get; set; }

 [BsonIgnore] // Don't persist this
 public double TemperatureFahrenheit => TemperatureCelsius * 9/5 + 32;
}

Identifier Strategy

Choose the right identifier type for your use case:

// Use ObjectId for general purpose (recommended)
[BsonId]
public ObjectId Id { get; set; }

// Use Guid for integration with existing .NET systems
[BsonId]
public Guid Id { get; set; } = Guid.NewGuid();

// Use auto-increment int for simple sequential IDs
[BsonId(autoId: true)]
public int Id { get; set; }

// Use string for business identifiers
[BsonId]
public string Id { get; set; } = $"EXP-{DateTime.Now:yyyyMMdd}-{Guid.NewGuid():N}";

Performance Tips

1. Create Indexes on Frequently-Queried Properties

Collection.EnsureIndex(x => x.Timestamp);
Collection.EnsureIndex(x => x.SensorId);

2. Use Pagination for Large Result Sets

// Bad - loads all documents into memory
var allReadings = Collection.FindAll().ToList();

// Good - loads only needed page
var page = Collection.Query()
 .OrderByDescending(x => x.Timestamp)
 .Skip(page * pageSize)
 .Limit(pageSize)
 .ToList();

3. Use Projection to Load Only Required Fields

// Bad - loads entire document
var readings = Collection.FindAll();

// Good - loads only needed properties
var summaries = Collection.Query()
 .Select(x => new { x.Id, x.Timestamp, x.Temperature })
 .ToList();

4. Batch Operations When Possible

// Bad - multiple individual inserts
foreach (var reading in readings)
{
 Collection.Insert(reading);
}

// Good - single batch insert
Collection.InsertBulk(readings);

Connection String Best Practices

Store connection strings in configuration:

// appsettings.json
{
 "ConnectionStrings": {
 "AnIMLDocumentDatabase": "filename=Data/animl.db",
 "ConfigDatabase": "filename=Data/config.db;password=secret"
 }
}

// In Program.cs
builder.Services.RegisterDocumentDatabase(
 builder.Configuration.GetConnectionString("AnIMLDocumentDatabase"),
 out var db);

Use encrypted databases for sensitive data:

"filename=sensitive_data.db;password=YourStrongPasswordHere"

Error Handling

Handle common LiteDB exceptions:

try
{
 var entity = repository.FindById(id);
}
catch (LiteException ex) when (ex.ErrorCode == LiteException.LOCK_TIMEOUT)
{
 // Database is locked by another thread
 _logger.LogWarning("Database lock timeout, retrying...");
 // Implement retry logic
}
catch (LiteException ex) when (ex.ErrorCode == LiteException.INDEX_DUPLICATE_KEY)
{
 // Unique constraint violation
 _logger.LogError("Duplicate key error: {Message}", ex.Message);
 throw new InvalidOperationException("Entity with this ID already exists", ex);
}
catch (LiteException ex)
{
 // General LiteDB error
 _logger.LogError(ex, "Database error");
 throw;
}

Repository Method Naming Conventions

Follow consistent naming patterns:

// Query methods - start with "Get" or "Find"
IEnumerable<T> GetReadingsBySensor(string sensorId);
T FindByCustomId(string customId);

// Check methods - start with "Has" or "Exists" or "Is"
bool HasReadingsInRange(DateTime start, DateTime end);
bool ExistsBySensorId(string sensorId);

// Count methods - start with "Count"
int CountReadingsToday();

// Delete methods - start with "Delete" or "Remove"
int DeleteOldReadings(DateTime cutoffDate);
bool RemoveByCustomId(string customId);

Comparison with SiLA2.Database.SQL

Aspect	SiLA2.Database.NoSQL	SiLA2.Database.SQL
Database Engine	LiteDB (embedded)	SQLite/SQL Server (embedded/server)
Data Model	Document-oriented (BSON)	Relational (tables)
Schema	Schema-less, flexible	Fixed schema, requires migrations
Query Language	LINQ-like API	Entity Framework LINQ or SQL
Setup Complexity	Low (single file)	Medium (EF Core + migrations)
Best For	AnIML data, configs, logs	User management, relational data
Performance	Good for documents	Excellent for joins/aggregations
File Size	Compact BSON	Larger (SQL overhead)
Transactions	ACID within single DB	ACID across multiple tables
Scalability	Up to ~100GB	Up to multi-TB (SQL Server)
Type Safety	Runtime (BSON serialization)	Compile-time (EF Core)
Deployment	Single DLL + data file	Multiple assemblies + migrations

When to Use Both:

Many SiLA2 servers use both modules for different purposes:

// SQL for user authentication (relational data)
builder.Services.AddDbContext<AuthenticationDbContext>(options =>
 options.UseSqlite("Data Source=users.db"));

// NoSQL for AnIML experimental data (document-oriented)
builder.Services.RegisterDocumentDatabase("Data/animl.db", out var docDb);
builder.Services.AddSingleton<IAnIMLRepository, AnIMLRepository>();

Related Packages

• SiLA2.Core - Core SiLA2 server implementation, domain models, network discovery
• SiLA2.AnIML - AnIML scientific data format support (uses this package for persistence)
• SiLA2.Database.SQL - SQL database module with Entity Framework Core
• SiLA2.AspNetCore - ASP.NET Core integration for SiLA2 servers
• LiteDB - Underlying embedded NoSQL database engine

Contributing & Development

This package is part of the sila_csharp project.

Building from Source

git clone --recurse-submodules https://gitlab.com/SiLA2/sila_csharp.git
cd sila_csharp/src
dotnet build SiLA2.Database.NoSQL/SiLA2.Database.NoSQL.csproj

Running Tests

# Run all tests
dotnet test Tests/SiLA2.Database.NoSQL.Tests/SiLA2.Database.NoSQL.Tests.csproj

# Run integration tests
dotnet test Tests/SiLA2.IntegrationTests.Server.Tests/SiLA2.IntegrationTests.Server.Tests.csproj --filter "FullyQualifiedName~NoSQL"

Project Structure

SiLA2.Database.NoSQL/
├── IBaseRepository.cs # Generic repository interface
├── BaseRepository.cs # LiteDB-backed implementation
├── DocumentDbServiceCollectionExtensions.cs # DI extension methods
├── SiLA2.Database.NoSQL.csproj # Project file
└── README.md # This file

Links & Resources

• SiLA2 Standard: https://sila-standard.com
• LiteDB Documentation: https://www.litedb.org
• Repository: https://gitlab.com/SiLA2/sila_csharp
• NuGet Package: https://www.nuget.org/packages/SiLA2.Database.NoSQL
• Documentation Wiki: https://gitlab.com/SiLA2/sila_csharp/-/wikis/home
• Issue Tracker: https://gitlab.com/SiLA2/sila_csharp/-/issues
• SiLA Slack: Join the community

License

This project is licensed under the MIT License.

Maintainer

Christoph Pohl (@Chamundi)

Security

For security vulnerabilities, please refer to the SiLA2 Vulnerability Policy.

---

Questions or Issues?

• Open an issue on GitLab
• Join the SiLA community on Slack
• Check the Wiki for additional documentation