👁 Image
UltraKV 1.0.4

🚀 UltraKV - Lightning-Fast Key-Value Storage Engine

<div align="center">

👁 MIT License
👁 .NET 8.0
👁 C#
👁 Performance

An ultra-high performance, enterprise-grade key-value storage solution designed for the .NET ecosystem

|

</div>

🎯 Minimal Code, Maximum Performance

<div align="center">

🚀 Less than 1,000 Lines of Code for a Complete Database Engine

Proving that simplicity and performance can coexist

</div>

One of the most remarkable aspects of UltraKV is achieving enterprise-grade database performance with an incredibly compact codebase. The entire UltraKV engine core is implemented in less than 1,000 lines of C# code, demonstrating the power of focused, efficient design.

💡 Philosophy: "Perfect is the Enemy of Good"

UltraKV proves that:

• Less code = Less bugs
• Simple design = Better performance
• Focused scope = Reliable execution
• Clear logic = Easy maintenance

"The best code is no code at all. The second best is code so simple and efficient that it feels like no code."

Installation

More Document / WIKI

dotnet add package UltraKV

using var engine = new UltraKVEngine<string, string>("test.db");
engine.Set("key1", "value1");
var value = engine.Get("key1");
engine.Remove("key1");
engine.Flush();

📋 Table of Contents

• 🌟 Project Overview
• 🔥 Key Features
• ⚡ Performance Benchmarks
• 🏗️ Architecture Design
• 🚀 Quick Start
• 📖 Detailed Usage Guide
• 🔧 Advanced Configuration
• 📊 Performance Testing
• 🔐 Security Features
• 🛠️ Best Practices
• 🤝 Contributing
• 📄 License

🌟 Project Overview

UltraKV is a modern, single file, high-performance key-value storage system.

🎯 Design Goals

• 🚀 Extreme Performance: Supporting millions of ops/sec for read/write operations per instance
• 🔒 Data Security: Enterprise-grade encryption, compression, and data integrity protection
• 🛡️ High Reliability: Atomic transactions, data persistence, and fault recovery
• ⚙️ Highly Configurable: Flexible configuration options for different application scenarios
• 📈 Scalable: Multi-engine management and horizontal scaling support

🔥 Key Features

🎯 UltraKV Engine Features

🏗️ File Storage Structure

UltraKV Storage Format

┌──────────────────────────────────────────────────┐
│ 1. Database Header (Fixed 64 bytes) │ ← File Start │
├──────────────────────────────────────────────────┤
│ 2. Value Data Area │
│ ├─ Data Record 1 (Variable length) │
│ ├─ Data Record 2 (Variable length) │
│ └─ ... │
├──────────────────────────────────────────────────┤
│ 3. Index Data Area [Start, End] │ ← File End │
│ ├─ Index Entry 1 (Key + Position info) │
│ ├─ Index Entry 2 (Key + Position info) │
│ └─ ... │
└──────────────────────────────────────────────────┘

⚡ Performance Benchmarks

🏆 UltraKV Performance Benchmark

Test Environment: .NET 8.0, Windows 11, SSD Storage

🧠 In-Memory Mode Performance

Test Environment: .NET 8.0, Windows 11, Average of 10 rounds, Memory mode operations

Memory mode provides 3-8x performance improvement over file-based storage, ideal for cache and temporary data scenarios, Automatic timed persistence, and the memory mode can be turned off at any time

🚀 Quick Start

📦 Installation

<PackageReference Include="UltraKV" Version="1.0.0" />

🔧 Basic Usage

UltraKV Basic Example

using UltraKV;

// Create engine manager
using var manager = new UltraKVManager<string, string>("./data");

// Get engine instance
var engine = manager.GetEngine("my_database");

// Basic operations
engine.Set("user:1001", "John Doe");
engine.Set("user:1002", "Jane Smith");

// Read data
var user = engine.Get("user:1001"); // Returns: "John Doe"
var exists = engine.ContainsKey("user:1001"); // Returns: true

// Delete data
engine.Remove("user:1002");

// Batch operations
var batch = new Dictionary<string, string>
{
 ["product:1"] = "Laptop",
 ["product:2"] = "Mouse",
 ["product:3"] = "Keyboard"
};
engine.SetBatch(batch);

// Persist data
engine.Flush();

Advanced Configuration Example

// Create high-performance configuration
var config = new UltraKVConfig
{
 // Compression configuration
 CompressionType = CompressionType.LZ4,
 
 // Encryption configuration
 EncryptionType = EncryptionType.AES256GCM,
 EncryptionKey = "MySecureKey32BytesLong!@#$%^&*()",
 
 // Performance configuration
 FileStreamBufferSizeKB = 1024, // 1MB buffer
 WriteBufferSizeKB = 2048, // 2MB write buffer
 FlushInterval = 10, // 10-second auto-flush
 
 // Maintenance configuration
 AutoCompactEnabled = true, // Enable auto-compaction
 AutoCompactThreshold = 30, // 30% fragmentation triggers compaction
 
 // File update mode
 FileUpdateMode = FileUpdateMode.Append // Append mode for higher performance
};

var engine = manager.GetEngine("high_performance_db", config);

📖 Detailed Usage Guide

🔧 Configuration Options Explained

UltraKVConfig Core Configuration

public class UltraKVConfig
{
 // 🎯 Performance Related
 public bool EnableMemoryMode { get; set; } = false; // Memory mode
 public int FileStreamBufferSizeKB { get; set; } = 64; // File buffer
 public bool EnableWriteBuffer { get; set; } = true; // Write buffer
 public int WriteBufferSizeKB { get; set; } = 1024; // Buffer size
 
 // 🔐 Security Related
 public CompressionType CompressionType { get; set; } // Compression algorithm
 public EncryptionType EncryptionType { get; set; } // Encryption algorithm
 public HashType HashType { get; set; } = HashType.XXH3; // Hash algorithm
 public string? EncryptionKey { get; set; } // Encryption key
 
 // 🔄 Maintenance Related
 public bool AutoCompactEnabled { get; set; } = false; // Auto-compaction
 public byte AutoCompactThreshold { get; set; } = 50; // Compaction threshold
 public ushort FlushInterval { get; set; } = 5; // Flush interval
 public FileUpdateMode FileUpdateMode { get; set; } // Update mode
 
 // 🛡️ Validation Related
 public bool EnableUpdateValidation { get; set; } = false; // Update validation
 public int MaxKeyLength { get; set; } = 4096; // Maximum key length
}

🔄 Lifecycle Management

// Engine manager supports multiple engines
using var manager = new UltraKVManager<string, object>("./databases");

// Create engines for different purposes
var userEngine = manager.GetEngine("users", UltraKVConfig.Default);
var sessionEngine = manager.GetEngine("sessions", UltraKVConfig.Minimal);
var cacheEngine = manager.GetEngine("cache", new UltraKVConfig 
{ 
 FlushInterval = 30, // Cache data can flush less frequently
 EnableMemoryMode = true // Enable memory mode for cache
});

// Batch flush
manager.FlushAll();

// Close specific engine
manager.CloseEngine("cache");

// Get engine list
var engineNames = manager.GetEngineNames();

📊 Performance Monitoring

// Get engine statistics
var stats = engine.GetStats();
Console.WriteLine($"Record Count: {stats.RecordCount}");
Console.WriteLine($"Deleted Count: {stats.DeletedCount}");
Console.WriteLine($"File Size: {stats.FileSize / 1024.0 / 1024.0:F2} MB");
Console.WriteLine($"Deletion Ratio: {stats.DeletionRatio:P1}");
Console.WriteLine($"Compaction Recommended: {stats.ShrinkRecommended}");

// Manually trigger compaction
if (engine.ShouldShrink())
{
 engine.Compact(fullRebuild: false);
}

🔧 Advanced Configuration

🚀 Performance Optimization Configuration

High Throughput Write Scenarios

var highThroughputConfig = new UltraKVConfig
{
 FileUpdateMode = FileUpdateMode.Append, // Append mode for best performance
 WriteBufferSizeKB = 4096, // 4MB large buffer
 FileStreamBufferSizeKB = 2048, // 2MB file buffer
 FlushInterval = 30, // Longer flush interval
 AutoCompactThreshold = 70 // Higher compaction threshold
};

Low Latency Read Scenarios

var lowLatencyConfig = new UltraKVConfig
{
 EnableMemoryMode = true, // Memory mode for lowest latency
 FlushInterval = 5, // Frequent flushing for data safety
 EnableUpdateValidation = true // Enable validation for data correctness
};

Storage Space Sensitive Scenarios

var compactConfig = new UltraKVConfig
{
 CompressionType = CompressionType.Zstd, // Best compression ratio
 AutoCompactEnabled = true, // Enable auto-compaction
 AutoCompactThreshold = 20, // Low threshold triggers compaction
 FileUpdateMode = FileUpdateMode.Replace // Replace mode reduces fragmentation
};

🔐 Security Configuration Examples

Enterprise-Grade Security Configuration

var secureConfig = UltraKVConfig.Secure("MyEnterprise256BitSecretKey!@#");
// Equivalent to:
var secureConfig = new UltraKVConfig
{
 CompressionType = CompressionType.Gzip,
 EncryptionType = EncryptionType.AES256GCM,
 EncryptionKey = "MyEnterprise256BitSecretKey!@#",
 HashType = HashType.SHA256,
 EnableUpdateValidation = true
};

Debug and Development Configuration

var debugConfig = UltraKVConfig.Debug; // Enable all validation options

🔄 Compression Algorithm Selection Guide

🔐 Encryption Algorithm Selection Guide

📊 Performance Testing

🧪 Built-in Performance Tests

The project includes a complete performance testing suite. You can run the following tests:

# Clone the project
git clone https://github.com/trueai-org/UltraKV.git
cd UltraKV

# Run UltraKV performance tests
dotnet run --project src/UltraKV --configuration Release

# Run comparison tests
dotnet test src/UltraKV.Tests --configuration Release

📈 Custom Performance Testing

// Performance testing example
public async Task BenchmarkWritePerformance()
{
 using var manager = new UltraKVManager<string, string>("./benchmark");
 var engine = manager.GetEngine("test");
 
 const int iterations = 100_000;
 var stopwatch = Stopwatch.StartNew();
 
 for (int i = 0; i < iterations; i++)
 {
 engine.Set($"key_{i}", $"value_{i}");
 
 if (i % 1000 == 0) // Flush every 1000 operations
 {
 engine.Flush();
 }
 }
 
 engine.Flush();
 stopwatch.Stop();
 
 var opsPerSecond = iterations * 1000.0 / stopwatch.ElapsedMilliseconds;
 Console.WriteLine($"Write Performance: {opsPerSecond:N0} ops/sec");
}

🔄 Compaction Performance Testing

// Test compaction operation performance
public void BenchmarkCompactPerformance()
{
 using var engine = new UltraKVEngine<string, string>("./compact_test.db");
 
 // Write large amount of data
 for (int i = 0; i < 50_000; i++)
 {
 engine.Set($"key_{i}", new string('x', 1024)); // 1KB data
 }
 
 // Delete 50% of data to create fragmentation
 for (int i = 0; i < 50_000; i += 2)
 {
 engine.Remove($"key_{i}");
 }
 
 var beforeSize = new FileInfo("./compact_test.db").Length;
 var beforeStats = engine.GetStats();
 
 var stopwatch = Stopwatch.StartNew();
 engine.Compact(fullRebuild: false);
 stopwatch.Stop();
 
 var afterSize = new FileInfo("./compact_test.db").Length;
 var afterStats = engine.GetStats();
 
 Console.WriteLine($"Compaction Time: {stopwatch.ElapsedMilliseconds}ms");
 Console.WriteLine($"File Size: {beforeSize / 1024 / 1024}MB -> {afterSize / 1024 / 1024}MB");
 Console.WriteLine($"Space Saved: {(1 - (double)afterSize / beforeSize):P1}");
}

🔐 Security Features

🔒 Data Encryption

UltraKV supports industry-standard encryption algorithms:

• AES256-GCM: Widely recognized enterprise-grade encryption standard
• ChaCha20-Poly1305: Modern high-performance encryption algorithm

// Enable encrypted storage
var encryptedEngine = manager.GetEngine("secure_data", new UltraKVConfig
{
 EncryptionType = EncryptionType.AES256GCM,
 EncryptionKey = "MySecure32ByteEncryptionKey12345"
});

// Data will be automatically encrypted when stored
encryptedEngine.Set("sensitive_data", "confidential_information");

📋 Data Integrity

Multiple data integrity protection mechanisms:

var validatedConfig = new UltraKVConfig
{
 EnableUpdateValidation = true, // Enable write validation
 HashType = HashType.SHA256, // Use SHA256 for data verification
};

Supported hash algorithms:

• XXH3: Ultra-fast hash, default choice
• SHA256: Cryptographic-grade hash
• BLAKE3: Modern high-performance hash
• XXH128: 128-bit hash with low collision rate

🛠️ Best Practices

💡 Performance Optimization Recommendations

1. Buffer Configuration

// Adjust buffer size based on memory availability
var config = new UltraKVConfig
{
 FileStreamBufferSizeKB = Environment.Is64BitProcess ? 1024 : 256,
 WriteBufferSizeKB = Environment.Is64BitProcess ? 4096 : 1024
};

2. Batch Operation Optimization

// Use batch operations to improve performance
var batch = new Dictionary<string, string>();
for (int i = 0; i < 10000; i++)
{
 batch[$"key_{i}"] = $"value_{i}";
}
engine.SetBatch(batch); // Much faster than individual Set operations

3. Reasonable Flush Strategy

// High-frequency write scenarios
var highWriteConfig = new UltraKVConfig
{
 FlushInterval = 30, // Flush every 30 seconds
 WriteBufferSizeKB = 8192 // 8MB buffer
};

// Low-latency scenarios
var lowLatencyConfig = new UltraKVConfig
{
 FlushInterval = 1, // Flush every 1 second
 EnableUpdateValidation = true // Enable validation
};

🔧 Maintenance and Monitoring

1. Regular Statistics Monitoring

// Regularly check engine status
var timer = new Timer(async _ =>
{
 var stats = engine.GetStats();
 if (stats.DeletionRatio > 0.3) // Deletion ratio exceeds 30%
 {
 Console.WriteLine("Compaction operation recommended");
 if (engine.ShouldShrink())
 {
 engine.Compact();
 }
 }
}, null, TimeSpan.Zero, TimeSpan.FromMinutes(5));

2. Graceful Shutdown Handling

// Ensure data safety when application shuts down
AppDomain.CurrentDomain.ProcessExit += (sender, e) =>
{
 manager.FlushAll(); // Flush all engines
 manager.Dispose(); // Release resources
};

🚨 Error Handling

try
{
 engine.Set("key", "value");
}
catch (InvalidOperationException ex) when (ex.Message.Contains("disposed"))
{
 // Engine has been disposed
 Console.WriteLine("Engine is closed, please reinitialize");
}
catch (ArgumentException ex) when (ex.Message.Contains("EncryptionKey"))
{
 // Encryption configuration error
 Console.WriteLine("Encryption key configuration error");
}
catch (IOException ex)
{
 // Disk IO error
 Console.WriteLine($"Disk operation failed: {ex.Message}");
}

🔄 Data Migration and Backup

📤 Data Export

// Export all data
public void ExportData(UltraKVEngine<string, string> engine, string backupFile)
{
 var allKeys = engine.GetAllKeys();
 var backup = new Dictionary<string, string>();
 
 foreach (var key in allKeys)
 {
 var value = engine.Get(key);
 if (value != null)
 {
 backup[key] = value;
 }
 }
 
 var json = JsonSerializer.Serialize(backup, new JsonSerializerOptions 
 { 
 WriteIndented = true 
 });
 File.WriteAllText(backupFile, json);
}

📥 Data Import

// Restore data from backup
public void ImportData(UltraKVEngine<string, string> engine, string backupFile)
{
 var json = File.ReadAllText(backupFile);
 var backup = JsonSerializer.Deserialize<Dictionary<string, string>>(json);
 
 if (backup != null)
 {
 engine.SetBatch(backup);
 engine.Flush();
 }
}

🌐 Integration with Other Technologies

🔄 ASP.NET Core Integration

// Startup.cs or Program.cs
services.AddSingleton<UltraKVManager<string, object>>(provider =>
 new UltraKVManager<string, object>("./app_data"));

services.AddSingleton<IMemoryCache>(provider =>
{
 var manager = provider.GetService<UltraKVManager<string, object>>();
 return new UltraKVCache(manager.GetEngine("cache"));
});

🔧 Custom Cache Implementation

public class UltraKVCache : IMemoryCache
{
 private readonly UltraKVEngine<string, object> _engine;
 
 public UltraKVCache(UltraKVEngine<string, object> engine)
 {
 _engine = engine;
 }
 
 public bool TryGetValue(object key, out object value)
 {
 value = _engine.Get(key.ToString()!);
 return value != null;
 }
 
 public ICacheEntry CreateEntry(object key)
 {
 return new UltraKVCacheEntry(key.ToString()!, _engine);
 }
 
 // ... other interface implementations
}

🤝 Contributing

We welcome all forms of contribution! Please read for detailed information.

🐛 Bug Reports

Before submitting an issue, please ensure:

1. Search existing issues
2. Provide detailed error information and reproduction steps
3. Include environment information (.NET version, operating system, etc.)

💡 Feature Requests

We welcome feature suggestions! Please describe in detail in the issue:

1. Use case for the feature
2. Expected behavior
3. Possible implementation approaches

🔧 Development Environment Setup

# 1. Fork and clone the project
https://github.com/trueai-org/ultra-kv.git
cd ultra-kv

# 2. Install dependencies
dotnet restore

# 3. Run tests
dotnet test

# 4. Build project
dotnet build --configuration Release

📚 References

This project references the following excellent open-source projects:

• Lightning.NET - LMDB .NET bindings
• Fast-Persistent-Dictionary - Persistent dictionary implementation
• Microsoft FASTER - High-performance key-value store
• DBreeze - .NET embedded database

📊 Performance Comparison

Performance data based on benchmark tests on the same hardware environment. Actual performance varies by environment

📈 Roadmap

🎯 Near-term Goals (v1.1)

• Distributed support and cluster mode
• More compression algorithm support
• Performance monitoring and metrics export
• Database repair tools

🚀 Mid-term Goals (v2.0)

• Support for complex queries and indexing
• Plugin-based storage backends
• Cloud-native support
• Graphical management interface

🌟 Long-term Goals (v3.0)

• Machine learning optimized performance tuning
• Automated operations and fault recovery
• Cross-platform mobile support

📱 Community and Support

• 💬 Discussions - Technical discussions and Q&A
• 📧 - Official announcements and updates
• 🐛 Issue Tracker - Bug reports and feature requests
• 📖 Wiki - Detailed documentation and tutorials

📄 License

This project is licensed under the .

<div align="center">

⭐ If this project helps you, please give us a Star! ⭐

🏠 Home • 📚 Documentation • 🐛 Report Issues • 💡 Feature Requests

Copyright © 2024 TrueAI.org. All rights reserved.

</div>