CrossCloudKit.File.AWS 2026.4.22.71

CrossCloudKit

👁 License: MIT
👁 .NET 8
👁 Tests

CrossCloudKit is a comprehensive .NET library that provides unified interfaces and implementations for working with multiple cloud services. It enables developers to write cloud-agnostic code that can seamlessly work across AWS, Google Cloud, MongoDB, Redis, and S3-compatible storage providers with consistent APIs and behavior.

Test Results

Last Updated: 2026-04-22 13:09:39 UTC

Metric	Count
✅ Tests Passed	2284
❌ Tests Failed	0
📊 Total Tests	2284

Features

• Unified Cloud Interfaces: Single interfaces for all cloud service categories
• Multi-Service Support:
  • Database Services: AWS DynamoDB, MongoDB, Google Cloud Datastore, Cross-Process Basic
  • File Storage Services: AWS S3, Google Cloud Storage, S3-Compatible providers, Cross-Process Basic
  • PubSub Messaging: AWS SNS/SQS Hybrid, Google Cloud Pub/Sub, Redis Pub/Sub, Cross-Process Basic
  • Memory/Caching: Redis with distributed locking and advanced data structures, Cross-Process Basic
  • LLM Services: OpenAI-compatible endpoints (Ollama, Groq, Azure, Bedrock…) with separate embedding model support + fully local CPU-only inference via LLamaSharp (bundled SmolLM2-135M + snowflake-arctic-embed-m-long embeddings)
  • Vector Database Services: Cross-process file-based brute-force search + Qdrant (gRPC)
  • LLM ↔ Vector Bridge: Built-in extension methods (EmbedAndUpsertAsync, EmbedAndUpsertBatchAsync, SemanticSearchAsync) for common embed-then-store and semantic-search workflows — single-line, no boilerplate
• Built-in Debug Panel: All Basic service providers automatically register with an embedded HTTP debug dashboard (localhost:57765) that shows live service status, operation logs via SSE, and interactive data browsing — zero configuration, auto-starts on first service instantiation, auto-shuts down when all services are disposed
• Type-Safe Operations: Strongly-typed primitive operations with Primitive system
• Modern Async/Await: Full asynchronous API with cancellation token support
• Advanced Features:
  • Database querying with rich condition system and atomic operations
  • Nested conditioning support for complex queries (Like: user.config.status)
  • Automatic backup and restore with scheduled backups and manual backups, all atomically
  • Atomic database migration between different providers
  • File operations with signed URLs, metadata, notifications, and streaming
  • Message queuing with topic management, subscriptions, and error handling
  • Distributed memory operations with mutex locking and data structures
  • Pub/Sub integration with file services for file event notifications
  • ASP.NET Core Integration: IDistributedCache adapter (MemoryServiceDistributedCache) and IFileProvider (FileServiceFileProvider) bridge for seamless framework integration
• Cloud-Agnostic Design: Write once, deploy anywhere across cloud providers
• Comprehensive Testing: Extensive integration test suites for all services

📦 Packages

Package	Description
CrossCloudKit.Interfaces	Core interfaces, base classes, and ASP.NET Core integration
Database Services
CrossCloudKit.Database.AWS	AWS DynamoDB implementation
CrossCloudKit.Database.Mongo	MongoDB implementation
CrossCloudKit.Database.GC	Google Cloud Datastore implementation
CrossCloudKit.Database.Basic	Cross-process file-based database implementation
File Storage Services
CrossCloudKit.File.AWS	AWS S3 file storage implementation
CrossCloudKit.File.GC	Google Cloud Storage implementation
CrossCloudKit.File.S3Compatible	S3-compatible storage providers
CrossCloudKit.File.Basic	Cross-process file-based storage implementation
PubSub Messaging Services
CrossCloudKit.PubSub.AWS	AWS SNS/SQS Hybrid implementation
CrossCloudKit.PubSub.GC	Google Cloud Pub/Sub implementation
CrossCloudKit.PubSub.Redis	Redis Pub/Sub implementation
CrossCloudKit.PubSub.Basic	Cross-process file-based Pub/Sub implementation
Memory/Caching Services
CrossCloudKit.Memory.Redis	Redis memory and caching implementation
CrossCloudKit.Memory.Basic	Cross-process file-based memory implementation
LLM Services
CrossCloudKit.LLM.OpenAI	OpenAI-compatible LLM provider — works with OpenAI, Azure OpenAI, Ollama, Groq, Bedrock, LM Studio, and any OpenAI-compatible endpoint
CrossCloudKit.LLM.Basic	⚡ CPU-only plug & play meta-package — installs both sub-packages below; completions + embeddings work out of the box
CrossCloudKit.LLM.Basic.Embeddings	Embeddings only — bundles snowflake-arctic-embed-m-long (Q8_0) via LLamaSharp CPU backend; works with zero configuration
CrossCloudKit.LLM.Basic.Completion	Completions only — bundles SmolLM2-135M-Instruct (Q8_0, ~139 MB, Apache-2.0) + LLamaSharp CPU backend; works with zero configuration
Vector Database Services
CrossCloudKit.Vector.Basic	Cross-process file-based vector store — zero external dependencies, ideal for development, testing, and lightweight workloads
CrossCloudKit.Vector.Qdrant	Qdrant vector database via official gRPC client
Debug Panel
CrossCloudKit.Basic.DebugPanel	Embedded HTTP debug dashboard for all Basic service providers — live service cards, operation log with SSE, interactive data browsing, automatic dead-process cleanup
Utilities
CrossCloudKit.Utilities.Common	Common utilities and primitive types
CrossCloudKit.Utilities.Windows	Windows-specific utilities

🛠️ Installation

# Choose your cloud provider package(s)
# Database Services
dotnet add package CrossCloudKit.Database.AWS
dotnet add package CrossCloudKit.Database.Mongo
dotnet add package CrossCloudKit.Database.GC
dotnet add package CrossCloudKit.Database.Basic

# File Storage Services
dotnet add package CrossCloudKit.File.AWS
dotnet add package CrossCloudKit.File.GC
dotnet add package CrossCloudKit.File.S3Compatible
dotnet add package CrossCloudKit.File.Basic

# PubSub Services
dotnet add package CrossCloudKit.PubSub.AWS
dotnet add package CrossCloudKit.PubSub.GC
dotnet add package CrossCloudKit.PubSub.Redis
dotnet add package CrossCloudKit.PubSub.Basic

# Memory Services
dotnet add package CrossCloudKit.Memory.Redis
dotnet add package CrossCloudKit.Memory.Basic

# LLM Services
dotnet add package CrossCloudKit.LLM.OpenAI # For any OpenAI-compatible endpoint
dotnet add package CrossCloudKit.LLM.Basic # CPU-only meta-package (embeddings + completions bundled)
dotnet add package CrossCloudKit.LLM.Basic.Embeddings # Embeddings only (bundles snowflake-arctic-embed-m-long GGUF)
dotnet add package CrossCloudKit.LLM.Basic.Completion # Completions only (bundles SmolLM2-135M GGUF)

# Vector Database Services
dotnet add package CrossCloudKit.Vector.Basic # File-based, no infra needed
dotnet add package CrossCloudKit.Vector.Qdrant # Qdrant gRPC client

# Core interfaces (automatically included as dependency)
dotnet add package CrossCloudKit.Interfaces

🏗️ Quick Start

LLM Services

OpenAI-Compatible Endpoint (Ollama, Groq, Azure…)

using CrossCloudKit.LLM.OpenAI;
using CrossCloudKit.Interfaces.Records;
using CrossCloudKit.Interfaces.Enums;

// Works with OpenAI, Ollama, Groq, Azure OpenAI, Bedrock, LM Studio, etc.
// embeddingModel is optional — use it when your completion and embedding models differ
// (the common case on self-hosted endpoints like Ollama).
await using var llmService = new LLMServiceOpenAI(
 baseUrl: "http://localhost:11434/v1", // Ollama
 apiKey: "",
 defaultModel: "qwen2.5:14b", // used for completions
 embeddingModel: "nomic-embed-text:v1.5" // used for CreateEmbeddingAsync / CreateEmbeddingsAsync
);
// If embeddingModel is omitted it falls back to defaultModel.

// Text completion
var response = await llmService.CompleteAsync(new LLMRequest
{
 Messages =
 [
 new LLMMessage { Role = LLMRole.System, Content = "You are a helpful assistant." },
 new LLMMessage { Role = LLMRole.User, Content = "Explain quantum computing briefly." }
 ],
 MaxTokens = 256
});
Console.WriteLine(response.Data.Content);

// Streaming completion
await foreach (var chunk in llmService.CompleteStreamingAsync(request))
{
 if (!chunk.Data.IsFinal)
 Console.Write(chunk.Data.ContentDelta);
}

// Embeddings
var embedding = await llmService.CreateEmbeddingAsync("Hello, world!");
Console.WriteLine($"Dimensions: {embedding.Data.Length}");

Local CPU-Only Inference (Plug & Play)

using CrossCloudKit.LLM.Basic;
using CrossCloudKit.Interfaces.Records;
using CrossCloudKit.Interfaces.Enums;

// Everything works out of the box — no downloads, no API keys, no configuration.
// • Embeddings: snowflake-arctic-embed-m-long (Q8_0) bundled via LLamaSharp
// • Completions: SmolLM2-135M-Instruct (Q8_0, ~139 MB) bundled in the NuGet package
// To use a different GGUF model, pass its path or set LLM_BASIC_MODEL_PATH / LLM_BASIC_EMBEDDING_MODEL_PATH.
await using var llmService = new LLMServiceBasic();

// Embeddings — always available
var vec = await llmService.CreateEmbeddingAsync("semantic search text");
// vec.Data is float[] (768-dim for snowflake-arctic-embed-m-long)

// Batch embeddings
var vecs = await llmService.CreateEmbeddingsAsync(["text a", "text b"]);

// Completions — available out of the box with the bundled SmolLM2-135M model
var reply = await llmService.CompleteAsync(new LLMRequest
{
 Messages = [new LLMMessage { Role = LLMRole.User, Content = "Say hello." }],
 MaxTokens = 64
});
Console.WriteLine(reply.Data.Content);

// Streaming completions
await foreach (var chunk in llmService.CompleteStreamingAsync(new LLMRequest
{
 Messages = [new LLMMessage { Role = LLMRole.User, Content = "Explain gravity briefly." }],
 MaxTokens = 128
}))
{
 if (!chunk.Data.IsFinal)
 Console.Write(chunk.Data.ContentDelta);
}

Embeddings Only

using CrossCloudKit.LLM.Basic.Embeddings;

// Bundles snowflake-arctic-embed-m-long (Q8_0); no completion model overhead.
await using var embedder = new LLMEmbeddingServiceBasic();
var vec = await embedder.CreateEmbeddingAsync("lightweight embedding");

Completions Only

using CrossCloudKit.LLM.Basic.Completion;

// Bundles SmolLM2-135M-Instruct; no embedding model overhead.
await using var completer = new LLMCompletionServiceBasic();
var reply = await completer.CompleteAsync(new LLMRequest
{
 Messages = [new LLMMessage { Role = LLMRole.User, Content = "Hello!" }],
 MaxTokens = 64
});
Console.WriteLine(reply.Data.Content);

Vector Database Services

Cross-process file-based (Zero Dependencies)

using CrossCloudKit.Vector.Basic;
using CrossCloudKit.Interfaces.Enums;
using CrossCloudKit.Interfaces.Records;
using CrossCloudKit.Utilities.Common;
using Newtonsoft.Json.Linq;

await using var vectorService = new VectorServiceBasic();

// Create collection
await vectorService.EnsureCollectionExistsAsync("products", vectorDimensions: 384, VectorDistanceMetric.Cosine);

// Upsert points
await vectorService.UpsertAsync("products", new VectorPoint
{
 Id = Guid.NewGuid().ToString(),
 Vector = new float[384], // typically from llmService.CreateEmbeddingAsync
 Metadata = new JObject { ["name"] = "Widget", ["price"] = 9.99, ["inStock"] = true }
});

// Similarity search with filter
var filter = vectorService.FieldEquals("inStock", new Primitive(true))
 .And(vectorService.FieldLessThan("price", new Primitive(20.0)));
var results = await vectorService.QueryAsync("products", queryVector, topK: 5, filter: filter);

foreach (var r in results.Data)
 Console.WriteLine($"{r.Id} score={r.Score:F4}");

Qdrant (Production Vector Database)

using CrossCloudKit.Vector.Qdrant;

// Uses the official Qdrant gRPC client (port 6334)
await using var vectorService = new VectorServiceQdrant(
 host: "localhost", // or qdrant.common-db.svc.cluster.local
 grpcPort: 6334
);

// Same unified IVectorService API as above
await vectorService.EnsureCollectionExistsAsync("embeddings", 384, VectorDistanceMetric.Cosine);
await vectorService.UpsertAsync("embeddings", point);
var hits = await vectorService.QueryAsync("embeddings", queryVec, topK: 10);

Combined: Semantic Store & Search via Bridge Extensions

using CrossCloudKit.Interfaces; // LLMVectorExtensions lives here
using CrossCloudKit.Interfaces.Records;
using Newtonsoft.Json.Linq;

// ── Single-item embed-then-upsert ──────────────────────────────────────────
var result = await vectorService.EmbedAndUpsertAsync(
 llmService: llmService,
 collectionName: "docs",
 id: "article-42",
 text: "cats are mammals",
 metadata: new JObject { ["source"] = "wiki" }
);

// ── Batch embed-then-upsert ────────────────────────────────────────────────
var items = new List<(string Id, string Text, JObject? Metadata)>
{
 ("doc-0", "cats are mammals", new JObject { ["text"] = "cats are mammals" }),
 ("doc-1", "dogs are loyal pets", new JObject { ["text"] = "dogs are loyal pets" }),
 ("doc-2", "C# is a statically typed language", new JObject { ["text"] = "C# is a statically typed language" })
};
await vectorService.EmbedAndUpsertBatchAsync(llmService, "docs", items);

// ── Semantic search ────────────────────────────────────────────────────────
var hits = await vectorService.SemanticSearchAsync(
 llmService: llmService,
 collectionName: "docs",
 queryText: "what animals are domestic pets?",
 topK: 2
);

foreach (var hit in hits.Data)
 Console.WriteLine($"[{hit.Score:F3}] {hit.Metadata?["text"]}");

Combined: Manual LLM Embeddings + Vector Search

// Embed documents
var texts = new[] { "cats are mammals", "dogs are loyal pets", "C# is a statically typed language" };
var embeddings = await llmService.CreateEmbeddingsAsync(texts);

// Store in vector DB
var points = texts.Zip(embeddings.Data, (t, e) => new VectorPoint
{
 Id = Guid.NewGuid().ToString(),
 Vector = e,
 Metadata = new JObject { ["text"] = t }
}).ToList();
await vectorService.UpsertBatchAsync("docs", points);

// Retrieve semantically similar documents
var queryVec = (await llmService.CreateEmbeddingAsync("what animals are domestic pets?")).Data;
var queryHits = await vectorService.QueryAsync("docs", queryVec, topK: 2, includeMetadata: true);

foreach (var hit in queryHits.Data)
 Console.WriteLine($"[{hit.Score:F3}] {hit.Metadata?["text"]}");

Database Services

AWS DynamoDB

using CrossCloudKit.Database.AWS;

// Initialize service
var dbService = new DatabaseServiceAWS(
 // Parameters here
);

// Create and store an item
var item = new JObject
{
 ["Name"] = "John Doe",
 ["Email"] = "john@example.com",
 ["Age"] = 30
};

var key = new DbKey("id", new Primitive("user-123"));
await dbService.PutItemAsync("Users", new DbKey("id", key, item);

// Retrieve the item
var result = await dbService.GetItemAsync("Users", key);
if (result.IsSuccessful && result.Data != null)
{
 Console.WriteLine($"User: {result.Data["Name"]}");
}

MongoDB

using CrossCloudKit.Database.Mongo;

// Initialize with connection string
var dbService = new DatabaseServiceMongoDB(
 // Parameters here
);

// Same unified API
var key = new DbKey("id", new Primitive("user-123"));

await dbService.PutItemAsync("Users", key, item);

Google Cloud Datastore

using CrossCloudKit.Database.GC;

// Initialize with service account
var dbService = new DatabaseServiceGC(
 // Parameters here
);

// Same unified API
var key = new DbKey("id", new Primitive("user-123"));

await dbService.PutItemAsync("Users", key, item);

File Storage Services

AWS S3

using CrossCloudKit.File.AWS;
using CrossCloudKit.Interfaces;

// Initialize service
var fileService = new FileServiceAWS(
 // Parameters here
);

// Upload a file
var content = StringOrStream.FromString("Hello, World!");
var uploadResult = await fileService.UploadFileAsync(
 content: content,
 bucketName: "my-bucket",
 keyInBucket: "files/hello.txt",
 accessibility: FileAccessibility.PublicRead
);

// Download a file
using var memoryStream = new MemoryStream();
var downloadResult = await fileService.DownloadFileAsync(
 bucketName: "my-bucket",
 keyInBucket: "files/hello.txt",
 destination: StringOrStream.FromStream(memoryStream)
);

// Create signed URLs
var signedUploadUrl = await fileService.CreateSignedUploadUrlAsync(
 bucketName: "my-bucket",
 keyInBucket: "uploads/new-file.txt",
 options: new SignedUploadUrlOptions
 {
 ValidFor = TimeSpan.FromHours(1),
 ContentType = "text/plain"
 }
);

Google Cloud Storage

using CrossCloudKit.File.GC;

// Initialize service
var fileService = new FileServiceGC(
 // Parameters here
);

// Same unified API as AWS S3
await fileService.UploadFileAsync(content, "my-bucket", "files/hello.txt");

S3-Compatible Storage

using CrossCloudKit.File.S3Compatible;

// Initialize for MinIO or other S3-compatible storage
var fileService = new FileServiceS3Compatible(
 // Parameters here
);

// Same API as AWS S3
await fileService.UploadFileAsync(content, "my-bucket", "files/hello.txt");

Memory Services

Redis Memory Service

using CrossCloudKit.Memory.Redis;
using CrossCloudKit.Interfaces;
using CrossCloudKit.Utilities.Common;

// Initialize service
var memoryService = new MemoryServiceRedis(
 // Parameters here
);

// Create a memory scope
var scope = new MemoryScopeLambda(() => "user:123");

// Set key-value pairs
await memoryService.SetKeyValuesAsync(scope, new[]
{
 new KeyValuePair<string, Primitive>("name", new Primitive("John Doe")),
 new KeyValuePair<string, Primitive>("age", new Primitive(30L))
});

// Get values
var name = await memoryService.GetKeyValueAsync(scope, "name");
Console.WriteLine($"Name: {name.Data?.AsString}");

// Atomic increment
var newAge = await memoryService.IncrementKeyByValueAndGetAsync(scope, "age", 1);
Console.WriteLine($"New age: {newAge.Data}");

// Distributed mutex locking
using var mutex = await MemoryScopeMutex.CreateScopeAsync(
 memoryService, scope, "user-lock", TimeSpan.FromMinutes(5)
);
await mutex.LockAsync();
// Critical section - only one process can execute this at a time

PubSub Services

AWS SNS/SQS Hybrid

using CrossCloudKit.PubSub.AWS;
using CrossCloudKit.Interfaces;

// Initialize service
var pubSubService = new PubSubServiceAWS(
 // Parameters here
);

// Ensure topic exists
await pubSubService.EnsureTopicExistsAsync("user-events");

// Subscribe to messages
await pubSubService.SubscribeAsync(
 topic: "user-events",
 onMessage: async (topic, message) =>
 {
 Console.WriteLine($"Received from {topic}: {message}");
 },
 onError: error =>
 {
 Console.WriteLine($"Error: {error.Message}");
 }
);

// Publish a message
await pubSubService.PublishAsync("user-events", "User logged in");

Google Cloud Pub/Sub

using CrossCloudKit.PubSub.GC;

// Initialize service
var pubSubService = new PubSubServiceGC(
 // Parameters here
);

// Same unified API as AWS
await pubSubService.EnsureTopicExistsAsync("user-events");
await pubSubService.PublishAsync("user-events", "User logged in");

Redis Pub/Sub

using CrossCloudKit.PubSub.Redis;

// Initialize service
var pubSubService = new PubSubServiceRedis(
 // Parameters here
);

// Same API for Redis pub/sub
await pubSubService.SubscribeAsync("user-events", async (topic, message) =>
{
 Console.WriteLine($"Redis message from {topic}: {message}");
});

🔧 Advanced Features

Database Operations

Advanced Conditioning System

CrossCloudKit provides a powerful, composable conditioning system that works consistently across all database providers. The system supports nested object structures, complex logical operations, and backwards compatibility with simple attribute names.

Basic Conditions

// Conditional update
var updateData = new JObject { ["LastLogin"] = DateTime.UtcNow };

var key = new DbKey("id", new Primitive("user-123"));

var result = await dbService.UpdateItemAsync(
 "Users", key, updateData,
 conditions:
 dbService.AttributeEquals("IsAdmin", new Primitive(true))
 .Or(dbService.AttributeEquals("Status", new Primitive("active"))
 .And(dbService.AttributeEquals("IsAdmin", new Primitive(false))))
);

// Check existence with conditions
var exists = await dbService.ItemExistsAsync("Users", "Id", keyValue, condition);

Existence Conditions

// Check if attribute exists
var condition = dbService.AttributeExists("Email");
var exists = await dbService.ItemExistsAsync(tableName, key, condition);

// Check if attribute does not exist
var notExistsCondition = dbService.AttributeNotExists("MiddleName");

Value Comparisons

// Equality checks
var equalsCondition = dbService.AttributeEquals("Status", new Primitive("active"));
var notEqualsCondition = dbService.AttributeNotEquals("Role", new Primitive("guest"));

// Numeric comparisons
var greaterCondition = dbService.AttributeIsGreaterThan("Age", new Primitive(18L));
var lessOrEqualCondition = dbService.AttributeIsLessOrEqual("Score", new Primitive(100.0));

Array Element Conditions

// Check if array contains element
var hasPermission = dbService.ArrayElementExists("Permissions", new Primitive("admin"));

// Check if array does not contain element
var noBlockedTag = dbService.ArrayElementNotExists("Tags", new Primitive("blocked"));

Nested Object Support

Access deeply nested properties using dot notation:

// Simple nested access
var nestedCondition = dbService.AttributeEquals("User.Email", new Primitive("john@example.com"));

// Deep nesting (multiple levels)
var deepCondition = dbService.AttributeIsGreaterThan("Account.Settings.Security.Level", new Primitive(5L));

// Nested arrays
var nestedArrayCondition = dbService.ArrayElementExists("Profile.Certifications", new Primitive("AWS"));

// Nested with all operations
await dbService.UpdateItemAsync(tableName, key, updateData,
 conditions: dbService.AttributeEquals("User.Status", new Primitive("verified"))
 .And(dbService.AttributeIsGreaterOrEqual("User.Account.Balance", new Primitive(100.0))));

Logical Operators

Combine conditions using And() and Or():

// AND logic - all conditions must be true
var adminCondition = dbService.AttributeEquals("Role", new Primitive("admin"))
 .And(dbService.AttributeEquals("Status", new Primitive("active")));

// OR logic - at least one condition must be true
var accessCondition = dbService.AttributeEquals("Role", new Primitive("admin"))
 .Or(dbService.AttributeEquals("Role", new Primitive("moderator")));

// Complex nested logic
var complexCondition = dbService.AttributeEquals("Department", new Primitive("IT"))
 .And(
 dbService.AttributeIsGreaterThan("Experience", new Primitive(5L))
 .Or(dbService.ArrayElementExists("Certifications", new Primitive("Senior")))
 );

Size function

// Check array size
var minItemsCondition = dbService.AttributeIsGreaterOrEqual("size(Items)", new Primitive(3L));

// Nested array size check
var nestedSizeCondition = dbService.AttributeEquals("size(Project.Team.Members)", new Primitive(5L));

Important Notes:

Array Element Access

Array indexing syntax (e.g., array[0]) is not supported. Instead, use the dedicated array element condition methods:

// ❌ INCORRECT - This will throw an ArgumentException
var wrongCondition = dbService.AttributeEquals("Tags[0]", new Primitive("admin"));
var wrongNestedCondition = dbService.AttributeEquals("User.Permissions[0]", new Primitive("read"));

// ✅ CORRECT - Use ArrayElementExists for checking array contents
var correctCondition = dbService.ArrayElementExists("Tags", new Primitive("admin"));
var correctNestedCondition = dbService.ArrayElementExists("User.Permissions", new Primitive("read"));

Why this design?

CrossCloudKit is designed to work consistently across all database providers (AWS DynamoDB, MongoDB, Google Cloud Datastore, and file-based storage). Array indexing semantics vary significantly across these platforms, and direct index access doesn't align with the distributed nature of NoSQL databases.

Atomic Increment

var key = new DbKey("id", new Primitive("user-123"));

// Atomically increment a counter
var newValue = await dbService.IncrementAttributeAsync(
 "Users", key, "LoginCount", incrementValue: 1
);

if (newValue.IsSuccessful)
{
 Console.WriteLine($"New login count: {newValue.Data}");
}

Scanning and Filtering

// Scan all items
var allUsers = await dbService.ScanTableAsync("Users");

// Scan with filter
var activeUsersFilter = dbService.AttributeEquals("Status", new Primitive("active"));
var activeUsers = await dbService.ScanTableWithFilterAsync("Users", activeUsersFilter);

// Paginated scan
var (items, nextToken, totalCount) = await dbService.ScanTablePaginatedAsync(
 "Users", pageSize: 10
);

Automatic Database Backup and Restore

• Automated Scheduled Backups: Configure backups using cron expressions
• Manual Backups: Create backups manually at any time
• Database migrations: Backups can be used to migrate databases between providers
• Cross-Provider Support: Works with all CrossCloudKit database services
• Point-in-Time Restoration: Restore from any backup file
• Cloud Storage Integration: Stores backups in any supported file service
• Distributed Operations: Uses mutex locking to prevent concurrent backup/restore operations
• Event-Driven Notifications: Publishes backup events via PubSub services
• Error Handling: Comprehensive error reporting and retry mechanism

Automatic Database Backup and Restore Example:

 using CrossCloudKit.Interfaces.Classes;

 // Initialize your services
 var databaseService = new DatabaseServiceAWS(/* parameters */);
 var fileService = new FileServiceAWS(/* parameters */);
 var pubSubService = new PubSubServiceAWS(/* parameters */);

 // Create backup service with daily backups at 1:00 AM UTC
 var backupService = new DatabaseServiceBackup(
 databaseService: databaseService,
 fileService: fileService,
 backupBucketName: "my-backup-bucket",
 pubsubService: pubSubService,
 cronExpression: "0 1 * * *", // Daily at 1:00 AM
 timeZoneInfo: TimeZoneInfo.Utc,
 backupRootPath: "database-backups/",
 errorMessageAction: ex => Console.WriteLine($"Backup error: {ex.Message}")
 );
 // Every hour: "0 * * * *"
 // Daily at 2:30 AM: "30 2 * * *"
 // Weekly on Sundays at midnight: "0 0 * * 0"
 // Monthly on the 1st at midnight: "0 0 1 * *"
 // Every 6 hours: "0 */6 * * *"

 // Get backup cursors
 var backupCursors = backupService.GetBackupFileCursorsAsync().ToListAsync();

 // Restore from a specific backup (e.g., most recent)
 var latestBackup = backupCursors.Last();
 var restoreResult = await backupService.RestoreBackupAsync(latestBackup);
 if (restoreResult.IsSuccessful)
 {
 Console.WriteLine("Database restored successfully!");
 }
 else
 {
 Console.WriteLine($"Restore failed: {restoreResult.ErrorMessage}");
 }

Manual Database Backup and Restore Example:

 using CrossCloudKit.Interfaces.Classes;

 // Initialize your services
 var databaseService = new DatabaseServiceAWS(/* parameters */);
 var fileService = new FileServiceAWS(/* parameters */);
 var pubSubService = new PubSubServiceAWS(/* parameters */);

 // Create backup service without automatic backups, for manual backups only
 var backupService = new DatabaseServiceBackup(
 databaseService: databaseService,
 fileService: fileService,
 backupBucketName: "my-backup-bucket",
 pubsubService: pubSubService,
 backupRootPath: "database-backups/",
 errorMessageAction: ex => Console.WriteLine($"Backup error: {ex.Message}")
 );

 //Take backup
 var result = await backupService.TakeBackupAsync();
 if (result.IsSuccessful)
 {
 Console.WriteLine("Database backup was successful!");
 }
 else
 {
 Console.WriteLine($"Backup failed: {result.ErrorMessage}");
 return;
 };

 //How to restore:
 var restoreResult = await backupService.RestoreBackupAsync(result.Data);
 if (restoreResult.IsSuccessful)
 {
 Console.WriteLine("Database restored successfully!");
 }
 else
 {
 Console.WriteLine($"Restore failed: {restoreResult.ErrorMessage}");
 return;
 }

Migrate from one database provider to another example:

 using CrossCloudKit.Interfaces.Classes;

 // Initialize your services
 var fileService = new FileServiceAWS(/* parameters */);
 var pubSubService = new PubSubServiceAWS(/* parameters */); //Needed for ensuring atomicity

 var sourceDatabaseService = new DatabaseServiceAWS(/* parameters */);
 var destinationDatabaseService = new DatabaseServiceGC(/* parameters */);

 var result = await DatabaseServiceMigration.MigrateAsync(
 sourceDatabaseService,
 destinationDatabaseService,
 fileService,
 pubSubService, //Needed for ensuring atomicity of this operation
 backupWorkBucketName: "my-tmp-bucket",
 cleanUpSourceDatabaseAfterMigrate: false,
 cleanUpDestinationDatabaseBeforeMigrate: false,
 errorMessageAction: ex => Console.WriteLine($"Migration error: {ex.Message}"));

 if (result.IsSuccessful)
 {
 Console.WriteLine("Database migration was successful!");
 }
 else
 {
 Console.WriteLine($"Migration failed: {result.ErrorMessage}");
 }

File Storage Operations

IFileProvider Integration

webAppBuilder.Services.AddSingleton<IFileProvider>(
 new FileServiceFileProvider(/**/);

File Operations with Metadata

// Get file metadata
var metadata = await fileService.GetFileMetadataAsync("my-bucket", "files/document.pdf");
if (metadata.IsSuccessful)
{
 Console.WriteLine($"Size: {metadata.Data.Size} bytes");
 Console.WriteLine($"Content-Type: {metadata.Data.ContentType}");
 Console.WriteLine($"Last Modified: {metadata.Data.LastModified}");
}

// Set file tags
var tags = new Dictionary<string, string>
{
 ["department"] = "engineering",
 ["classification"] = "internal"
};
await fileService.SetFileTagsAsync("my-bucket", "files/document.pdf", tags);

Bulk File Operations

// List files with pagination
var listResult = await fileService.ListFilesAsync("my-bucket", new ListFilesOptions
{
 Prefix = "uploads/",
 MaxResults = 100
});

foreach (var fileKey in listResult.Data.FileKeys)
{
 Console.WriteLine($"File: {fileKey}");
}

// Delete entire folder
var deletedCount = await fileService.DeleteFolderAsync("my-bucket", "temp/");
Console.WriteLine($"Deleted {deletedCount.Data} files");

Memory Operations

IDistributedCache Integration

webAppBuilder.Services.AddSingleton<IDistributedCache>(
 new MemoryServiceDistributedCache(
 new MemoryServiceRedis(/**/),
 new MemoryScopeLambda(() => "my-app-scope")
 );

List and Data Structure Operations

// Work with Redis lists
await memoryService.PushToListTailAsync(scope, "queue", new[]
{
 new Primitive("task1"),
 new Primitive("task2")
});

// Pop from list
var task = await memoryService.PopFirstElementOfListAsync(scope, "queue");
Console.WriteLine($"Processing: {task.Data?.AsString}");

// Check if list contains value
var contains = await memoryService.ListContainsAsync(scope, "queue", new Primitive("task2"));

Advanced Memory Operations

// Set expiration time
await memoryService.SetKeyExpireTimeAsync(scope, TimeSpan.FromHours(1));

// Conditional set (only if not exists)
var wasSet = await memoryService.SetKeyValueConditionallyAsync(
 scope, "initialized", new Primitive("true")
);

// Get all keys in scope
var keys = await memoryService.GetKeysAsync(scope);
Console.WriteLine($"Keys in scope: {string.Join(", ", keys.Data!)}");

PubSub Integration with File Services

File Event Notifications

// Set up file notifications using PubSub integration
var notificationId = await fileService.CreateNotificationAsync(
 bucketName: "my-bucket",
 topicName: "file-events",
 pathPrefix: "uploads/",
 eventTypes: new[] { FileNotificationEventType.ObjectCreated, FileNotificationEventType.ObjectDeleted },
 pubSubService: pubSubService
);

// Subscribe to file events
await pubSubService.SubscribeAsync("file-events", async (topic, message) =>
{
 Console.WriteLine($"File event: {message}");
 // Process file upload/deletion events
});

� Debug Panel

All Basic service providers (Database, File, Memory, Vector, PubSub) automatically register with an embedded HTTP debug dashboard — zero configuration required.

Features

• Live service cards showing registered instances, PIDs, machine names, and operation counts
• Real-time operation log via Server-Sent Events (SSE) — see every CRUD operation as it happens
• Interactive data browsing — click "Browse Data" to explore tables, buckets, collections, and scopes
• Cross-process support — multiple processes on the same machine share a single dashboard
• Automatic dead-process cleanup — stale entries from crashed processes are purged every 10 seconds and on each new registration
• Auto-lifecycle — the server starts on the first Basic service instantiation and shuts down when all services are disposed

Usage

// Just create any Basic service — the debug panel starts automatically
await using var memoryService = new MemoryServiceBasic();
await using var vectorService = new VectorServiceBasic();
var dbService = new DatabaseServiceBasic("myapp", memoryService);

// Open http://localhost:57765 in your browser

Configuration

Environment Variable	Default	Description
CROSSCLOUDKIT_DEBUG_PANEL_PORT	57765	Port for the dashboard
CROSSCLOUDKIT_DEBUG_PANEL_DISABLED	false	Set to true to disable entirely

Disabling in Tests

Add a [ModuleInitializer] in your test project to prevent test runs from spawning debug panels:

using System.Runtime.CompilerServices;

internal static class ModuleInit
{
 [ModuleInitializer]
 internal static void DisableDebugPanel()
 {
 Environment.SetEnvironmentVariable("CROSSCLOUDKIT_DEBUG_PANEL_DISABLED", "true");
 }
}

�📊 Supported Data Types

CrossCloudKit uses a unified Primitive system that seamlessly maps across all cloud providers:

// String values
var stringKey = new Primitive("hello-world");

// Numeric values
var integerKey = new Primitive(12345L);
var doubleKey = new Primitive(123.45);

// Binary data
var binaryKey = new Primitive(new byte[] { 1, 2, 3, 4 });

// Type-safe access
Console.WriteLine(stringKey.AsString);
Console.WriteLine(integerKey.AsInteger);
Console.WriteLine(doubleKey.AsDouble);
Console.WriteLine(Convert.ToBase64String(binaryKey.AsByteArray));

🧪 Testing

The library includes comprehensive integration tests for all providers:

# Run all tests
dotnet test

# Run tests for specific service type
dotnet test CrossCloudKit.Database.AWS.Tests
dotnet test CrossCloudKit.Database.Mongo.Tests
dotnet test CrossCloudKit.Database.GC.Tests
dotnet test CrossCloudKit.Database.Basic.Tests

dotnet test CrossCloudKit.File.AWS.Tests
dotnet test CrossCloudKit.File.GC.Tests
dotnet test CrossCloudKit.File.S3Compatible.Tests
dotnet test CrossCloudKit.File.Basic.Tests

dotnet test CrossCloudKit.PubSub.AWS.Tests
dotnet test CrossCloudKit.PubSub.GC.Tests
dotnet test CrossCloudKit.PubSub.Redis.Tests
dotnet test CrossCloudKit.PubSub.Basic.Tests

dotnet test CrossCloudKit.Memory.Redis.Tests
dotnet test CrossCloudKit.Memory.Basic.Tests

dotnet test CrossCloudKit.LLM.OpenAI.Tests
dotnet test CrossCloudKit.LLM.Basic.Tests

dotnet test CrossCloudKit.Vector.Basic.Tests
dotnet test CrossCloudKit.Vector.Qdrant.Tests

# Bridge / unit tests (no external services required)
dotnet test CrossCloudKit.Interfaces.Tests

Test Configuration

Tests support environment variables for real cloud service integration:

AWS Services (DynamoDB, S3, SNS/SQS Hybrid):

AWS_ACCESS_KEY=your-key
AWS_SECRET_KEY=your-secret
AWS_REGION=us-east-1

MongoDB:

MONGODB_CONNECTION_STRING=mongodb+srv://user:password@your-host/

Google Cloud Services (Datastore, Storage, Pub/Sub):

GOOGLE_CLOUD_PROJECT=your-project-id
GOOGLE_APPLICATION_CREDENTIALS_BASE64=your-base64-encoded-service-account
GOOGLE_CLOUD_TEST_BUCKET=test-bucket-name

Redis (Memory & PubSub):

REDIS_HOST=localhost
REDIS_PORT=6379
# Optional:
REDIS_USER=your-redis-user
REDIS_PASSWORD=your-redis-password
REDIS_ENABLE_SSL=true

LLM Services:

# OpenAI / any OpenAI-compatible endpoint (e.g. Ollama)
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY= # leave empty for Ollama
OPENAI_MODEL=qwen2.5:14b # completion model
OPENAI_EMBEDDING_MODEL=nomic-embed-text:v1.5 # embedding model (optional, falls back to MODEL)
# LLM.Basic — completions work out of the box with the bundled SmolLM2-135M model.
# Set this only to override with a different GGUF model:
LLM_BASIC_MODEL_PATH=/path/to/custom-model.gguf

Qdrant (Vector Database):

QDRANT_HOST=localhost
QDRANT_PORT=6334
# Optional:
QDRANT_API_KEY=your-api-key

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────────────────────┐
│ Application Layer │
├─────────────────────────────────────────────────────────────────────────────────────┤
│ CrossCloudKit.Interfaces (Unified API + Bridge Layer) │
│ IDatabaseService | IFileService | IPubSubService | IMemoryService │
│ ILLMService | IVectorService | LLMVectorExtensions (bridge) │
├──────────┬──────────┬──────────┬──────────┬──────────────┬──────────────────────────┤
│ Database │ File │ PubSub │ Memory │ LLM │ Vector DB │
│ Services │ Storage │ Services │ Services │ Services │ Services │
├──────────┼──────────┼──────────┼──────────┼──────────────┼──────────────────────────┤
│ AWS │ AWS S3 │ AWS │ Redis │ OpenAI-compat│ In-memory (Basic) │
│ DynamoDB │ Google │ SNS/SQS │ (Lists, │ (Ollama, │ (zero deps, unit tests) │
│ MongoDB │ Storage │ Google │ KV,Mutex)│ Groq, Azure, │ Qdrant (gRPC) │
│ Google │ S3-compat│ Pub/Sub │ Basic │ Bedrock, OAI)│ (production) │
│ Datastore│ Basic │ Redis │ (Cross- │ Local CPU │ │
│ Basic │ (local) │ Basic │ Process) │ (LLamaSharp +│ │
│ │ │ │ │ SmolLM2-135M)│ │
├──────────┴──────────┴──────────┴──────────┴──────────────┴──────────────────────────┤
│ CrossCloudKit.Basic.DebugPanel │
│ (Embedded HTTP dashboard, SSE, data browsing, dead-process cleanup) │
├─────────────────────────────────────────────────────────────────────────────────────┤
│ CrossCloudKit.Utilities.Common │
│ (Primitive, OperationResult, etc.) │
└─────────────────────────────────────────────────────────────────────────────────────┘

🔒 Service-Specific Considerations

Database Services

AWS DynamoDB

• Automatic table creation with string partition keys for maximum flexibility
• Expression attribute names used to handle reserved keywords
• Proper type mapping for conditions (string keys, numeric condition values)
• Support for both managed AWS service and local DynamoDB

MongoDB

• Native BSON support with automatic ObjectId handling
• Flexible schema with dynamic collection creation
• Base64 encoding for binary keys to ensure consistency
• Support for MongoDB connection strings and advanced configurations

Google Cloud Datastore

• Native entity and key support
• Automatic project and namespace handling
• Support for service account authentication and Application Default Credentials
• Efficient batch operations

File Storage Services

AWS S3

• Automatic bucket creation and configuration
• Support for multipart uploads for large files
• S3 event notifications integration with SNS/SQS
• Proper IAM role and policy management

Google Cloud Storage

• Native integration with Google Cloud IAM
• Support for signed URLs with custom expiration
• Automatic retry logic for transient errors
• Efficient streaming uploads and downloads
• Google Pub/Sub integration for file notifications

S3-Compatible Storage

• Generic S3 API compatibility layer
• Support for custom endpoints (MinIO, Wasabi, etc.)
• Path-style and virtual-hosted-style URL support
• Flexible authentication mechanisms
• Redis Pub/Sub integration for file notifications

PubSub Services

AWS SNS/SQS Hybrid

• Automatic topic and queue creation with proper permissions
• Dead letter queue support for failed messages
• Message filtering and fan-out patterns
• Integration with S3 bucket notifications
• Automatic retry logic and error handling

Google Cloud Pub/Sub

• Native push and pull subscription models
• Message ordering and delivery guarantees
• Automatic scaling and load balancing
• Integration with Cloud Storage notifications

Redis Pub/Sub

• High-performance in-memory messaging
• Pattern-based subscriptions
• Cluster support for high availability
• Integration with Redis memory operations
• Integration with all file service notifications (polling-based) (tested with MinIO)

Memory Services

Redis Memory Service

• Distributed mutex locking with automatic expiration
• Advanced data structures (lists, sets, sorted sets)
• Atomic operations and transactions
• Memory-efficient operations with streaming support
• Cluster and sentinel support for high availability

Basic Memory Service

• Cross-process file-based storage using JSON serialization
• Cross-process mutex locking with automatic expiration using OS-level named mutexes
• Thread-safe and process-safe operations for concurrent access
• Key-value storage, list operations, and distributed locking
• Automatic file cleanup and expiration handling
• No external dependencies required (perfect for development and single-machine deployments)

Basic Database Service

• File-based JSON document storage with cross-process synchronization
• Full CRUD operations with atomic updates and conditional operations
• Cross-process mutex locking using memory service for data consistency
• Support for complex queries, filtering, and pagination
• Automatic file organization with table-based directory structure
• Type-safe key handling for strings, numbers, and binary data
• No external dependencies beyond memory service (perfect for development and testing)

Basic File Service

• Local file system storage with metadata support
• Cross-process file operations with mutex synchronization
• Support for signed URLs via ASP.NET Core endpoint registration
• File upload/download with streaming and partial content support
• Metadata persistence with tags and custom properties support
• Automatic cleanup of expired signed URL tokens
• Optional web integration for HTTP-based file access

Signed URL Setup for FileServiceBasic

The Basic File Service supports HTTP-based signed URLs through ASP.NET Core endpoint registration. You can see FileServiceBasicIntegrationTest.cs for an example of how to set up the endpoint for signed urls to work with asp.net core.

PubSub Services

Basic Pub/Sub Service

• Cross-process message delivery using file-based storage and polling
• Cross-process subscription management with OS-level mutex synchronization
• Message persistence and delivery across multiple processes on the same machine
• Topic-based routing with cross-process visibility
• Automatic message cleanup and subscription management
• No external dependencies required (perfect for development and single-machine deployments)

🚀 Advanced Use Cases

Multi-Cloud Deployment

// Switch between providers seamlessly
IDatabaseService dbService = useCloud switch
{
 "aws" => new DatabaseServiceAWS(/*Parameters*/),
 "mongo" => new DatabaseServiceMongoDB(/*Parameters*/),
 "gcp" => new DatabaseServiceGC(/*Parameters*/),
 _ => new DatabaseServiceBasic(/*Parameters*/) // Local fallback
};

IFileService fileService = useCloud switch
{
 "aws" => new FileServiceAWS(/*Parameters*/),
 "gcp" => new FileServiceGC(/*Parameters*/),
 "s3" => new FileServiceS3Compatible(/*Parameters*/),
 _ => new FileServiceBasic(/*Parameters*/) // Local fallback
};

// Use Basic implementations for development or single-machine multi-process deployments
IMemoryService memoryService = useRedis
 ? new MemoryServiceRedis(/*Parameters*/) : new MemoryServiceBasic(/*Parameters*/);
IPubSubService pubSubService = useCloud
 ? new PubSubServiceAWS(/*Parameters*/) : new PubSubServiceBasic(/*Parameters*/);

Microservices Integration

// Service registry pattern
public class CloudServiceRegistry
{
 public IDatabaseService Database { get; }
 public IFileService FileStorage { get; }
 public IPubSubService Messaging { get; }
 public IMemoryService Cache { get; }
 public CloudServiceRegistry(IConfiguration config)
 {
 Database = CreateDatabaseService(config);
 FileStorage = CreateFileService(config);
 Messaging = CreatePubSubService(config);
 Cache = CreateMemoryService(config);
 }
}

Event-Driven Architecture

// Complete event-driven workflow
public class OrderProcessingService
{
 private readonly IDatabaseService _db;
 private readonly IFileService _files;
 private readonly IPubSubService _pubsub;
 private readonly IMemoryService _cache;
 public async Task ProcessOrderAsync(Order order) {
 // Store order in database
 await _db.PutItemAsync("orders", "id", new Primitive(order.Id), JObject.FromObject(order));

 // Generate receipt and store in file storage
 var receipt = GenerateReceipt(order); await _files.UploadFileAsync(receipt, "receipts", $"{order.Id}.pdf");

 // Cache order status
 var cacheScope = new MemoryScopeLambda(() => $"order:{order.Id}");

 await _cache.SetKeyValuesAsync(cacheScope, new[]
 {
 new KeyValuePair<string, Primitive>("status", new Primitive("processing"))
 });

 // Publish order event await
 _pubsub.PublishAsync("order-events", JsonConvert.SerializeObject(new { OrderId = order.Id, Status = "processing" })); }}

🤖 AI Code Assistance

CrossCloudKit ships with pre-built instruction files that teach AI coding assistants (GitHub Copilot, Cursor, Windsurf, and others) the correct API patterns, types, anti-patterns, and provider constructors. These files are automatically loaded by each IDE when you work inside the repository.

Included Files

IDE / Tool	File	Auto-loaded?
GitHub Copilot (VS Code, VS, JetBrains)	✅ Yes
Cursor	✅ Yes (alwaysApply: true)
Windsurf	✅ Yes (trigger: always_on)

Using as a NuGet Consumer

If you consume CrossCloudKit via NuGet (rather than cloning this repo), the AI instruction files aren't automatically present in your project. To enable AI assistance in your own codebase:

1. Copy the appropriate file from this repository into your project:

   # GitHub Copilot — place in your repo's .github/ folder
   mkdir -p .github
   curl -o .github/copilot-instructions.md \
    https://raw.githubusercontent.com/bkio/CrossCloudKit/main/.github/copilot-instructions.md
   
   # Cursor — place in your repo's .cursor/rules/ folder
   mkdir -p .cursor/rules
   curl -o .cursor/rules/crosscloudkit.mdc \
    https://raw.githubusercontent.com/bkio/CrossCloudKit/main/.cursor/rules/crosscloudkit.mdc
   
   # Windsurf — place in your repo's .windsurf/rules/ folder
   mkdir -p .windsurf/rules
   curl -o .windsurf/rules/crosscloudkit.md \
    https://raw.githubusercontent.com/bkio/CrossCloudKit/main/.windsurf/rules/crosscloudkit.md
   

2. Commit the file to your repository so the entire team benefits.

3. Other IDEs: Any AI assistant that supports markdown-based project rules can use the same content. Copy any of the files above and place it according to your IDE's convention:

   • JetBrains AI Assistant: .junie/guidelines.md
   • Cline: .clinerules
   • Aider: CONVENTIONS.md
   • Generic / multi-IDE: AGENTS.md (recognized by Cursor, Windsurf, and others)

Tip: The instruction files are identical in content — only the frontmatter/header differs per IDE. You only need one copy for each IDE you use.

📄 License

This project is licensed under the MIT License - see the file for details.

---

Made with ❤️ by the Burak Kara