👁 Image
MACRIM.Core 6.8.1

MACRIM.Caching Project

Purpose

The MACRIM.Caching project extends the MACRIM framework’s caching capabilities to support table-based caching in Redis, replacing the in-memory LookupService with a scalable, persistent solution. It caches IEntity instances for SQL tables (or other data sources), enabling fast lookups for batch processing (5-10K rows) with dynamic indexing and segmented caching. The rebuilt RedisService addresses compile errors from the original CacheService<T>, integrating with IConfigService for configuration-driven caching.

Phase 1: Table-Based Caching

Phase 1 implements caching in Redis, focusing on:

• Caching: Stores IEntity instances as Redis Hashes (segment:cacheName:segment:catalog) with TTL from expires.
• Indexing: Indexes jsondata via keypattern expressions (e.g., {catalog},{name},{catalog}:{name}) using Redis Sets (segment:cacheName:index:value).
• Loading: Retrieves data via IRepositoryBypass.SearchAsync, using LoadJSON for IEntity creation. Synchronized to ensure only one load per cache, with concurrent requests waiting.
• Upsert: Updates cache with IEntity instances (SQL sync deferred to Phase 2).
• Configuration: Uses IConfigService (_configs[segment]["commands"]["caching"][cacheName]) for table, profile, keypattern, expires, segment.
• Synchronization: Syncs on initial load, TTL expiration, and cache misses. No ongoing sync within TTL.
• Lookups: Supports lookups by any keypattern index entry (e.g., svc123) using GetRecord/GetRecordAsync, with expressions evaluated via ApplyLookups.

Usage Example

To perform a lookup like {?:all-services(svc123,(catalog))} (retrieve a service by svc123 and return its catalog):

1. Configure Cache:

   <command name="all-services" table="services" profile="sql" expires="600" segment="true" keypattern="{catalog},{name},{servicename},{catalog}:{name}">
    <methods>
    <method name="retrieve" type="MACRIM.Data.Extensions.Base.Search">
    <parameters>
    <parameter name="model" value="services" />
    <parameter name="profile" value="sql" />
    <parameter name="segment" value="{var:tenant}" />
    </parameters>
    </method>
    </methods>
   

2. Set Up DI:

   services.AddSingleton<ITableCachingService, RedisService>();
   

3. Update ApplyLookups (as static extension method):

   case "lookup":
   case "?":
    var cacheSvc = me.Context.Services.GetRequiredService<ITableCachingService>();
    if (cacheSvc == null) break;
    var argStart = what.IndexOf('(');
    var instruct = argStart > -1 ? what.Substring(++argStart) : string.Empty;
    instruct = instruct.Length > 0 ? instruct.Remove(instruct.Length - 1) : string.Empty;
    what = argStart > -1 ? what.Substring(0, argStart - 1) : what;
   
    if (what == ".")
    {
    var pstart = instruct.IndexOf(',');
    var lkey = instruct.Split(',').FirstOrDefault();
    var lexp = pstart > -1 ? instruct.Substring(++pstart) : string.Empty;
    if (lkey.StartsWith('(')) lkey = (string)me.ApplyLookups(lkey.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
    var el = me.LookupCache.ContainsKey(lkey) ? me.LookupCache[lkey] : me.Spawn();
    if (el != null && el.Found)
    {
    oFound = el.ApplyLookups(lexp.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
    }
    }
    else
    {
    var pstart = instruct.IndexOf(',');
    var lkey = instruct.Split(',').FirstOrDefault();
    var lexp = pstart > -1 ? instruct.Substring(++pstart) : string.Empty;
    if (string.IsNullOrEmpty(lkey) || string.IsNullOrEmpty(lexp))
    {
    oFound = string.Empty;
    break;
    }
   
    var keyvalue = me.Attributes.GetValueOrDefault(lkey, "");
    if (lkey.StartsWith('(')) keyvalue = (string)me.ApplyLookups(lkey.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
    if (string.IsNullOrEmpty(keyvalue))
    {
    oFound = string.Empty;
    break;
    }
   
    var record = cacheSvc.GetRecord(
    keyvalue: keyvalue,
    cacheName: what,
    segment: me.Context?.Segment ?? "default"
    );
   
    if (record != null && record.Found)
    {
    oFound = record.ApplyLookups(lexp.Replace('(', '{').Replace(')', '}'), returnType, subReturnType, command, defaults);
    }
    else
    {
    oFound = string.Empty;
    }
    }
    break;
   

4. Test Lookup:

   • Input: {?:all-services(svc123,(catalog))} with me.Attributes["servicename"] = "svc123".
   • Output: oFound = "cat1" (from record.ApplyLookups("{catalog}", ...)).
   • Complex Input: {?:all-services(svc123,(user:{id}:name:upper))} → oFound = "JOHN".

Phase 2: Queuing (Deferred)

Phase 2 will implement async SQL sync using Redis Lists, leveraging StartWatchingList, Polly retries, and a poison queue.

Key Components

• Interfaces (in MACRIM.Core.Abstractions):
  • ITableCachingService: LoadCacheAsync, GetRecordAsync, GetRecord, UpsertRecordAsync (version: a37df99a-233d-424d-9c2d-fd5b56875548).
  • IQueueService, QueueTask (Phase 2).
• RedisService (in MACRIM.Caching):
  • Implements ICachingService and ITableCachingService (version: b9f8c7b5-4d2e-4f9b-9e2c-6f7a8b9c0f4e).
  • Uses ConnectionHelper for Redis.
  • Resolves IRepositoryBypass via IContextService.Services.
• Configuration:
  • IEntity with attributes (table, profile, keypattern, expires, segment).

IEntity Mapping

• segment: Attributes["segment"], partition key.
• catalog: Attributes["catalog"], typically Key.
• jsondata: Attributes (e.g., {"segment": "seg1", "catalog": "cat1", "name": "John", "age": "30"}).
• Key: Attributes["catalog"].
• Element: Table name (e.g., contacts).
• Value: Null.
• Index: keypattern expressions (e.g., Index["cat1"], Index["cat1:John"]).

Implementation Status

• Phase 1 (Complete):
  • RedisService.cs rebuilt for caching, supporting lookups by any keypattern index entry using keyvalue (e.g., svc123).
  • Removed profile parameter, deriving it from configuration.
  • Fixed batch execution error (ITransaction.ExecuteAsync).
  • Added synchronous GetRecord for ApplyLookups, optimized for high-frequency calls without logging.
  • Synchronized LoadCacheAsync to ensure one load per cache, with concurrent requests waiting.
• Phase 2 (Deferred): Queuing pending prioritization.

Next Steps

1. Test Phase 1:

   • Test RedisService with sample data (e.g., contacts, keypattern="{catalog},{name},{ssn},{catalog}:{name}").
   • Verify concurrent lookups (e.g., 20 queued processes) to ensure single cache load.
   • Validate {?:cacheName(keyvalue,(expression))} in ApplyLookups (e.g., {?:all-services(svc123,(catalog))}).

2. Sync Enhancements:

   • Decide if sync_version checks are needed for Phase 1 to detect table changes within TTL.
   • Plan Phase 2 for cache-to-SQL sync.

3. DI Confirmation:

   • Confirm Microsoft.Extensions.DependencyInjection and singleton lifetime.

4. T Handling:

   • Confirm BaseEntityConvertible for IRepository<T>, or specify models (e.g., Documents).

5. Configuration:

   • Provide example IEntity output for _configs[segment]["commands"]["caching"][cacheName].

6. Phase 2:

   • Define queuing requirements when ready.