👁 Image
D4S.Indexer.Application 1.0.20

D4S.Indexer

Document indexing library for Azure AI Search: extracts text, generates vector embeddings, and uploads searchable chunks.

Quick start

var indexer = IndexerBuilder.Create("my-index")
 .WithAzureSearch(searchEndpoint, searchKey)
 .WithAzureOpenAI(aoaiEndpoint, aoaiKey, embeddingDeployment, embeddingDimensions)
 .WithLocalFiles("./documents")
 .WithFileMetadataFields()
 .Build();

var result = await indexer.IndexAsync();

See src/Rag/samples/ for working examples (local files, SharePoint, OCR, agentic retrieval).

Architecture

D4S.Indexer.Domain Entities, abstractions (interfaces)
D4S.Indexer.Application Orchestration (DocumentIndexerService, DocumentExtractor)
D4S.Indexer.Infrastructure Azure implementations, builder, processors, sources

Built-in sources: local filesystem, multi-site SharePoint (PnP Core). Built-in processors: PDF, DOCX, XLSX, PPTX, TXT/Markdown.

Indexing modes

• Full (default): all documents fetched from every source; documents missing from the source list are deleted from the index.
• Delta (.WithDeltaMode()): only changed/new/deleted documents are provided; deletion is driven by DocumentMetadata.DeletedDate (set it and pass null for GetContentAsync). No implicit cleanup.

Both modes compare LastModifiedDate against the index to skip unchanged documents.

Builder options

IndexerBuilder.Create("index-name")
 // Required
 .WithAzureSearch(endpoint, apiKey)
 .WithAzureOpenAI(endpoint, apiKey, deployment, dimensions)
 // Sources (at least one)
 .WithLocalFiles("./docs") // or: opts => { opts.Path = …; opts.FileExtensions = […]; }
 .WithSharePointMultiSite(spOptions, contextFactory)
 .WithCustomDocumentSource<T>(serviceProvider, serviceKey)
 // Optional
 .WithDeltaMode()
 .WithFileMetadataFields()
 .WithChunkSize(maxSize: 1000, overlap: 200)
 .WithBatchSize(50)
 .WithKeywordExtraction(gptDeployment, maxKeywords: 10)
 .WithAzureDocumentIntelligence(endpoint, apiKey) // OCR
 .WithCustomDocumentProcessor<T>(serviceProvider, serviceKey)
 .ContinueOnError(true)
 .Filter(meta => meta.Extension == ".pdf")
 .ConfigureMetadata(meta => meta with { CustomFields = … })
 .AddCustomField("Status", CustomFieldType.String, filterable: true)
 .AddIndexFieldsFromAttributes<MyModel>()
 .OnProgress(p => Console.WriteLine(p.Phase))
 .WithLogging()
 .Build();