Linger.FileSystem 1.4.2

Linger.FileSystem

A unified file system abstraction library providing a consistent interface for accessing different file systems, including local file system, FTP, and SFTP. With this library, you can use the same API to operate on different types of file systems, simplifying the development process and improving code reusability.

Project Structure

The Linger.FileSystem solution includes the following NuGet packages:

• Linger.FileSystem: Core library, providing unified interfaces and local file system implementation
• Linger.FileSystem.Ftp: FTP file system implementation, based on FluentFTP
• Linger.FileSystem.Sftp: SFTP file system implementation, based on SSH.NET

Installation

# Install core library
dotnet add package Linger.FileSystem

# Install FTP support
dotnet add package Linger.FileSystem.Ftp

# Install SFTP support
dotnet add package Linger.FileSystem.Sftp

Key Features

• Unified Interface: Consistent API through IFileSystemOperations interface to operate different types of file systems
• Multiple File System Support: Including local file system, FTP, and SFTP
• Asynchronous Operations: All operations support async methods, suitable for modern application development
• Automatic Retry: Built-in retry mechanism with configurable retry count and delay for improved operation reliability
• Connection Management: Automatic handling of remote file system connections and disconnections
• Multiple Naming Rules: Support for MD5, UUID, and normal naming rules
• Streaming Upload Optimization: Local file system uses IncrementalHash and ArrayPool<byte> for memory-efficient large file processing, supporting files of any size
• Batch Operation Progress: Real-time progress tracking via IProgress<BatchProgress> for batch upload, download, and delete operations
• Connection Pool Idle Timeout: Automatic cleanup of idle connections in the pool with configurable timeout via ConnectionPoolIdleTimeout
• Batch Operation Retry: Per-file retry support for batch operations with configurable BatchRetryOptions settings

Supported .NET Versions

• .NET Framework 4.6.2+
• .NET Standard 2.0+
• .NET 5+

Quick Start

Using Local File System

// Create local file system instance
var localFs = new LocalFileSystem("C:/Storage");

// Upload file
using var fileStream = File.OpenRead("test.txt");
var result = await localFs.UploadAsync(fileStream, "uploads/destination-file.txt", true);

// Check upload result
if (result.Success)
{
 Console.WriteLine($"File uploaded successfully: {result.FilePath}");
}

Concurrency and Batch (Local)

Local batch operations support configurable parallelism via LocalFileSystemOptions.MaxDegreeOfParallelism:

• 1: serial execution (default), lower resource usage
• >1: parallel execution (internally throttled), ideal for large batches

Example:

// Configure parallelism and use unified batch operations
var options = new LocalFileSystemOptions
{
 RootDirectoryPath = "C:/Storage",
 MaxDegreeOfParallelism = 4 // 1 = serial, >1 = parallel
};

var localFs = new LocalFileSystem(options);

// Batch upload into a directory under the root (copies into C:/Storage/uploads)
var uploadResult = await localFs.UploadFilesAsync(new[]
{
 "C:/in/a.txt",
 "C:/in/b.txt"
}, "uploads", overwrite: true);
Console.WriteLine($"Uploaded: {uploadResult.SucceededFiles.Count}, Failed: {uploadResult.FailedFiles.Count}");

// Batch download: copy from root-relative paths into a local directory
var downloadResult = await localFs.DownloadFilesAsync(new[]
{
 "uploads/a.txt",
 "uploads/b.txt"
}, "C:/downloads", overwrite: true);
Console.WriteLine($"Downloaded: {downloadResult.SucceededFiles.Count}, Failed: {downloadResult.FailedFiles.Count}");

// Batch delete: pass root-relative paths
var deleteResult = await localFs.DeleteFilesAsync(new[]
{
 "uploads/a.txt",
 "uploads/b.txt"
});
Console.WriteLine($"Deleted: {deleteResult.SucceededFiles.Count}, Failed: {deleteResult.FailedFiles.Count}");

These batch APIs return BatchOperationResult with SucceededFiles and FailedFiles (failed items include error message and exception).

Batch Operation Progress Reporting

Use IProgress<BatchProgress> to monitor the progress of batch operations:

// Create a progress handler
var progress = new Progress<BatchProgress>(p =>
{
 Console.WriteLine($"Progress: {p.Completed}/{p.Total} ({p.PercentComplete:F1}%)");
 Console.WriteLine($"Current file: {p.CurrentFile}");
 Console.WriteLine($"Succeeded: {p.Succeeded}, Failed: {p.Failed}");
});

// Batch upload with progress
var uploadResult = await fileSystem.UploadFilesAsync(files, "/uploads", overwrite: true, progress);

// Batch download with progress
var downloadResult = await fileSystem.DownloadFilesAsync(remoteFiles, "C:/Downloads", overwrite: true, progress);

// Batch delete with progress
var deleteResult = await fileSystem.DeleteFilesAsync(filesToDelete, progress);

BatchProgress structure contains:

• Completed: Number of files processed (reported after each file completes)
• Total: Total number of files
• CurrentFile: Path of the file that was just processed
• Succeeded: Number of successful operations
• Failed: Number of failed operations
• PercentComplete: Completion percentage (0-100)

Note: Progress is reported after each file operation completes, ensuring Completed always reflects the accurate count.

Using FTP File System

For FTP file system operations, install the FTP package:

Install-Package Linger.FileSystem.Ftp

Basic usage example:

using Linger.FileSystem.Ftp;

var ftpSetting = new RemoteSystemSetting
{
 Host = "ftp.example.com",
 Port = 21,
 UserName = "username",
 Password = "password",
 Type = "FTP"
};

var ftpFs = new FtpFileSystem(ftpSetting);
var result = await ftpFs.UploadAsync(fileStream, "/public_html/test.txt", true);

📖 For detailed FTP documentation and advanced features, see:

Using SFTP File System

For SFTP file system operations, install the SFTP package:

Install-Package Linger.FileSystem.Sftp

Basic usage example:

using Linger.FileSystem.Sftp;

var sftpSetting = new RemoteSystemSetting
{
 Host = "sftp.example.com",
 Port = 22,
 UserName = "username",
 Password = "password",
 Type = "SFTP"
};

var sftpFs = new SftpFileSystem(sftpSetting);
var result = await sftpFs.UploadAsync(fileStream, "/home/user/test.txt", true);

📖 For detailed SFTP documentation and advanced features, see:

Common Operations

File Upload

// Upload stream
using var stream = File.OpenRead("local-file.txt");
var result = await fileSystem.UploadAsync(stream, "uploads/destination-file.txt", true);

// Upload local file
result = await fileSystem.UploadFileAsync("local-file.txt", "uploads", true);

// Upload with cancellation token
using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(5));
result = await fileSystem.UploadAsync(
 stream, 
 "uploads/destination-file.txt", 
 true, 
 cts.Token);

File Download

// Download to stream
using var outputStream = new MemoryStream();
var result = await fileSystem.DownloadToStreamAsync("uploads/file.txt", outputStream);

// Download to local file
result = await fileSystem.DownloadFileAsync("uploads/file.txt", "C:/Downloads/downloaded-file.txt", true);

// Download with cancellation token
using var cts = new CancellationTokenSource();
result = await fileSystem.DownloadFileAsync(
 "uploads/large-file.zip", 
 "C:/Downloads/large-file.zip", 
 true, 
 cts.Token);

File Deletion

var result = await fileSystem.DeleteAsync("uploads/file-to-delete.txt");

Directory Operations

// Check if directory exists
bool exists = await fileSystem.DirectoryExistsAsync("uploads/images");

// Create directory
await fileSystem.CreateDirectoryIfNotExistsAsync("uploads/documents");

// Check if path is a directory
if (await fileSystem.IsDirectoryAsync("uploads/images"))
{
 Console.WriteLine("Path is a directory");
}

Stream Factory API

For efficient streaming operations without loading entire files into memory:

// Open file for reading (returns raw Stream)
await using var readStream = await fileSystem.OpenReadAsync("data/large-file.bin", cancellationToken);
await ProcessLargeFileAsync(readStream);

// Open file for writing
await using var writeStream = await fileSystem.OpenWriteAsync("output/result.bin", overwrite: true, cancellationToken);
await writeStream.WriteAsync(data, cancellationToken);

// Text file reading with StreamReader
using var reader = await fileSystem.GetReaderAsync("logs/app.log", Encoding.UTF8, cancellationToken);
while (await reader.ReadLineAsync() is { } line)
{
 ProcessLine(line);
}

// Text file writing with StreamWriter
await using var writer = await fileSystem.GetWriterAsync("output/report.csv", overwrite: true, Encoding.UTF8, cancellationToken);
await writer.WriteLineAsync("Name,Value");
await writer.WriteLineAsync("Item1,100");

Metadata Query API

// Get file size (returns null if file doesn't exist)
var fileSize = await fileSystem.GetFileSizeAsync("uploads/document.pdf", cancellationToken);
if (fileSize.HasValue)
{
 Console.WriteLine($"File size: {fileSize.Value} bytes");
}
else
{
 Console.WriteLine("File not found");
}

Configuration Options

Local File System Options

var options = new LocalFileSystemOptions
{
 RootDirectoryPath = "C:/Storage", // Root directory path
 DefaultNamingRule = NamingRule.Md5, // Default naming rule: Md5, Uuid, Normal
 DefaultOverwrite = false, // Whether to overwrite files with same name by default
 DefaultUseSequencedName = true, // Whether to use sequence naming on file name conflicts
 ValidationLevel = FileValidationLevel.Full, // Validation level: None, SizeOnly, Full
 CleanupOnValidationFailure = true, // Whether to cleanup files on validation failure
 UploadBufferSize = 81920, // Upload buffer size
 DownloadBufferSize = 81920, // Download buffer size
 RetryOptions = new RetryOptions 
 { 
 MaxRetryAttempts = 3, 
 DelayMilliseconds = 1000 
 }
};

var localFs = new LocalFileSystem(options);

Remote File System Settings

var remoteSetting = new RemoteSystemSetting
{
 Host = "example.com", // Host address
 Port = 21, // Port (FTP default 21, SFTP default 22)
 UserName = "username", // Username
 Password = "password", // Password
 Type = "FTP", // Type: "FTP" or "SFTP"
 ConnectionTimeout = 30000, // Connection timeout (milliseconds)
 OperationTimeout = 60000, // Operation timeout (milliseconds)
 MaxDegreeOfParallelism = 4, // Batch operation concurrency
 
 // Connection pool idle timeout (connections idle longer than this will be recreated)
 ConnectionPoolIdleTimeout = TimeSpan.FromMinutes(5),
 
 // Batch operation retry settings
 BatchRetryOptions = new RetryOptions
 {
 MaxRetryAttempts = 3,
 DelayMilliseconds = 1000
 },
 
 // SFTP specific settings
 CertificatePath = "", // Certificate path
 CertificatePassphrase = "" // Certificate passphrase
};

Advanced Features

Local File System Advanced Features

File Naming Rules

The local file system supports three file naming rules:

• Normal: Keep original file name
• Md5: Use MD5 hash of file content for naming, ideal for deduplication
• Uuid: Use UUID to generate unique file name, ideal for avoiding conflicts

Advanced Upload Example

// Use advanced upload functionality (with naming rule)
var uploadedInfo = await localFs.UploadAsync(
 stream,
 "source-file.txt", // Source file name
 "container1", // Container name
 "images", // Target path
 NamingRule.Md5, // Naming rule: Normal, Md5, Uuid
 false, // Whether to overwrite
 true // Whether to use sequence naming
);

// Access uploaded information
Console.WriteLine($"File hash: {uploadedInfo.HashData}");
Console.WriteLine($"Relative path: {uploadedInfo.FilePath}");
Console.WriteLine($"Full path: {uploadedInfo.FullFilePath}");

FTP and SFTP Advanced Features

For advanced FTP and SFTP features such as:

• Batch file operations
• Directory listing and manipulation
• Working directory management
• Certificate-based authentication (SFTP)
• Custom timeout configurations

Please refer to the dedicated documentation:

• 📖
• 📖

Connection Management

// Method 1: Use using statement for automatic connection management
using (var ftpFs = new FtpFileSystem(remoteSetting))
{
 // Operations automatically handle connection and disconnection
 await ftpFs.UploadFileAsync("local.txt", "/remote/path");
}

// Method 2: Manual connection management
try
{
 ftpFs.Connect();
 // Execute multiple operations...
 await ftpFs.UploadFileAsync("file1.txt", "/remote");
 await ftpFs.UploadFileAsync("file2.txt", "/remote");
}
finally
{
 ftpFs.Disconnect();
}

Cancellation Support

All file system operations support CancellationToken for graceful cancellation:

public class FileUploadService
{
 private readonly IFileSystemOperations _fileSystem;
 
 public FileUploadService(IFileSystemOperations fileSystem)
 {
 _fileSystem = fileSystem;
 }
 
 // Upload with timeout
 public async Task<FileOperationResult> UploadWithTimeoutAsync(
 Stream stream, 
 string destinationPath, 
 int timeoutSeconds = 300)
 {
 using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(timeoutSeconds));
 
 try
 {
 return await _fileSystem.UploadAsync(
 stream, 
 destinationPath, 
 overwrite: true, 
 cancellationToken: cts.Token);
 }
 catch (OperationCanceledException)
 {
 return FileOperationResult.Failure("Upload cancelled due to timeout");
 }
 }
 
 // Batch upload with cancellation
 public async Task<List<FileOperationResult>> UploadMultipleFilesAsync(
 Dictionary<Stream, string> files, 
 CancellationToken cancellationToken)
 {
 var results = new List<FileOperationResult>();
 
 foreach (var (stream, path) in files)
 {
 cancellationToken.ThrowIfCancellationRequested();
 
 var result = await _fileSystem.UploadAsync(
 stream, 
 path, 
 overwrite: true, 
 cancellationToken);
 results.Add(result);
 }
 
 return results;
 }
}

Using with ASP.NET Core

[ApiController]
[Route("api/[controller]")]
public class FileController : ControllerBase
{
 private readonly IFileSystemOperations _fileSystem;
 
 public FileController(IFileSystemOperations fileSystem)
 {
 _fileSystem = fileSystem;
 }
 
 [HttpPost("upload")]
 public async Task<IActionResult> UploadFile(
 IFormFile file, 
 CancellationToken cancellationToken)
 {
 try
 {
 using var stream = file.OpenReadStream();
 var result = await _fileSystem.UploadAsync(
 stream, 
 $"uploads/{file.FileName}", 
 overwrite: true, 
 cancellationToken);
 
 if (result.Success)
 {
 return Ok(new { path = result.FilePath });
 }
 
 return BadRequest(result.ErrorMessage);
 }
 catch (OperationCanceledException)
 {
 return StatusCode(499, "Upload cancelled by client");
 }
 }
}

Exception Handling

try
{
 var result = await ftpFs.UploadFileAsync("local.txt", "/remote");
 if (result.Success)
 {
 Console.WriteLine($"Upload successful: {result.FilePath}");
 }
 else
 {
 Console.WriteLine($"Upload failed: {result.ErrorMessage}");
 }
}
catch (OperationCanceledException)
{
 Console.WriteLine("Operation was cancelled");
}
catch (FileSystemException ex)
{
 Console.WriteLine($"File system operation exception: {ex.Message}");
 Console.WriteLine($"Operation: {ex.Operation}, Path: {ex.Path}");
 if (ex.InnerException != null)
 {
 Console.WriteLine($"Inner exception: {ex.InnerException.Message}");
 }
}

Retry Mechanism

var retryOptions = new RetryOptions
{
 MaxRetryAttempts = 5, // Maximum 5 retries
 DelayMilliseconds = 1000, // Initial delay 1 second
 MaxDelayMilliseconds = 30000, // Maximum delay 30 seconds
 UseExponentialBackoff = true // Use exponential backoff algorithm
};

// Configure retry options for remote file system
var ftpFs = new FtpFileSystem(remoteSetting, retryOptions);

Performance Optimization

Buffer Size Tuning

Default buffer size is 81920 bytes (80KB). For large file processing scenarios, you can increase the buffer size to improve throughput:

var options = new LocalFileSystemOptions
{
 RootDirectoryPath = "C:/Storage",
 UploadBufferSize = 262144, // 256KB - suitable for large file uploads
 DownloadBufferSize = 262144 // 256KB - suitable for large file downloads
};

💡 Larger buffers improve performance for large files but increase memory usage per concurrent operation.

Batch Operation Optimization

For scenarios requiring processing of large numbers of files, use batch processing APIs to reduce connection overhead:

// FTP system batch operation example - more efficient than individual operations
string[] localFiles = Directory.GetFiles("local/directory", "*.txt");
await ftpFs.UploadFilesAsync(localFiles, "/remote/path");

Architecture Design

Core Interface Hierarchy

 IFileSystem
 │
 IFileSystemOperations
 / \
ILocalFileSystem IRemoteFileSystem

Core Interfaces

• IFileSystem: Defines basic file operation interfaces
• IFileSystemOperations: Unified file system operation interface, inheriting from IFileSystem
• ILocalFileSystem: Local file system specific interface, extending unique functionalities
• IRemoteFileSystem: Remote file system connection management interface

Implementation Class Hierarchy

 FileSystemBase
 / \
 LocalFileSystem RemoteFileSystemBase
 / \
 FtpFileSystem SftpFileSystem

Base Classes

• FileSystemBase: Abstract base class for all file systems, implementing the IFileSystemOperations interface
• RemoteFileSystemBase: Abstract base class for remote file systems, inheriting from FileSystemBase and implementing IRemoteFileSystem
• LocalFileSystem: Concrete implementation of local file system
• FtpFileSystem: FTP file system implementation based on FluentFTP library
• SftpFileSystem: SFTP file system implementation based on SSH.NET library

Design Patterns

This library uses the following design patterns:

• Strategy Pattern: Different file systems implement the same interface (IFileSystemOperations) but have their own implementation strategies
• Template Method: Defines algorithm skeleton in FileSystemBase base class, with subclasses implementing specific steps
• Adapter Pattern: Adapts different file system APIs like FluentFTP and SSH.NET to a unified interface
• Simple Factory: The CreateClient() method within each file system class creates specific client instances
• Command Pattern: Encapsulates operation results through FileOperationResult for unified execution status handling

Key Workflows

1. File Upload Workflow:

   • Client calls IFileSystemOperations.UploadAsync
   • Executes different implementations based on actual file system type
   • Applies configured file naming rules and validation measures
   • Returns unified FileOperationResult

2. Remote System Connection Management:

   • Uses EnsureConnectedAsync() for automatic connection management
   • Automatic connection, operation, and disconnection lifecycle management

Contributing

We welcome Pull Requests and Issues to help us improve this library.

License

MIT