Shared Storage Validation

Purpose and Scope

Shared Storage Validation ensures that all workers in a distributed AReaL deployment can access a common filesystem for checkpoints, logs, and model weights. This validation is critical for distributed training workflows where multiple nodes must coordinate through shared files, such as when using the nfs name resolution strategy or disk-based weight synchronization.

In AReaL's asynchronous architecture, shared storage serves several purposes:

• Checkpoint persistence: Saving model weights, optimizer states, and training metadata areal/infra/scheduler/local.py153-165
• Weight synchronization: Disk-based weight updates between training and inference engines areal/infra/scheduler/local.py49
• Log aggregation: Centralized logging and experiment-specific log files areal/infra/scheduler/slurm.py167-187
• Name Resolution: Coordinating worker addresses using a shared record root for discovery via the nfs type areal/infra/scheduler/local.py140-144

Sources: areal/infra/scheduler/local.py49-165 areal/infra/scheduler/slurm.py122-187

---

Shared Storage in AReaL Architecture

Shared storage sits at the intersection of the scheduler system and the execution engines, enabling coordination across distributed workers. The validate_shared_path utility areal/infra/scheduler/local.py49 is the primary mechanism for ensuring these paths are valid and reside on appropriate network filesystems during scheduler initialization.

Distributed Storage Interaction

Diagram: Shared Storage Validation Architecture

Sources: areal/infra/scheduler/local.py138-144 areal/infra/scheduler/slurm.py120-126 areal/api/scheduler_api.py13-33

---

Storage Requirements

AReaL's shared storage must satisfy several requirements to support distributed training:

Requirement	Description	Implementation Context
Cross-node accessibility	All workers must access identical paths	LocalScheduler / SlurmScheduler init areal/infra/scheduler/local.py138
Network FS Type	Must be a distributed filesystem (NFS, Lustre, etc.)	validate_shared_path areal/infra/scheduler/local.py49
Path Existence	The directory must exist or be creatable	log_dir.mkdir areal/infra/scheduler/local.py171
Experiment Context	Paths are namespaced by experiment and trial	names.trial_root areal/infra/scheduler/local.py150

Scheduler Integration

Both LocalScheduler and SlurmScheduler perform validation of the fileroot and nfs_record_root during instantiation:

• LocalScheduler: Validates self.fileroot areal/infra/scheduler/local.py138-139 and nfs_record_root if the name resolve type is "nfs" areal/infra/scheduler/local.py140-144
• SlurmScheduler: Similarly validates these paths to ensure Slurm jobs can coordinate across the cluster areal/infra/scheduler/slurm.py120-126
• RayScheduler: While it uses Ray's internal object store for some communication, it also respects the enable_offload flag which may trigger shared storage dependencies for offloading parameters areal/infra/scheduler/ray.py69-72

Sources: areal/infra/scheduler/local.py138-171 areal/infra/scheduler/slurm.py120-134 areal/infra/scheduler/ray.py69-72

---

Validation Implementation

The validation logic ensures that the provided paths are not only present but also reside on a filesystem capable of multi-node consistency.

Validation Flow

Diagram: Path Validation Sequence in Schedulers

Sources: areal/infra/scheduler/local.py138-151 areal/infra/scheduler/slurm.py120-133

---

Infrastructure Validation

Beyond filesystem paths, the system validates the network and hardware environment required for distributed RPC communication.

Network and Port Validation

The scheduler ensures that network resources are available before launching worker processes:

• Port Discovery: find_free_ports is used in LocalScheduler to allocate ranges for RPC and engine communication areal/infra/scheduler/local.py51
• IP Resolution: gethostip determines the local IP address for worker identification areal/infra/scheduler/local.py53
• Worker Connectivity: The RayScheduler uses _ping_workers to verify that remote actors are ready and reachable via the Ray runtime areal/infra/scheduler/ray.py104-129

Hardware and Resource Validation

• GPU Detection: _get_device_count_safely scans /dev for nvidia or davinci devices to count GPUs without initializing a CUDA context, preventing premature resource lock areal/infra/scheduler/local.py73-90
• GPU Allocation: _allocate_gpus ensures the requested number of devices does not exceed available hardware on the local node areal/infra/scheduler/local.py203-210
• Placement Strategies: RayScheduler validates resource constraints using strategies like DeferredDeviceRayPlacementStrategy, SeparatedRayPlacementStrategy, or SharedRayPlacementStrategy areal/infra/scheduler/ray.py145-166

Sources: areal/infra/scheduler/local.py51-210 areal/infra/scheduler/ray.py104-166

---

Common Validation Failures and Warnings

Non-Network Filesystem Warning

If a path is detected on a local disk (e.g., /tmp on a local SSD), the system warns the user via validate_shared_path areal/infra/scheduler/local.py49 This is critical because:

1. Weight Sync: Training engines write weights to a directory that inference engines on other nodes must read.
2. Name Resolution: The nfs record root MUST be shared for name_resolve to function across different IPs areal/infra/scheduler/local.py140-144

Scheduler Exception Handling

The validation system uses specific exceptions to signal storage and environment failures:

• GPUAllocationError: Raised when GPU resources are insufficient areal/infra/scheduler/local.py208
• WorkerTimeoutError: Raised when workers fail to initialize or respond within the startup_timeout areal/infra/scheduler/ray.py116
• WorkerConfigurationError: Raised when worker specs or shared paths are misconfigured areal/infra/scheduler/local.py35
• WorkerFailedError: Raised when a worker process terminates unexpectedly during startup or ping areal/infra/scheduler/ray.py128

Sources: areal/infra/scheduler/local.py35-208 areal/infra/scheduler/ray.py116-128