 |
VOOZH | about |
|
VOOZH | about |
This page documents the installation validation infrastructure that ensures AReaL is correctly installed and configured in both local and CI environments. The validation system performs environment checks, dependency verification, and basic functionality tests before training operations begin.
For information about the broader testing infrastructure, see 15.1 Test Infrastructure
Installation validation serves several critical functions:
The validation system operates at the boundary between system setup and actual training execution, ensuring that common configuration errors are caught before resource-intensive operations commence.
The validation infrastructure is built on a modular class hierarchy. BaseInstallationValidator provides core logic for parsing pyproject.toml and checking standard dependencies, while DockerInstallationValidator extends this for specialized deep-learning environments.
| Class / Function | File Path | Role |
|---|---|---|
BaseInstallationValidator | areal/tools/validation_base.py29 | Base class for dependency and version validation. |
DockerInstallationValidator | areal/tools/validate_docker_installation.py24 | Specialized validator for Docker environments with CUDA extensions. |
parse_pyproject | areal/tools/validation_base.py100-101 | Extracts dependency requirements from pyproject.toml. |
test_cuda_functionality | areal/tools/validate_docker_installation.py176-177 | Executes functional tests for FP8 (Transformer Engine) and Fused Optimizers (Apex). |
Sources: areal/tools/validation_base.py29-160 areal/tools/validate_docker_installation.py24-177
The following diagram maps the Natural Language validation steps to the specific code entities in the validate_docker_installation.py and validation_base.py files.
Diagram: Installation Validation Logic Flow
Sources: areal/tools/validation_base.py100-196 areal/tools/validate_docker_installation.py75-178
The Docker runtime environment requires validation of specialized CUDA kernels and libraries that are often compiled from source during the image build process.
The validator checks for the existence of specific C++ extensions and submodules that are critical for performance:
| Package | Submodules Validated | Purpose |
|---|---|---|
flash-attn | flash_attn_2_cuda | High-performance attention kernels areal/tools/validation_base.py65 |
megatron-core | megatron.core.parallel_state, megatron.core.tensor_parallel | Distributed training state management areal/tools/validate_docker_installation.py36-39 |
transformer_engine | transformer_engine.pytorch | FP8 training support areal/tools/validate_docker_installation.py32 |
apex | apex.optimizers, apex.normalization | Fused CUDA kernels for optimizers areal/tools/validate_docker_installation.py31 |
deep_ep | deep_ep | Expert Parallelism for DeepSeek-V3 MoE areal/tools/validate_docker_installation.py43 |
flash_mla | flash_mla | Multi-head Latent Attention (SM90+) areal/tools/validate_docker_installation.py41 |
deep_gemm | deep_gemm | FP8 GEMM library (SM90+) areal/tools/validate_docker_installation.py42 |
fla | fla.ops, fla.layers, fla.modules | Linear attention with Triton kernels areal/tools/validate_docker_installation.py44 |
causal_conv1d | causal_conv1d_cuda | Mamba/SSM dependency areal/tools/validate_docker_installation.py45 |
Sources: areal/tools/validate_docker_installation.py28-46 areal/tools/validation_base.py61-70
AReaL supports multiple inference backends. The DockerInstallationValidator dynamically detects which backend is present and ensures they are mutually exclusive within the environment areal/tools/validate_docker_installation.py94-138
Diagram: Inference Backend Validation Logic
Sources: areal/tools/validate_docker_installation.py94-138
Beyond simple imports, the system performs functional tests on CUDA-dependent features to ensure that the compiled extensions are compatible with the hardware.
The test_cuda_functionality method attempts to initialize a te.Linear layer and run a forward/backward pass using te.fp8_autocast areal/tools/validate_docker_installation.py182-195 This validates that the transformer_engine C++ extensions are correctly linked and the GPU supports required FP8 operations.
The script validates apex.optimizers.FusedAdam by performing a weight update on a small tensor areal/tools/validate_docker_installation.py180-195 This ensures the fused CUDA kernels for optimization are functional.
Sources: areal/tools/validate_docker_installation.py176-195
The validator maintains a list of CRITICAL_PACKAGES that must be present for the system to function. Failure to import any of these results in a non-zero exit code.
| Category | Packages |
|---|---|
| Core Training | torch, megatron-core, mbridge, flash-attn |
| Infrastructure | ray, hydra-core, omegaconf, wandb |
| Data/Models | transformers, datasets |
| Docker-Specific | grouped_gemm, apex, transformer_engine, causal_conv1d, megatron-bridge |
Sources: areal/tools/validation_base.py74-90 areal/tools/validate_docker_installation.py51-61
Validation extends beyond static checks to dynamic execution of representative examples. The tests/test_examples.py file contains integration tests that run full training loops for short durations to validate the end-to-end stack.
The run_example function tests/test_examples.py26-32 executes AReaL training scripts in a subprocess and monitors for a success pattern (e.g., "Train step 1/X done") tests/test_examples.py21
Key examples used for installation validation include:
Sources: tests/test_examples.py21-215
The validation is typically executed via the CLI:
Users should ensure their hardware and software meet the minimum requirements before running validation. For multi-node setups, a shared storage path (NAS) is required for checkpointing and logs.
| Requirement | Recommended Version |
|---|---|
| NVIDIA Driver | 550.127.08 |
| CUDA | 12.8 |
| Docker | 27.5.1 |
| Python | 3.11 or 3.12 |
Sources: areal/tools/validation_base.py16-26 areal/tools/validate_docker_installation.py1-16
Refresh this wiki