Parallelism Constraint Validation

Purpose and Scope

This page documents the constraint validation system that ensures parallelism configurations—Tensor Parallelism (TP), Pipeline Parallelism (PP), Context Parallelism (CP), and Expert Parallelism (EP)—are compatible with model architectures and distributed runtime environments. Before distributing a model across GPUs, AReaL validates that the requested parallelism strategy satisfies mathematical constraints imposed by the model's structure (e.g., attention head counts, expert counts, and block alignment). These validations prevent silent failures, dimension mismatches, or incorrect behavior during distributed training across backends like FSDPEngine, MegatronEngine, and ArchonEngine.

Sources: areal/engine/fsdp_engine.py218-240 areal/engine/megatron_engine.py168-186 areal/experimental/engine/archon_engine.py147-200

---

Why Validation is Necessary

Parallelism strategies require specific divisibility relationships between model architecture parameters and parallelism sizes. Violating these constraints leads to:

• Incorrect Tensor Sharding: If the number of attention heads is not divisible by tp_size, heads cannot be evenly distributed across devices, leading to runtime crashes in attention layers.
• Collective Communication Failures: Dimension mismatches during All-to-All operations in Context Parallelism (Ulysses) or Expert Parallelism (EP). For example, ulysses_pad_and_slice_inputs handles sequence length alignment for Ulysses SP areal/engine/fsdp_engine.py91-94
• MoE Routing Errors: In Mixture-of-Experts (MoE) models, the number of experts must be compatible with the Expert Parallelism (EP) size to ensure correct dispatching areal/engine/megatron_engine.py81-83
• Resource Underutilization: Invalid mesh dimensions that do not multiply to the total world size.
• Weight Synchronization Mismatches: When syncing weights between training and inference engines, tensor shapes and dtypes must be validated to match the receiver's model structure.

Sources: areal/engine/megatron_engine.py81-83 areal/engine/fsdp_engine.py91-94 areal/engine/megatron_engine.py117-119

---

Validation Function Overview

AReaL provides several validation layers, spanning from configuration-level dataclass checks to engine-specific runtime validations.

Function / Class	Purpose	Key Constraints
NormConfig.__post_init__	Validates normalization settings	Ensures group_size is positive if using group-level normalization areal/api/cli_args.py79-95
MicroBatchSpec.__post_init__	Validates packing algorithm	Ensures packing_algorithm is one of ffd or kk areal/api/cli_args.py140-146
ArchonEngine._validate_model_type	Architecture support check	Verifies model type is supported by the specific engine areal/experimental/engine/archon_engine.py161-166
is_valid_attn_impl	FSDP attention validation	Validates if the attention implementation is supported areal/api/cli_args.py19-23
is_valid_vision_model	Multi-modal validation	Checks if the model configuration supports vision tasks areal/engine/fsdp_engine.py70-78

Sources: areal/api/cli_args.py79-146 areal/experimental/engine/archon_engine.py161-166 areal/engine/fsdp_engine.py70-78

---

Data and Configuration Validation

Configuration objects use __post_init__ to enforce constraints immediately upon instantiation, typically during YAML parsing or CLI argument processing.

Normalization Constraints

The NormConfig class ensures that if mean_level or std_level is set to "group", a valid group_size is provided. It also restricts levels to {"batch", "group", None} areal/api/cli_args.py81-95

Micro-batching Constraints

MicroBatchSpec validates the packing_algorithm against the PACKING_ALGORITHMS constant (typically ffd for First Fit Decreasing or kk for Karmarkar-Karp) areal/api/cli_args.py142-146

Sources: areal/api/cli_args.py79-95 areal/api/cli_args.py140-146

---

System Integration and Data Flow

Validation logic is triggered during the initialization phase of the training engines and before critical collective operations.

Engine Initialization Flow

Title: Engine Initialization and Architecture Validation

Sources: areal/experimental/engine/archon_engine.py150-213 areal/engine/fsdp_engine.py219-240

Code Entity Association

Title: Mapping Constraints to Code Implementation

Sources: areal/experimental/engine/archon_engine.py161-166 areal/api/cli_args.py79-95 areal/api/cli_args.py140-146 areal/infra/rpc/rtensor.py1-25

---

Common Constraint Violations

Error Message / Symptom	Likely Cause	Solution
ValueError: mean_level must be 'batch', 'group' or None	Invalid normalization level provided	Update NormConfig to use supported level areal/api/cli_args.py82-85
ValueError: group_size must be a positive integer...	NormConfig misconfigured for group normalization	Set group_size >= 1 when using mean_level='group' areal/api/cli_args.py90-95
ValueError: packing_algorithm must be one of...	Unsupported micro-batch packing algorithm	Choose ffd or kk in MicroBatchSpec areal/api/cli_args.py142-146
404 Shard {shard_id} not found	RTensor storage retrieval failure	Ensure shard was stored via PUT /data/{shard_id} before fetching areal/infra/rpc/guard/data_blueprint.py85-94
400 One or more requested shards were not found	Batch RTensor retrieval failure	Verify all shard_ids exist in the data proxy areal/infra/rpc/guard/data_blueprint.py140-150

Sources: areal/api/cli_args.py82-146 areal/infra/rpc/guard/data_blueprint.py85-150