 |
VOOZH | about |
|
VOOZH | about |
This document describes the core data structures used for IO operations throughout AReaL. These dataclasses serve as the primary interfaces between different components of the system, enabling communication between training engines, inference servers, workflow executors, and configuration systems. The structures covered include:
ModelRequest and ModelResponse structures for inference. For details, see Request and Response Types.WeightUpdateMeta for disk/NCCL weight synchronization and LoRA support. For details, see Weight Update Metadata.SaveLoadMeta, FinetuneSpec, and checkpoint-related structures. For details, see Checkpoint Metadata.DeviceRuntimeInfo, StepInfo, and server monitoring structures. For details, see Runtime Information.For information about how these structures flow through training pipelines, see Training Engines. For workflow integration patterns, see Workflow and Rollout System. For comprehensive configuration reference, see Configuration System.
The ModelRequest dataclass represents a request for model inference. It serves as the primary input structure for both local and remote inference engines.
Key Fields:
| Field | Type | Purpose |
|---|---|---|
rid | str | Unique request ID (UUID) areal/api/io_struct.py29 |
input_ids | list[int] | Tokenized input sequence areal/api/io_struct.py30 |
gconfig | GenerationHyperparameters | Generation parameters (temperature, top_p, etc.) areal/api/io_struct.py31-33 |
image_data | list[str] | None | Base64-encoded images for VLM inputs areal/api/io_struct.py39 |
vision_msg_vllm | list | None | vLLM-specific vision message format areal/api/io_struct.py43 |
Sources: areal/api/io_struct.py28-60 areal/engine/sglang_remote.py43-89 areal/engine/vllm_remote.py44-96 areal/api/workflow_api.py14-40 areal/api/engine_api.py258-268
The ModelResponse dataclass contains the result of model inference, including generated tokens, log probabilities, and timing statistics.
Key Fields:
| Field | Type | Purpose |
|---|---|---|
output_tokens | list[int] | Generated token IDs areal/api/io_struct.py66 |
output_logprobs | list[float] | Log probability for each output token areal/api/io_struct.py67 |
stop_reason | str | Why generation stopped (length, stop, tool_calls, abort) areal/api/io_struct.py69 |
latency | float | Total generation latency (seconds) areal/api/io_struct.py78 |
routed_experts | np.ndarray | None | MoE routing info if requested areal/api/io_struct.py83 |
Derived Properties:
end_with_stop: Checks if output ends with EOS/PAD areal/api/io_struct.py94-104output_tokens_without_stop: Strips trailing stop tokens for training areal/api/io_struct.py107-130Sources: areal/api/io_struct.py62-132 areal/engine/sglang_remote.py91-127 areal/engine/vllm_remote.py98-127
Weight synchronization between training and inference engines requires coordinating metadata structures that describe how weights should be transferred and applied.
The WeightUpdateMeta dataclass coordinates weight synchronization. It supports disk, xccl (distributed), and awex update types areal/api/io_struct.py184
Core Fields:
type: Transfer mechanism (disk, xccl, or awex) areal/api/io_struct.py184path: Disk checkpoint path areal/api/io_struct.py185nccl_group_name: Unique process group name for XCCL/NCCL updates areal/api/io_struct.py190use_lora: Boolean flag for LoRA updates areal/api/io_struct.py193peft_config: Configuration for PEFT/LoRA adapters areal/api/io_struct.py197version: Version number for the weights areal/api/io_struct.py201Sources: areal/api/io_struct.py183-216 areal/api/engine_api.py175-183 areal/engine/sglang_remote.py129-187
The ParamSpec dataclass describes a single parameter tensor for distributed weight updates.
| Field | Type | Purpose |
|---|---|---|
name | str | Parameter name areal/api/io_struct.py151 |
shape | tuple | Tensor shape areal/api/io_struct.py152 |
dtype | str | Data type string areal/api/io_struct.py153 |
Sources: areal/api/io_struct.py150-160
The SaveLoadMeta dataclass contains information for checkpoint save/load operations, defining the target path, weight format, and associated tokenizer/processor. This is used by engines during initialization to restore model state areal/api/io_struct.py22 areal/api/engine_api.py22
The FinetuneSpec dataclass encapsulates training configuration needed for optimizer and scheduler initialization areal/api/io_struct.py134-147
Derived Properties:
total_train_steps: Computed as epochs * (dataset_size // batch_size) areal/api/io_struct.py140-142steps_per_epoch: Steps in a single epoch areal/api/io_struct.py144-146Sources: areal/api/io_struct.py134-147
AReaL uses strongly-typed dataclasses for all configuration. These structures are defined primarily in areal/api/cli_args.py.
Sources: areal/api/cli_args.py26 areal/api/io_struct.py17 examples/math/gsm8k_grpo_lora.yaml36-43
The DeviceRuntimeInfo dataclass provides a snapshot of GPU/device memory usage, including allocated and reserved stats. It is returned by TrainEngine and InferenceEngine to monitor health areal/api/io_struct.py17 areal/api/engine_api.py17
The LocalInfServerInfo dataclass tracks information about a locally launched inference server subprocess areal/api/io_struct.py18
| Field | Type | Description |
|---|---|---|
host | str | Server hostname/IP areal/api/io_struct.py18 |
port | int | Server port number areal/api/io_struct.py18 |
process | subprocess.Popen | Server process handle areal/api/io_struct.py18 |
Sources: areal/api/io_struct.py16-24 areal/api/engine_api.py17-23
Refresh this wiki