Async Rollout Execution

Purpose and Scope

This page describes the asynchronous rollout execution system in AReaL's inference engines, focusing on the submit, wait, and prepare_batch patterns that enable concurrent rollout generation and training. The system is designed to decouple the generation of trajectories (rollouts) from the optimization steps, allowing the training engine to process one batch while the inference engines generate the next.

For details on the inference engine API contracts, see 4.1 InferenceEngine API For workflow implementation details, see 5.1 RolloutWorkflow API

---

Submit/Wait Pattern

The core async rollout pattern separates request submission from result collection through two primary methods defined in the InferenceEngine interface areal/api/engine_api.py703-779

submit()

The submit method enqueues a rollout task and returns immediately with a unique task ID areal/api/engine_api.py703-750

Parameter	Type	Description
data	dict[str, Any]	Input data for the workflow (e.g., prompt, reference solution).
workflow	WorkflowLike	The RolloutWorkflow implementation to execute areal/api/workflow_api.py110-115
workflow_kwargs	dict[str, Any] | None	Arguments passed to the workflow constructor.
should_accept_fn	Callable | str | None	Optional filter to reject trajectories (e.g., based on reward).
group_size	int	Number of independent trajectories to generate for this input.
is_eval	bool	Flag to indicate if this is an evaluation task (affects metrics).

Sources: areal/api/engine_api.py703-750 areal/infra/controller/rollout_controller.py53-62 areal/api/workflow_api.py110-115

wait()

The wait method blocks until a specified number of trajectories have been completed and accepted areal/api/engine_api.py752-779

Parameter	Type	Description
count	int	Number of accepted trajectories to retrieve.
timeout	float | None	Maximum time to wait in seconds.
raise_timeout	bool	If True, raises TimeoutError on expiry; else returns available results.

Returns: list[dict[str, Any] | None]. None entries represent slots where a trajectory was generated but rejected by the should_accept_fn areal/api/workflow_api.py23-24

Sources: areal/api/engine_api.py752-779 areal/infra/controller/rollout_controller.py270-295 areal/api/workflow_api.py23-24

---

Prepare Batch Pattern

The prepare_batch method provides a high-level abstraction for continuous data consumption. It automatically pulls data from a StatefulDataLoader, submits tasks, and waits until a full training batch is ready areal/api/engine_api.py852-897

Internal Mechanism

1. Initialization: On the first call, the engine creates an internal generator by calling cycle_dataloader on the provided dataloader areal/api/engine_api.py875-885
2. Continuous Submission: It submits tasks to fill the pipeline capacity, respecting the max_concurrent_rollouts setting areal/infra/controller/rollout_controller.py205-210
3. Batch Assembly: It calls wait() to collect the required number of trajectories and packages them into a format ready for the TrainEngine.

⚠️ Configuration Persistence: Parameters like workflow, group_size, and should_accept_fn are captured during the first prepare_batch call and reused. Subsequent calls with different arguments will not change the behavior of the active generator areal/api/engine_api.py855-865

Sources: areal/api/engine_api.py852-897 areal/infra/controller/rollout_controller.py205-210 areal/api/engine_api.py232-237

---

Async Request Flow

The following diagram illustrates the interaction between the Trainer, the RolloutController, and the remote inference backends.

Rollout Execution Sequence

Sources: areal/infra/controller/rollout_controller.py245-295 areal/infra/remote_inf_engine.py125-176 areal/api/engine_api.py703-779 areal/engine/sglang_remote.py39-126 areal/engine/vllm_remote.py39-124 areal/infra/remote_inf_engine.py60-61

---

Version Tracking and Off-policyness

To ensure training stability in an asynchronous environment, AReaL tracks the model version used for every rollout.

1. Version Assignment: When submit is called, the current actor_version is attached to the ModelRequest areal/api/io_struct.py28-59
2. Metadata Propagation: The inference backend includes this version in the ModelResponse areal/api/io_struct.py63-83
3. Staleness Check: The StalenessManager in the RolloutController monitors the delta between the current trainer version and the rollout version. If current_version - rollout_version > max_head_offpolicyness, the rollout may be discarded or flagged areal/infra/staleness_manager.py1-40

Versioning Logic Diagram

Sources: areal/infra/staleness_manager.py1-40 areal/api/engine_api.py683-701 areal/api/io_struct.py28-83 areal/infra/controller/rollout_controller.py100-102

---

Dynamic Filtering with should_accept_fn

The should_accept_fn allows the system to filter out low-quality trajectories before they ever reach the training engine.

• Function-based: A Python callable that takes a trajectory dictionary and returns a boolean.
• String-based: A string expression (e.g., "reward.mean() > 0.0") that is evaluated against the trajectory data areal/api/engine_api.py715-725

When wait(count=N) is called, the controller will continue processing generated trajectories until N items have passed the should_accept_fn. This is critical for algorithms like Best-of-N or rejection sampling areal/infra/remote_inf_engine.py70-123

Sources: areal/api/engine_api.py703-750 areal/infra/controller/rollout_controller.py245-295 areal/infra/remote_inf_engine.py70-123

---

Summary of Patterns

Method	Execution Mode	Best Use Case
submit	Asynchronous	Custom complex scheduling where submission logic is decoupled from collection.
wait	Synchronous (Blocking)	Collecting results after performing intermediate training or IO tasks.
prepare_batch	Iterative / Streaming	Standard RL training loops where the engine handles data loading and batching.
rollout_batch	Synchronous (Blocking)	Offline dataset generation or simple evaluation scripts areal/api/engine_api.py197-230

Sources: areal/api/engine_api.py703-897 areal/infra/controller/rollout_controller.py245-310 areal/api/engine_api.py197-230