RolloutWorkflow API

This document describes the abstract RolloutWorkflow API, which defines the contract for implementing custom episode generation logic in AReaL's asynchronous RL training pipeline. A workflow encapsulates the logic for generating a single trajectory (episode) using an inference engine and producing training data with rewards.

For implementing custom workflows, see Implementing Custom Workflows For information about how workflows are coordinated during training, see Rollout Coordination For built-in workflow implementations, see Built-in Workflows

---

Purpose and Scope

The RolloutWorkflow API serves as the abstraction layer between the training orchestration system and episode generation logic. It allows users to define custom:

• Prompt construction strategies (e.g., chat templates with thinking tokens) areal/workflow/rlvr.py31-42
• Multi-turn conversation flows with retries and discounting areal/workflow/multi_turn.py19-135
• Tool-integrated reasoning (TIR) involving external tool execution and multi-round loops examples/tir/tir_workflow.py47-210
• Reward computation logic via async wrappers areal/workflow/rlvr.py63-109
• Trajectory filtering criteria by returning None areal/api/workflow_api.py21-23

All workflows implement a single async method arun_episode() that accepts an InferenceEngine and input data, then returns a dictionary of training tensors, a dictionary of interaction objects, or None to reject the trajectory areal/api/workflow_api.py14-18

Sources: areal/api/workflow_api.py14-18 areal/workflow/rlvr.py49-178 areal/workflow/multi_turn.py19-136 examples/tir/tir_workflow.py47-210

---

System Integration

The following diagram shows how RolloutWorkflow integrates with AReaL's training architecture, bridging the gap between high-level orchestration and low-level inference.

Workflow Architecture Integration

Sources: areal/api/workflow_api.py14-18 areal/workflow/rlvr.py139-162 areal/workflow/vision_rlvr.py103-137 areal/workflow/multi_turn.py58-84 examples/tir/tir_workflow.py101-143

---

Core API Definition

The RolloutWorkflow class is an abstract base class defined in areal/api/workflow_api.py14-39 All custom workflows must inherit from this class and implement the arun_episode() method.

Class Definition

Sources: areal/api/workflow_api.py14-39

---

Method Signature

arun_episode()

The core method that defines episode generation logic.

Signature:

Parameters:

Parameter	Type	Description
engine	InferenceEngine	The inference engine instance used for generating model responses. Provides agenerate() areal/api/workflow_api.py29-30
data	dict[str, Any]	Input data for the episode, typically a single sample from the dataset areal/api/workflow_api.py31-32

Returns:

Return Type	Meaning
dict[str, Any]	Training data dictionary containing tensors (e.g., input_ids, logprobs, rewards) areal/api/workflow_api.py36-37
None	Trajectory is rejected and will not be used for training areal/api/workflow_api.py21-23
dict[str, InteractionWithTokenLogpReward]	For agentic workflows: maps completion IDs to interaction objects with rewards areal/api/workflow_api.py11-18

Sources: areal/api/workflow_api.py14-38

---

Return Value Semantics

Standard Workflow Return Format

For traditional RL workflows, the return dictionary typically contains PyTorch tensors prepared for the trainer. Returning None from arun_episode() signals that the trajectory should be discarded areal/api/workflow_api.py21-23 This is useful for:

• Filtering low-quality generations.
• Rejecting trajectories that violate constraints.
• Implementing acceptance criteria.

The DistRolloutCoordinator will skip None results and not add them to the training queue. Built-in implementations like RLVRWorkflow return a dictionary of tensors with a batch dimension of 1, including input_ids, loss_mask, logprobs, versions, and rewards areal/workflow/rlvr.py170-178

Sources: areal/api/workflow_api.py21-23 areal/api/workflow_api.py36-37 areal/workflow/rlvr.py170-178

---

Workflow Type System

The following diagram illustrates the type hierarchy and type alias system:

Workflow Type Hierarchy

WorkflowLike Type Alias

The WorkflowLike type alias provides flexibility in how workflows are specified:

This allows workflows to be specified in configuration files as:

• Python class instances (programmatic usage).
• Python class types (instantiated by trainer).
• String import paths (YAML configuration) areal/workflow/rlvr.py75-81
• Duck-typed objects with compatible methods (specifically for agentic workflows using the run() method) areal/api/workflow_api.py71-74

Sources: areal/api/workflow_api.py110-115 areal/workflow/rlvr.py75-81

---

Performance Tracing in Workflows

Concrete workflow implementations use PerfTracer and SessionTracer to track the timing of different phases (e.g., generation vs. reward calculation).

• @trace_session("reward"): Decorates reward computation methods to track their duration areal/workflow/rlvr.py83-84
• atrace_session_phase("generate"): Async context manager used to wrap inference engine calls areal/workflow/rlvr.py130-131
• @session_context(): Decorates sample collection methods to manage session lifecycle areal/workflow/rlvr.py111-112

These traces are integrated with stats_tracker to log metrics like rewards and turn counts per episode areal/workflow/multi_turn.py123-125

Sources: areal/workflow/rlvr.py83-137 areal/workflow/multi_turn.py123-125

---

Integration with InferenceEngine

The workflow receives an InferenceEngine instance which provides async generation capabilities.

Rollout Sequence Diagram

The InferenceEngine provides agenerate() for non-blocking generation. In complex workflows like TIRWorkflow, this may be called multiple times in a loop to handle tool outputs examples/tir/tir_workflow.py171-210

Sources: areal/api/workflow_api.py29-30 areal/api/workflow_api.py16-18 areal/workflow/rlvr.py139-162 examples/tir/tir_workflow.py171-210

---

Deprecated AgentWorkflow

The AgentWorkflow class is deprecated as of version 1.0.0 areal/api/workflow_api.py66 It previously provided a separate abstraction for agent-based workflows that use external SDKs (like OpenAI's SDK).

Deprecation Details

Migration Path:

• Old: Inherit from AgentWorkflow and implement run(data, **extra_kwargs).
• New: Any class with a compatible async def run(data, **extra_kwargs) method works without inheritance areal/api/workflow_api.py71-74

The metaclass _DeprecatedAgentWorkflowMeta ensures a DeprecationWarning is emitted when any subclass is instantiated areal/api/workflow_api.py50-60

Sources: areal/api/workflow_api.py42-104

---

Code Entity Reference

Concept	Code Entity	Location
Abstract workflow interface	RolloutWorkflow	areal/api/workflow_api.py14
Core generation method	arun_episode()	areal/api/workflow_api.py16
Type alias for flexible specification	WorkflowLike	areal/api/workflow_api.py110
Phase tracing context	atrace_session_phase	areal/workflow/rlvr.py130
Deprecated agent abstraction	AgentWorkflow	areal/api/workflow_api.py63
Multi-turn retry logic	MultiTurnWorkflow	areal/workflow/multi_turn.py19
Tool calling logic	TIRWorkflow	examples/tir/tir_workflow.py47

Sources: areal/api/workflow_api.py14-115 areal/workflow/rlvr.py83-137 areal/workflow/multi_turn.py19-136 examples/tir/tir_workflow.py47-210