Tool Call Integration

This page documents how AReaL parses and integrates tool calls within agentic RL workflows. Tool call integration enables language models to invoke external functions during reasoning or interaction, with structured parsing and full token-level tracking for training.

Scope: This page covers tool call parsing logic, data structures, and integration with the OpenAI-compatible client. For the broader agentic RL architecture, see 6.1 Agentic RL Overview For multi-turn conversation mechanics, see 6.4 Multi-turn Conversations

Overview

Tool call integration in AReaL provides:

1. Structured Parsing: Extracts tool calls from raw model text using configurable parsers (e.g., Qwen-style tags, XML, or JSON) .
2. OpenAI API Compatibility: Maps internal model responses to ChatCompletionMessageFunctionToolCall or ResponseFunctionToolCall objects .
3. Reasoning Extraction: Separates <think> blocks from the actual tool call content to support reasoning models like DeepSeek or Qwen .
4. Backend Agnostic Parsing: Provides specialized logic for both SGLang and vLLM backends to ensure consistent parsing behavior regardless of the inference engine .

Implementation Detail: process_tool_calls

The core logic resides in . The function process_tool_calls acts as a dispatcher that routes the raw text to backend-specific parsers.

Logic Flow

1. Backend Dispatch: If an SGLang-specific parser is requested, it calls _process_tool_calls_sglang . Otherwise, it defaults to _process_tool_calls_vllm .
2. Reasoning Detection: Both paths use _detect_think_and_return_ori_think to split the response into reasoning_text and content_text .
3. Parser Execution:
   • SGLang: Uses sglang.srt.function_call.function_call_parser.FunctionCallParser .
   • vLLM: Uses vllm.tool_parsers.ToolParserManager to load the appropriate parser class (e.g., qwen3_xml, llama3_json) .
4. Object Mapping: It constructs either ChatCompletionMessageFunctionToolCall (for standard Chat API) or ResponseFunctionToolCall (for Responses API) .

Sources:

Natural Language to Code Entity Mapping

This diagram illustrates how raw model output containing tool tags is transformed into structured Python objects within the AReaL framework.

Tool Call Transformation Flow

Sources: ,

Comparison of Tool Call Types

Feature	ChatCompletionMessageFunctionToolCall	ResponseFunctionToolCall
API Source	OpenAI Chat Completions	OpenAI Responses API
Type Field	"function"	"function_call"
ID Prefix	call_	fc- and call_
Arguments	function.arguments	arguments

Integration with Agentic Workflows

Tool calls are often used in multi-turn reasoning loops where the agent must perform a calculation or query a database before providing a final answer. AReaL supports this through its experimental inference and agent services, which can route tool-calling agents through a proxy gateway.

Multi-Turn Workflow Integration

The MultiTurnWorkflow uses the ArealOpenAI client to manage turn-based interactions that may include tool usage.

1. Client Management: ArealOpenAI wraps the inference engine and provides OpenAI-compatible interfaces for chat completions .
2. Tool Parsing during Generation: When client.chat.completions.create is called, the underlying implementation uses process_tool_calls to detect if the model attempted to use a tool .
3. State Persistence: Tool calls and their reasoning are stored in the InteractionCache as part of the InteractionWithTokenLogpReward object .

Sources: ,

Reward Assignment for Tool Use

In multi-turn tool-integrated workflows, rewards are typically assigned based on the final outcome, and intermediate tool-calling steps are incentivized through reward propagation.

Reward Flow and Discounting

The MultiTurnWorkflow and ArealOpenAI client handle the temporal credit assignment for tool calls.

Tool Workflow Reward Flow

1. Iteration: The MultiTurnWorkflow loops until a correct answer is found or max_turns is reached .
2. Interaction Persistence: Each turn's response is fetched and cached via client.get_interaction .
3. Discounting: If multiple turns (including tool calls) are required, a turn_discount is applied to the final reward .
4. Reward Propagation: The ArealOpenAI client applies the reward to the interaction session via set_reward , which allows the cache to propagate rewards back through the conversation tree using apply_reward_discount .

Sources: , ,