SARSA (State-Action-Reward-State-Action) in Reinforcement Learning

SARSA (State-Action-Reward-State-Action) is an on-policy reinforcement learning (RL) algorithm that helps an agent to learn an optimal policy by interacting with its environment. The agent explores its environment, takes actions, receives feedback and continuously updates its behavior to maximize long-term rewards.

Unlike off-policy algorithms like Q-learning which learn from the best possible actions, it updates its knowledge based on the actual actions the agent takes. This makes it suitable for environments where the agent's actions and their immediate feedback directly influence learning.

Components

Components of the SARSA Algorithm are as follows:

1. State (S): The current situation or position in the environment.
2. Action (A): The decision or move the agent makes in a given state.
3. Reward (R): The immediate feedback or outcome the agent receives after taking an action.
4. Next State (S'): The state the agent transitions to after taking an action.
5. Next Action (A'): The action the agent will take in the next state based on its current policy.
6. Discount Factor (γ): Determines how much importance is given to future rewards compared to immediate rewards.

SARSA focuses on updating the agent's Q-values (a measure of the quality of a given state-action pair) based on both the immediate reward and the expected future rewards.

How does SARSA Updates Q-values?

SARSA updates the Q-value using the Bellman Equation for SARSA:

Where:

• is the current Q-value for the state-action pair at time step t.
• is the learning rate (a value between 0 and 1) which determines how much the Q-values are updated.
• ​ is immediate reward the agent receives after taking action ​ in state ​.
• is the discount factor (between 0 and 1) which shows the importance of future rewards.
• is the Q-value for the next state-action pair.

Understanding the Update

• Immediate Reward: The agent gets reward ​ after taking action ​ in state ​.
• Future Reward: It uses to estimate future returns.
• Correction: The Q-value is adjusted based on the difference between expected and actual rewards.

This helps the agent improve its decisions step by step.

SARSA Algorithm Steps

Lets see how the SARSA algorithm works step-by-step:

1. Initialize Q-values: Begin by setting arbitrary values for the Q-table (for each state-action pair).

2. Choose Initial State: Start the agent in an initial state .

3. Episode Loop: For each episode (a complete run through the environment) we set the initial state ​ and choose an action ​ based on a policy like .

4. Step Loop: For each step in the episode:

• Take action ​ observe reward ​ and transition to the next state ​.
• Choose the next action ​ based on the policy for state .
• Update the Q-value for the state-action pair using the SARSA update rule.
• Set ​ and ​.

5. End Condition: Repeat until the episode ends either because the agent reaches a terminal state or after a fixed number of steps.

Implementation

Let’s consider a practical example of implementing SARSA in a Grid World environment where the agent can move up, down, left or right to reach a goal.

Step 1: Defining the Environment (GridWorld)

• Start Position: Initial position of the agent.
• Goal Position: Target the agent aims to reach.
• Obstacles: Locations the agent should avoid with negative rewards.
• Rewards: Positive rewards for reaching the goal, negative rewards for hitting obstacles.

GridWorld environment simulates the agent's movement, applying the dynamics of state transitions and rewards.

Here we will be using Numpy library for its implementation.

Step 2: Defining the SARSA Algorithm

The agent uses the SARSA algorithm to update its Q-values based on its interactions with the environment, adjusting its behavior over time to reach the goal.

Step 3: Defining the Epsilon-Greedy Policy

The epsilon-greedy policy balances exploration and exploitation:

• With probability ϵ, the agent selects a random action (exploration)
• With probability 1 − ϵ, it selects the action with the highest Q-value (exploitation)

To avoid bias when multiple actions have the same Q-value, ties are broken randomly among the best actions.

Step 4: Setting Up the Environment and Running SARSA

This step involves:

• Defining the grid world parameters like width, height, start, goal, obstacles.
• Setting the SARSA hyperparameters like episodes, learning rate, discount factor, exploration rate.
• Running the SARSA algorithm and printing the learned Q-values.

Output:

After running the SARSA algorithm the Q-values represent the expected cumulative reward for each state-action pair. The agent uses these Q-values to make decisions in the environment. Higher Q-values shows better actions for a given state.

You can download the complete code from here.

SARSA vs. Q-Learning

Exploration Strategies in SARSA

SARSA uses an exploration-exploitation strategy to choose actions. A common strategy is :

• Exploration: With probability , the agent chooses a random action (exploring new possibilities).
• Exploitation: With probability , the agent chooses the action with the highest Q-value for the current state (exploiting its current knowledge).

Over time, is often decayed to shift from exploration to exploitation as the agent gains more experience in the environment.

Advantages

• Learns from actual actions taken, making it suitable for environments where exploration directly affects outcomes.
• Produces safer and more realistic behavior since updates consider the agent’s current policy.
• Maintains stability during learning, especially when continuous exploration is required.
• Useful in real-world scenarios where decisions must adapt based on ongoing interaction with the environment.

Limitations

• Converges slower compared to off-policy methods like Q-learning.
• Strongly dependent on the exploration strategy (e.g., ε-greedy), which impacts performance.
• May learn suboptimal policies if exploration is not properly balanced.
• Less efficient in environments where the optimal action is clearly defined.