Note

Access to this page requires authorization. You can try signing in or .

Access to this page requires authorization. You can try .

Update Kubernetes and node images across multiple clusters using Azure Kubernetes Fleet Manager

Applies to: ✔️ Fleet Manager ✔️ Fleet Manager with hub cluster

Platform admins managing large number of clusters often have problems with staging the updates of multiple clusters (for example, upgrading node OS image or Kubernetes versions) in a safe and predictable way. To address this challenge, Azure Kubernetes Fleet Manager (Fleet) allows you to orchestrate updates across multiple clusters using update runs.

Update runs consist of stages, groups, and strategies and can be applied manually for one-time updates, or automatically, for ongoing regular updates using auto-upgrade profiles. All update runs (manual or automated) honor member cluster maintenance windows.

This guide covers how to configure and manually execute update runs.

Prerequisites

  • Read the conceptual overview of this feature, which provides an explanation of update strategies, runs, stages, and groups referenced in this guide.

  • You must have a Fleet resource with one or more member clusters. If not, follow the quickstart to create a Fleet resource and join Azure Kubernetes Service (AKS) clusters as members.

  • Set the following environment variables:

    export GROUP=<resource-group>
export FLEET=<fleet-name>

  • If you're following the Azure CLI instructions in this article, you need Azure CLI version 2.58.0 or later installed. To install or upgrade, see Install the Azure CLI.

  • You also need the fleet Azure CLI extension, which you can install by running the following command:

    az extension add --name fleet

    Run the az extension update command to update to the latest version of the extension released:

    az extension update --name fleet

Defining the cluster upgrade sequence

Update run supports two options for the cluster upgrade sequence:

  • One by one: If you don't care about controlling the cluster upgrade sequence, one-by-one provides a simple approach to upgrade all member clusters of the fleet in sequence one at a time.
  • Control sequence of clusters using update groups and stages: If you want to control the cluster upgrade sequence, you can structure member clusters in update groups and update stages. You can store this sequence as a template in the form of an update strategy. You can create update runs later using the update strategies instead of defining the sequence every time you need to create an update run.

Note

Update runs honor the planned maintenance windows that you set at the AKS cluster level. For more information, see planned maintenance across multiple member clusters, which explains how update runs handle member clusters configured with planned maintenance windows.

Update all clusters one by one

  1. In the Azure portal, navigate to your Azure Kubernetes Fleet Manager resource.

  2. From the service menu, under Settings, select Multi-cluster update > Create a run.

  3. Enter a name for the update run, and then select One by one for the upgrade type.

    👁 Screenshot of the Azure portal pane for creating update runs that update clusters one by one in Azure Kubernetes Fleet Manager.

  4. Select one of the following options for the Upgrade scope:

    • Kubernetes version for both control plane and node pools
    • Kubernetes version for only control plane of the cluster
    • Node image version only

  5. Select one of the following options for the Node image:

    • Latest image: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
    • Consistent image: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.

    Important

    Node image versions are only valid for 90 days from their original publish date. If the target node image version selected by an update run exceeds the 90-day window by the time a member cluster is upgraded, the upgrade for that member cluster might fail.

    👁 Screenshot of the Azure portal pane for creating update runs. The upgrade scope section is shown.

  6. Select Create to create the update run.

  1. Create an update run using the az fleet updaterun create command with your chosen values for the --upgrade-type and --node-image-selection flags. The following command creates an update run that upgrades the Kubernetes version for both control plane and node pools and uses the latest node image available for each cluster in its region.

    The --upgrade-type flag supports the following values:

    • Full upgrades Kubernetes version for control plane and node pools along with the node images.
    • ControlPlaneOnly only upgrades the Kubernetes version for the control plane of the cluster.
    • NodeImageOnly only upgrades the node images.

    The --node-image-selection flag supports the following values:

    • Latest: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
    • Consistent: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.
    az fleet updaterun create \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name run-1 \
 --upgrade-type Full \
 --kubernetes-version 1.26.0 \
 --node-image-selection Latest

  2. Start the update run using the az fleet updaterun start command.

    az fleet updaterun start \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name run-1

When creating an update run, you have the ability to control the scope of the update run. The --upgrade-type flag supports the following values:

  • ControlPlaneOnly only upgrades the Kubernetes version for the control plane of the cluster.
  • Full upgrades Kubernetes version for control plane and node pools along with the node images.
  • NodeImageOnly only upgrades the node images.

Also, --node-image-selection flag supports the following values:

  • Latest: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
  • Consistent: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.

Important

Node image versions are only valid for 90 days from their original publish date. If the target node image version selected by an update run exceeds the 90-day window by the time a member cluster is upgraded, the upgrade for that member cluster might fail.

Starting an update run:

To start update runs, run the following command:

az fleet updaterun start \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name <run-name>

Update clusters using groups and stages

You can define an update run using update stages to sequentially order the application of updates to different update groups. For example, a first update stage might update test environment member clusters, and a second update stage would then update production environment member clusters. You can also specify what approvals are required before or after each stage, and a wait time between the update stages. You can store this sequence as a template in the form of an update strategy.

  1. In the Azure portal, navigate to your Azure Kubernetes Fleet Manager resource.

  2. From the service menu, under Settings, select Multi-cluster update > Create a run.

  3. Enter a name for the update run, and then select Stages for the update sequence type.

    👁 Screenshot of the Azure portal page for choosing stages mode within update run.

  4. Select Create stage, and then enter a name for the stage and set approvals and a wait time between stages if desired.

    👁 Screenshot of the Azure portal page for creating a stage and defining wait time.

  5. Select the update groups that you want to include in this stage. You can also specify the order of the update groups if you want to update them in a specific sequence. When you're done, select Create.

    👁 Screenshot of the Azure portal page for stage creation that shows the selection of upgrade groups.

  6. Select one of the following options for the Upgrade scope:

    • Kubernetes version for both control plane and node pools
    • Kubernetes version for only control plane of the cluster
    • Node image version only

  7. Select one of the following options for the Node image:

    • Latest image: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
    • Consistent image: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.

    👁 Screenshot of the Azure portal pane for creating update runs. The upgrade scope section is shown.

  8. Select Create to create the update run.

    Specifying stages and their order every time when creating an update run can get repetitive and cumbersome. Update strategies simplify this process by allowing you to store templates for update runs. For more information, see update strategy creation and usage.

  9. In the Multi-cluster update menu, select the update run, and then select Start.

  1. Create a JSON file to define the stages and groups for the update run. Here's an example of input from the stages file (example-stages.json):

    {
 "stages": [
 {
 "name": "stage1",
 "groups": [
 {
 "name": "group-1a"
 },
 {
 "name": "group-1b"
 },
 {
 "name": "group-1c"
 }
 ],
 "afterStageWaitInSeconds": 3600
 },
 {
 "name": "stage2",
 "groups": [
 {
 "name": "group-2a"
 },
 {
 "name": "group-2b"
 },
 {
 "name": "group-2c"
 }
 ]
 }
 ]
}

  2. Create an update run using the az fleet updaterun create command with the --stages flag set to the name of your JSON file and your chosen values for the --upgrade-type and --node-image-selection flags. The following command creates an update run that upgrades the Kubernetes version for both control plane and node pools and uses the latest node image available for each cluster in its region.

    The --upgrade-type flag supports the following values:

    • Full upgrades Kubernetes version for control plane and node pools along with the node images.
    • ControlPlaneOnly only upgrades the Kubernetes version for the control plane of the cluster.
    • NodeImageOnly only upgrades the node images.

    The --node-image-selection flag supports the following values:

    • Latest: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
    • Consistent: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.
    az fleet updaterun create \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name run-1 \
 --upgrade-type Full \
 --kubernetes-version 1.26.0 \
 --node-image-selection Latest \
 --stages example-stages.json

  3. Start the update run using the az fleet updaterun start command.

    az fleet updaterun start \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name run-1

Create an update run using update strategies

Creating an update run requires you to specify the stages, groups, order each time. Update strategies simplify this process by allowing you to store templates for update runs.

Note

It's possible to create multiple update runs with unique names from the same update strategy.

You can create an update strategy using one of the following methods:

Save an update strategy while creating an update run

Manage an update run

The following sections explain how to manage an update run using the Azure portal and Azure CLI.

  • You can Start an update run that's either in Not started or Failed state using the az fleet updaterun start command:

    az fleet updaterun start \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name <run-name>

  • You can Stop a currently Running update run using the az fleet updaterun stop command:

    az fleet updaterun stop \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name <run-name>

  • You can skip update stages or groups by specifying them in the --targets flag using the az fleet updaterun skip command:

    az fleet updaterun skip \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name <run-name> \
 --targets Group:my-group-name Stage:my-stage-name

    For more information, see conceptual overview on the update run states and skip behavior on runs/stages/groups.

Automate update runs using auto-upgrade profiles

Auto-upgrade profiles are used to automatically execute update runs across member clusters when new Kubernetes or node image versions are made available.

For more information on configuring auto-upgrade profiles, see automate upgrades of Kubernetes and node images using Azure Kubernetes Fleet Manager.

Generate an update run from an auto-upgrade profile

When you create an auto-upgrade profile, your clusters can be on various versions of Kubernetes or node image. Depending on your selected auto-upgrade channel, it might be some time before a new version release triggers auto-upgrade to create and execute an update run.

Auto-upgrade allows you to generate a new update run at any time using the az fleet autoupgradeprofile generate-update-run command. The resulting update run is based on the current AKS-published Kubernetes or node image version.

az fleet autoupgradeprofile generate-update-run \
 --resource-group $GROUP \
 --fleet-name $FLEET \
 --name $AUTOUPGRADEPROFILE

The generated update run isn't automatically started, allowing you to review it. If you're satisfied with the generated update run, you can start and manage it by following the steps in manage an update run.

Next steps

Feedback

Was this page helpful?

Additional resources

  • Last updated on