How do I resolve security vulnerabilities in Amazon Linux packages that remain unpatched after I applied the patches?

I applied all available security updates to my Amazon Elastic Compute Cloud (Amazon EC2) instances that run on Amazon Linux. However, vulnerability scanners still report unpatched common vulnerabilities and exposures (CVEs).

Short description

Amazon Linux applies security fixes to stable package versions without a full version upgrade. However, third-party scanners that rely on upstream version numbers can't detect the backport fixes.

You might also experience security vulnerability issues for the following reasons:

• The repository version is locked.
• Your system requires a reboot to apply pending kernel updates.
• You use packages from third-party repositories.
• The package version is locked.
• Source-compiled packages override the RPM versions.
• The packages are bundled.
• The CVEs have no available patch.

To resolve persistent security vulnerabilities on Amazon EC2 instances that run on Amazon Linux, take the following troubleshooting actions.

Important: If you manage multiple instances, then use Patch Manager, a capability of AWS Systems Manager, to manage patching. With Patch Manager, you can define a patch baseline, schedule a maintenance window, and apply fleet-wide updates on a recurring basis. Manual updates from the shell aren't scalable and don't produce compliance evidence.

Resolution

Important: Amazon Linux 2 (AL2) will reach its end of life on June 30, 2026. After end of support, AL2 won't receive security updates or produce new patches for new CVEs. If you run AL2, then migrate to Amazon Linux 2023 (AL2023) before the end-of-support date. Test migrated applications in a nonproduction environment first. For changes that might affect your workloads, see Amazon Linux 2023 release notes. For commands, AL2023 uses dnf instead of yum.

Prerequisite: Create an Amazon Machine Image (AMI) of your instance to use as a backup in case you experience boot failures or service outages.

Use Amazon Inspector to scan your instances

Use Amazon Inspector to scan your instances for package vulnerabilities. Amazon Inspector integrates with Amazon Linux Security Advisories to remove false positives and report backported fixes. For information about costs, see Amazon Inspector pricing.

Verify that your patch resolves the CVE

Complete the following steps:

1. Open the Amazon Linux Security Center, and then choose Explore CVEs.
2. Wait for the page to load, and then in the search bar, enter the CVE ID from your vulnerability scanner output.
   Note: If the CVE isn't listed in the Amazon Linux Security Center, then Amazon Linux might not maintain the package. The vulnerability might be in a third-party package, a manually installed version, or a bundled dependency. For troubleshooting steps, proceed to Replace packages from third-party repositories.
3. On the CVE page, check for one of the following statues in the Affected Packages table:
   If the Status is Fixed, then the patch resolve the CVE in the package version that's listed in the Advisory column.
   If the Status is Not affected, then the CVE isn't applicable to your configuration.
   If the Status is Pending Fix, then a fix is in progress.
   If the Status is No Fix Planned, then there's no planned fix.
   Note: For best practices on how to resolve Pending Fix and No Fix Planned status, proceed to Manage vulnerabilities with no available patch.
4. To find a specific package version, choose the advisory ID in the Advisory column for the package.
5. Under New packages, check the list for the package versions that contain the fix. Note the package version that's listed for your architecture.

To confirm that your installed package was fixed, run the following command to check the package change log:

rpm -q --changelog PACKAGE_NAME | grep -i "CVE-YYYY-#####"

Note: Replace PACKAGE_NAME with the package name and CVE-YYYY-##### with the CVE ID. If the command returns a matching line, then the fix for the CVE is included in your installed package version.

To use the package manager to check for a specific CVE fix, run the following command:

dnf updateinfo info "CVE-YYYY-#####"

Note: Replace CVE-YYYY-##### with the CVE ID. For AL2, replace dnf with yum.

To list only pending advisories that reference the CVE, run the following command:

dnf updateinfo list --cve CVE-YYYY-#####

Note: Replace CVE-YYYY-##### with the CVE ID. For AL2, replace dnf with yum.

If you updated your package, then run the following command to check your installed package version:

rpm -qa | grep PACKAGE_NAME

Note: Replace PACKAGE_NAME with the package name. The package version must be equal to or later than the version that's listed under New packages.

If you used an equal or later version, then your patches resolve the CVE regardless of what scanner reports. To show an auditor or compliance team that the CVE was fixed, share the Amazon Linux Security Center advisory for the CVE. The advisory includes the related packages, fixed version, and CVE ID.

(Only AL2023) Update to the latest repository version

Note: AL2 doesn't use versioned repositories. Instead, it pulls the latest available packages when you run updates.

AL2023 uses versioned repositories and locks each AMI to a specific repository version by default. If your EC2 instance is locked to an earlier version, then the instance doesn't have access to the latest security patches.

To make sure that you use the latest repository version, complete the following steps:

1. To check whether a later repository version is available for your instance, run the following command:

   dnf check-release-update

   Note: If you use custom base images that are locked to a specific release for stability, then test the release update in a test environment first. For more information, see Best practices for safely deploying updates.
2. To update to the latest repository version and apply all available updates, run the following command:

   sudo dnf update --releasever=latest

   Note: To update to a specific release version instead of to the latest, replace latest with the release version number. For information about available release versions, see Amazon Linux 2023 release notes.
   If you receive the "Errors during downloading metadata for repository 'amazonlinux'" error message, then confirm that you entered the correct version number in the command.
3. To verify the repository version after the update, run the following command:

   cat /etc/os-release
   

Apply available security updates

Complete the following steps:

1. To apply all available security updates, run the following command:

   sudo dnf update –security

   Note: For AL2, replace dnf with yum.

2. Run the following command to verify that there are no pending security updates:

   dnf updateinfo list --security

   Note: For AL2, replace dnf with yum and remove --.

3. To confirm that the running kernel version is the same as the installed version, run the following command:

   uname -r 

   Then, run the following command:

   rpm -q kernel | sort -V | tail -1

4. If the running kernel version is different from the installed version, then run the following command to reboot your instance:

   sudo reboot

5. Confirm that the kernel version matches the latest installed version. To apply kernel updates, you must reboot your system.

To apply kernel security patches without a reboot, use Kernel Live Patching for AL2023 or AL2. Live patching covers only Important and Critical CVEs. You must still update and reboot for lower severity or non-kernel CVEs. Live patches are published only for the support window for the kernel version. After the window closes, you must reboot into a later kernel to continue to receive security updates. Live patches also don't apply to all kernel subsystems. Some changes, such as data-structure changes, require a reboot.

Note: Plan for periodic reboots even if you use live patching.

Replace packages from third-party repositories

Packages from third-party repositories, such as Extra Packages for Linux (EPEL), don't receive Amazon Linux security patches.

To resolve security vulnerability issues, complete the following steps to replace the packages with versions from official Amazon Linux repositories:

1. To check whether the package is from a third-party repository, run the following command:

   dnf info PACKAGE_NAME

   Note: Replace PACKAGE_NAME with the package name. For AL2, replace dnf with yum.
   In the output, check the From repo value. If the package is from EPEL or another third-party repository, then replace the package with the Amazon Linux version.
2. To list other packages that you installed from EPEL, run the following command:

   sudo dnf list installed | awk '$3 ~ /epel/'

   Note: For AL2, replace dnf with yum.
3. If other packages in your configuration depend on EPEL libraries, then replace them with Amazon Linux equivalents before you deactivate the repository.
   Example command for AL2 to replace an EPEL nginx package:

   sudo yum remove nginx -y
    sudo amazon-linux-extras install nginx1

   Example command for AL2023 to replace an EPEL nginx package:

   sudo dnf remove nginx -y
    sudo dnf install nginx

   Note: Replace nginx with your package name. The amazon-linux-extras command is available only on AL2. AL2023 provides packages directly through the default repositories.
4. Run the following command to deactivate EPEL:

   sudo dnf config-manager --set-disabled epel
    sudo dnf clean all

   Important: When you deactivate EPEL, packages that came from EPEL no longer receive updates. For AL2, replace dnf with yum and set-disabled with --disable.
5. To verify that the change log for the updated package shows the CVE fixes, run the following command:

   rpm -q --changelog nginx | head -20

   Note: Replace nginx with your package name.

Check for version locking

Important: Version locking might occur in Amazon Elastic Container Service (Amazon ECS)-optimized AMIs and Amazon Elastic Kubernetes Service (Amazon EKS)-optimized AMIs. For these AMIs, it's a best practice to keep version locks on the kernel and on packages such as containerd, runc, kubelet, and kube-proxy. The version locks keep the AMI compatible with the AWS managed container runtime or Kubernetes control plane. If you remove the version locks, then the underlying packages might break the node, fail node registration, or cause workload outages. Instead, upgrade the AMI version.

If your package version is locked, then you can't install security updates.

To remove unnecessary version locks, complete the following steps:

1. If there's no plugin, then run the following command to install it:

   sudo dnf install -y python3-dnf-plugin-versionlock

2. To check for version locks, run the following command:

   sudo dnf versionlock list

   Note: For AL2, replace dnf with yum.

3. If there's a version lock, then run the following command to remove the lock:

   sudo dnf versionlock delete PACKAGE_NAME

   Note: Replace PACKAGE_NAME with your package name. For AL2, replace dnf with yum.
   -or-
   To remove all version locks, run the following command:

   sudo dnf versionlock clear

   Note: For AL2, replace dnf with yum.

4. To refresh the repository metadata and apply updates, run the following command:

   sudo dnf clean all 
   sudo dnf repolist 
   sudo dnf check-update 
   sudo dnf update --security

   Note: For AL2, replace dnf with yum.

Resolve version discrepancies from manual installations

Prerequisites: You must meet the requirements for Session Manager, a capability of Systems Manager. Also, use Fleet Manager, a capability of Systems Manager, to make sure that the instance appears as Online.

Versions that you manually installed don't receive security patches from Amazon Linux repositories. To receive security updates, use repository-provided packages. If you manually installed both software and repository versions, then your vulnerability scanner might report unresolved CVEs. To resolve this issue, remove the manually installed software.

To identify the active binaries, run the following command:

which -a ssh 
which -a sshd 
ssh -V 
rpm -qa | grep openssh

If ssh -V has a different version from rpm -qa, then run the following command to locate the manual installation:

ls -l /usr/local/bin/ssh* ls -l /usr/local/sbin/sshd* ls -l /bin/ssh ls -l /sbin/sshd

To use only repository versions, complete the following steps:

1. Use Session Manager to connect to your instance.
   Note: You can't connect with SSH because you must stop the SSH service to return to repository versions. If you can't connect with Session Manager, then use EC2 Serial Console or Amazon EC2 Instance Connect to connect during a maintenance window. Before you proceed, make sure that you can connect to the instance so that you don't lose access.
2. Run the following command to use only the repository version:

   sudo systemctl stop sshd 
   sudo mkdir /usr/local/openssh-backup 
   sudo mv /usr/local/bin/ssh* /usr/local/openssh-backup/ 
   sudo mv /usr/local/sbin/sshd* /usr/local/openssh-backup/ 
   sudo dnf reinstall openssh openssh-server openssh-clients -y sudo systemctl daemon-reload 
   sudo systemctl enable sshd 
   sudo systemctl start sshd 
   sudo reboot

   Note: For AL2, replace dnf with yum.
3. After your computer reboots, run the following commands to verify that you use only the Amazon Linux repository:

   ssh -V 
   shd -V
   rpm -qa | grep openssh

Resolve vulnerabilities in bundled or service-specific packages

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Some packages are bundled within other tools, so you can't update them through standard package management commands. For example, the AWS CLI bundles its own Python and OpenSSL libraries.

To update bundled libraries, update the tool. For the AWS CLI, update the AWS CLI.

For packages that are specific to AWS services, such as Amazon CloudWatch Agent, run the following command to update the package directly:

sudo dnf update amazon-cloudwatch-agent

Note: Replace amazon-cloudwatch-agent with the service-specific package name. For AL2, replace dnf with yum.

If the latest available version still contains the reported CVE, then contact AWS Support to request an updated package.

Note: Amazon Corretto for Java doesn't use backports. You must manually update to the latest Corretto version to receive security patches. For available downloads, see Amazon Corretto Documentation.

Manage vulnerabilities with no available patch

For CVEs with the Pending Fix status, take the following actions:

• Monitor the Amazon Linux Security Center for updates.
• Use security groups and network access control lists (network ACLs) to minimize network exposure.
• Work with your security team to mitigate risk when you're waiting for the fix.

If you don't need the package for runtime operations, then temporarily uninstall the package. Before you remove the package, run the following command based on your Amazon Linux version to understand dependencies on the package.

AL2:

sudo yum deplist PACKAGE_NAME
rpm -q --whatrequires PACKAGE_NAME

AL2023:

sudo dnf repoquery --whatrequires PACKAGE_NAME

Note: Replace PACKAGE_NAME with your package name.

Then, run the following command to remove the package but not its unrelated dependencies:

sudo dnf remove --noautoremove PACKAGE_NAME

Note: Replace PACKAGE_NAME with your package name. For AL2, replace dnf with yum.

To check the transaction ID so that you can roll back changes if needed, run the following command:

sudo dnf history 

Note: For AL2, replace dnf with yum.

To undo a transaction, run the following command:

sudo dnf history undo transaction-id

Note: Replace transaction-id with the transaction ID. For AL2, replace dnf with yum.

For CVEs with the No Fix Planned status, check the Advisory for the reason and mitigation best practices.

Related information

Amazon Linux 2 FAQs

How do I make sure that my Amazon EC2 Linux instance is updated with the latest patches and security updates?