April 3, 2024

CVE-2024-3094: xz-utils (liblzma) Backdoor

We will continue to provide updates as more information becomes available.
Update - 2024-04-03 12:15 ET: Added information around Jia Tan persona theories.
Update - 2024-04-19 17:25 ET: Updated GitHub information; exploit trigger on specific private key.

This article is divided into two sections: an Executive Summary and Technical Details. The Executive Summary provides a high-level overview of the CVE (Common Vulnerabilities and Exposures) and recommends immediate actions to take. For practitioners seeking more in-depth, low-level context, the Technical Details section offers additional information.

Executive Summary

What Happened?

The xz-utils package, versions 5.6.0 and 5.6.1, has been identified as containing a backdoor in a compromised library dependency liblzma5. The presence of the backdoor potentially allows unauthorized access to affected systems through the manipulation of the sshd authentication process. This issue has been assigned CVE-2024-3094 and given a CVSS severity score of 10.0 Critical.

"Malicious code was discovered in the upstream tarballs of xz, starting with version 5.6.0. Through a series of complex obfuscations, the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code. This results in a modified liblzma library that can be used by any software linked against this library, intercepting and modifying the data interaction with this library"

Source: NIST

How Bad is This?

xz-utils is a XZ-format compression utility widely used across Linux distributions. The severity of the vulnerability could have posed a major threat, especially to Debian and Red Hat Linux based distributions. Fortunately, the backdoor was identified and reported by software engineer Andres Freund shortly after it was introduced into a small number of bleeding-edge Linux distributions, limiting potential impact.

While the impact was limited, affected systems are at critical risk. The vulnerability may allow a threat actor to compromise a system by sending a maliciously crafted payload to the SSH daemon (sshd). This payload could potentially grant unauthorized access to the targeted system.

The following versions of xz-util are impacted:

5.6.0
5.6.1

The following Linux distributions are affected:

The following stable Linux distributions have reported they are not affected:

*Note: Blumira Sensors are not affected.

What Should I Do?

System administrators are advised to immediately patch affected systems to mitigate this security risk, prioritizing those systems with publicly accessible SSH. The xz-utils 5.6.0 and 5.6.1 packages are considered untrustworthy. It is recommended to promptly apply package manager updates across all Linux and macOS systems to upgrade to trusted versions of xz-utils (i.e., versions before 5.6.0).

If you are using any affected systems in production, be sure to consult the guidance provided by the respective Linux Distribution. Continue to monitor impacted systems for unusual SSH activity.

Systems administrators can check the current version of xz-utils using the following command shared by @Kostastsale on Twitter:

for xz_p in $(type -a xz | awk '{print $NF}' | uniq); do strings "$xz_p" | grep "xz (XZ Utils)" || echo "No match found for $xz_p"; done

Technical Details

On Friday, March 29th 2024 Andres Freund sent an email to an open-source security (oss-security) mailing list to share with Open Source projects, distributors, researchers, and developers that they had found a backdoor in the liblzma library included in the xz-utils tarball used by their Debian sid systems. They discovered the backdoor after noticing SSH logins were using more CPU and generating valgrind errors. The email was also shared by Freund via Mastodon Social as well.

https://mastodon.social/@AndresFreundTec/112180083704606941

Freund first reported the issue to Debian's security team. They then reported the issue to the operating system distribution security contacts list (distros@) used by the oss-security mailing list. CISA was notified by one of the distributions. Red Hat later assigned the issue CVE-2024-3094. Finally the vulnerability was shared with the open-source security mailing list.

You can read more about the oss-security Mailing List Charter here. The list of the Linux distros included in the distros@ mailing group can be reviewed here.

Freund describes that one portion of the backdoor is in the distributed tarball for xz-utils versions 5.6.0 and 5.6.1. They point to a line of code for the xz-utils debian unstable branch that injects an obfuscated script to be executed at the end of xz-util configure. If the preconditions are met, the script modifies the liblzma makefile to insert the backdoor code.

The compromised files that contain most of the exploit are located at paths:

tests/files/bad-3-corrupt_lzma2.xz
 tests/files/good-large_compressed.lzma

Freund writes these files were initially added in 5.6.0 but are not actually used in any tests. He goes on to point out that the injected code caused valgrind errors (a debugging and profiling tool suite) and crashes in some configurations. Additional commits are seen in in xz-utils verson 5.6.1 that attempt to workaround these errors, including an adjusted exploit code.

Freund goes on to explain that logins with ssh became a lot slower once liblzma was backdoored. They reference before and after timestamps, namely a login that previously took 0.299 seconds now took 0.807 seconds. He writes, "openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma." From this we can better understand how the compromized liblzma library can result in unauthenticated access to a system via the sshd daemon used by SSH.

Finally, Freund describes their analysis of the compromised code in granular detail. Ultimately, the malicious code modified ifunc resolvers that are resolved during startup resulting in a modified sshd authentication process. The login slowdown is attributed to symbols being parsed in memory, notably liblzma's symbols being parsed before the main sshd binary. Another notable characteristic is the injected code appears to wait for the RSA_public_decrypt symbol in order to modify it so that it redirects to the backdoor code.

Freund confirms that when they attempted to login with a public key they observed the exploit code execute and then normal authentication processes resume. Freund hypothesizes, "I have not yet analyzed precisely what is being checked for in the injected code, to allow unauthorized access. Since this is running in a pre-authentication context, it seems likely to allow some form of access or other form of remote code execution." He then urges any vulnerable systems be upgraded immediately.

Brief History on Git Commits and Lobbying for Distribution

We should avoid speculation on the open-source maintainers involved in the project. Open-source contributors generously provide libraries that are widely used in software and operating systems, often without compensation. Their work is altruistic, built on collaboration, and countless hours of dedication. To pass judgement against open-source contributors is not an effective use of our time and will do little to secure our systems. Law enforcement is better equipped to identify and hold accountable those responsible for any malicious actions.

This all being said, examining the facts around how a popular open-source utility got backdoored and very nearly made its way into major Linux distributions serves as a valuable cautionary tale. By understanding the details of this incident, we can better protect against future supply-chain attacks targeting open-source code and the critical infrastructure that relies upon it.

The backdoored xz-util tarballs were signed by Jia Tan (JiaT75). JiaT75 is also the author of the exploit code introduced in 5.6.0 and updated in 5.6.1. It is noted there is growing speculation that Jia Tan could potentially be a persona used by group of people, potentially a nation-state sponsored team.

Timeline Overview

From 2005 to 2008 Lasse Collins along with a small group of developers work to create the XZ file format that uses the LZMA compression algorithm. Over time the format becomes popular and is used to efficiently compress things like tar files and Linux kernel images.

2021-10-29: JiaT75 sends their first patch to the xz-devel mailing list.

2022-05-19: Lasse Collins responds to an email complaint apologizing for slow response and shares that they may being handing off responsibilities to Jia Tan in an effort to better support xz-utils maintainence and response times.

2022-11-30: Jia Tan is officially identified by Lasse Collin as a xz-utils maintainer.

2023-06-22: The hook used by the backdoor is introduced by user Hans Jansen. It is speculated that this account is not a real user because the account returns later to promote the backdoored xz-utils versions and otherwise does not exist on the internet.

2023-07-07: A suspicious commit is made by JiaT75 to a popular fuzzing library oss-fuzz to disable ifunc support that would help prevent the exploit from being discovered.

2024-02-23: JiaT75 merges the backdoor code inside "test" files.

2024-02-24: xz-utils v5.6.0 is released.

2024-02-26: xz-utils v5.6.0 is added to Debian unstable.

2024-03-08: JiaT75 commits updates to the exploit to fix valgrind errors showing up in unstable Debian and beta Red Hat distributions.

2024-03-09: JiaT75 publishes xz-utils v5.6.1.

2023-03-24: JiaT75 removes details around vulnerability reporting from the SECURITY.md file typically used by researchers to provide vulnerability report guidance.

2024-03-25: Hans Jansen returns to advocate for an xz-utils update to v5.6.1 in Debian.

2023-03-28: JiaT75 advocates for xz-utils update to v5.6.1 in Ubuntu.

2024-03-29: Andres Freund discloses the vulnerability to Debian and the oss-security distros@ distribution list at OpenWall. RedHat issues CVE-2024-3094

Response from xz-utils maintainer

In response to CVE-2024-3094 GitHub has disabled the xz-utils repository. The GitHub accounts for Jia Tan (JiaT75) ~~and Lasse Collin (Larhzu)~~* have been suspended, as reported by Lasse Collin in a post on The Tukaani Project site last updated at 2024-04-02 21:44:17 +0300.

A warning message from GitHub that the repository has been disabled due to a violation of terms of service.

* UPDATE - 2024-04-19: Lasse Collin's GitHub account and the GitHub page for xz have been reinstated.

Lasse Collin, who has maintained xz-utils since around 2009, shared that they plan to write an article on how the backdoor got into the releases and what can be learned from the situation. You can read more updates from Collin in post linked here.

Response from the Security Community

It is still early days into a complex vulnerability that made its way to bleeding-edge Linux distributions. The security community is continuing to analyze the backdoor. We can expect additional details to become available in the coming weeks. As major details are discovered this article will be updated accordingly.

UPDATE - 2024-04-19: In order to utilize the backdoor a specific private key must be used to authenticate to sshd that has been compromised by the backdoored liblzma library.

https://bsky.app/profile/filippo.abyssdomain.expert/post/3koyhqjgcke2k

So far researchers have reported that successful exploitation does not generate any log entries (reported here and here). Consequently, the public detections made available for this vulnerability so far take a vulnerability management approach by identifying outdated liblzma library versions 5.6.0 and 5.6.1 being run by sshd. This would not detect successful exploitation, but rather identify an asset that is vulnerable to CVE-2024-3094.

Security researchers are asking themselves if there are other libraries that may have or could be compromised in a similar manner, and if that were to happen how could one identify them.

If you'd like to read more about this vulnerability in greater detail, check out the following blog posts and articles around CVE-2024-3094:

Recreating the Vulnerability in a Test Environment

The following guides, tools, and repositories are available if you'd like to explore the CVE in a lab setting:

How Blumira Can Help

It’s nearly impossible for admins to track every vulnerability, but Blumira’s security experts perform threat hunting on your behalf and develop detections in real time to protect your environment.

Blumira’s Free SIEM is easy to deploy; IT and security teams can start seeing immediate security value for their organizations.

Sign up for free and connect to your Microsoft 365 environment in minutes to start detecting and mitigating exposure related to Windows vulnerabilities.

Tag(s): Security Alerts , Blog , CVE