Is Ubuntu Withholding Security Patches for Some Software?

Update 2023-11-14: While I still support my previous statements, they proved to be a bit controversial. Therefore, I wanted to elaborate.

My main intention was to highlight the fact that many users are unaware of the security differences between the "main" and "universe" repository components in Ubuntu. When I performed a clean install, I found that the "universe" component is enabled by default. This default setting means that users can install software not supported by Ubuntu through a simple apt install and no warning is provided by Canonical. It is inevitable that a vulnerability will be discovered in one of the "universe" packages, which, by policy, would leave these end-users vulnerable. If these users encounter the potentially confusing "Get more security updates" message during an apt update, they are unlikely to understand its implications. Furthermore, the fact that the "universe" component is enabled by default suggests it is an integral part of the operating system on which most people rely.

There is further commentary at the end of the post.

Note: I am not affiliated with Canonical.

Recently, IntelTechniques, an organization known for publishing books and training materials on "Privacy, Security, and Open Source Intelligence," released a post cautioning against subscribing to Ubuntu Pro service. Their experiments with Ubuntu led them to falsely believe that software installed using Apt was 'up-to-date,' despite producing a security warning.

Quoting from their post:

The [Ubuntu packages] "Available version" is the exact same product as the currently installed software. The update does nothing.

This statement reflects broader misunderstandings about package management. Given the post's prominence on the front page of Hacker News, I decided I would take some time to clarify.

Are Security Patches Withheld from Mainstream Ubuntu?

Officially supported software is found in the "main" repository and receives automatic security updates. For example, nginx, a widely-used web server, and python3, a popular programming language runtime, are part of this repository. In contrast, the "universe" component relies on community support. This means that anyone who joins the packaging mailing lists can submit packages to this repository. These submissions are overseen by community maintainers rather than by Canonical directly. The quality of community contributions varies, and sometimes security updates in point releases are delayed or missed. Often, security updates are intertwined with feature updates, which can lead to issues in downstream applications.

Ubuntu Pro, therefore, aims to provide security updates specifically for the "universe" component, extracting and applying these patches to the affected software. A cursory review indicates that the most critical packages, particularly those that are web-facing and crucial for businesses, are in the "main" repository. As of this writing, some notable packages in the "universe" repository include ffmpeg (a command-line tool and its libraries), kodi, and lighttpd. For example, the current version of ffmpeg in use has known vulnerabilities. In my estimation, the most likely way that this could be exploited is that a web application, such as a file converter running on Ubuntu, uses the community-maintained ffmpeg library

ffmpeg: A Case Study

In their original post, IntelTechniques highlighted ffmpeg as an example. To understand the changes brought by Ubuntu Pro, I first installed ffmpeg, then upgraded to Ubuntu Pro and updated my system. This update demonstrated that ffmpeg had indeed received ESM patches.

Initially, the command installed version of ffmpeg is as follows:

[#] sudo apt show ffmpeg
Package: ffmpeg
Version: 7:4.4.2-0ubuntu0.22.04.1
...

After upgrading to Ubuntu Pro:

sudo apt show ffmpeg
Package: ffmpeg
Version: 7:4.4.2-0ubuntu0.22.04.1+esm2
...

Ubuntu Pro introduces additional sources to Apt, deriving from their ESM repository. To access the source code used for building this ESM version, one can uncomment the source repository in deb-src located in /etc/apt/sources.list.d/ubuntu-esm-apps.list. Following this, we can obtain the ffmpeg source:

sudo apt update
sudo apt install dpkg-dev
sudo apt-get source ffmpeg

Ubuntu and Debian utilize a source management system named quilt. Patches applied to the package, which are not from the original authors, are found under ffmpeg-4.4.2/debian/patches/series. Examining this file reveals the CVEs addressed:

cat ffmpeg-4.4.2/debian/patches/series

0001-avcodec-arm-sbcenc-avoid-callee-preserved-vfp-regist.patch
0002-configure-arm-Don-t-add-march-to-the-compiler-if-no-.patch
CVE-2022-3109.patch
CVE-2022-3341.patch
CVE-2022-3964.patch
CVE-2022-48434.patch

Each file represents a commit addressing a specific vulnerability. (At least this is what Canonical intends, I haven't reviewed these fixes.) Referring to the security section of FFmpeg's website reveals that these are the CVEs identified since the release of 4.4.2. Consequently, it appears that Canonical is actively patching these universe packages and offering them to enterprises.

A Scare Tactic?

The author originally included this image in their post writing:

This warning appears concerning as it insinuates that some updates are being withheld from your machine unless you subscribe to the Pro service.

If you encounter a warning like this, it indicates that there are universe packages on your system which have received security updates through Ubuntu Pro. Clearly, this message is designed to encourage users to upgrade to Ubuntu Pro. For non-enterprise users, upgrading is a sensible choice as it is free and offers significant security enhancements. Business users encountering this message should evaluate whether any of this software is web-facing or processes untrusted inputs. If that's the case, it's likely that they need to apply some of these fixes.

Is This Fair?

Canonical essentially repackages security patches developed by the open-source community and markets them as a part of their support services, which can be a point of contention for some. However, it's important to maintain perspective on this issue. Package management is a complex task, requiring maintainers to learn specific tools and navigate what is often referred to as dependency hell. This process can involve creating bespoke patches for a particular system, supporting older language runtimes, or dealing with outdated compilers, all while ensuring not to disrupt an LTS release. At times, it necessitates collaboration with upstream maintainers. The open-source landscape is diverse, with different organizations releasing updates in various ways. Updating a package can inadvertently break dependencies for downstream users, even if the change seems benign. Canonical plays a crucial role in mitigating these issues for users of open-source software. Business users encountering this message should assess whether the software in question is web-facing or handles untrusted inputs. It's my belief that this approach will make open-source a safer choice for businesses, encouraging them in turn to contribute fixes and features back to these projects.

Could Rolling Releases Be More Secure?

In the area of package management, two predominant approaches exist: rolling releases and point releases. Rolling releases deliver continuous updates to packages as soon as they become available from upstream maintainers. Point releases, on the other hand, focus on stabilizing major package versions to prevent incompatible updates that could disrupt user code or other tools. While rolling releases are often criticized for potentially breaking the user experience — a rare but genuine concern in my view — there's an overlooked aspect in this debate. Stabilizing a release necessitates that the packaging team selectively extracts security fixes from the version control history, a process inherently prone to errors. This also presupposes that maintainers are adept at identifying and isolating security vulnerabilities. Although many of these issues are cataloged in the CVE database, numerous security vulnerabilities initially emerge as simple crashes. It's plausible that many fixes escape CVE reporting and remain unnoticed.

The lack of backporting and the management of multiple releases make rolling releases considerably easier to maintain. Taking ffmpeg as an example again, Debian's unstable version is currently free of known vulnerabilities, thanks to its upgrade to the latest release. However, I haven't seen any formal research comparing the two approaches.

Update 2023-11-14:

Regarding the more contentious issue: to some people, Ubuntu Pro simply adds additional security by patching known CVEs in community repository. Conversely, some view this as Ubuntu putting crucial security patches behind a paywall, obliging enterprises to financially support their product. These aren't mutually exclusive views. In my view, this situation creates an inherent conflict of interest. Canonical's value proposition rests on the fact some community software is vulnerable and that they can pluck the patches from upstream for this software. By providing these patches to non-enterprise users, they also support the broader community, thus mitigating this conflict. Additionally, by supporting Ubuntu's development, the revenue generated might benefit the open-source ecosystem as a whole.

Changed "Yes, Ubuntu Withholding Security Patches for Some Software" to "Is Ubuntu Withholding Security Patches for Some Software?" Although this is indeed true, and it is important for people to understand that the 'secure by default' principle does not apply here, my phrasing was interpreted by many as suggesting malice, which was not my intention.