Are simple hash functions good enough?

Hash functions are critical components of almost every computer program and a basic building block of data structures. They are used to retrieve data, perform fast similarity searches, implement caches, route network traffic, count objects, to name just a few applications. All of these applications rely on a property of some hash functions: that they map inputs to a set of outputs in a uniform manner. For instance, associative arrays - often called hash maps, dictionaries, or unordered maps by software engineers - rely on a hash function that uniformly maps keys to a series of 'slots' which store information about the values. If a hash function is too biased, it can cause the program to revert to slow collision resolution algorithms - often making computations infeasible.

In practice, most hash functions are designed to distribute data uniformly at random across a codomain, say of length $n$, so that for any key $k$ chosen at random from the domain will have a probability $\frac{1}{n}$ of mapping to each output value. Take, for example, Daniel J. Bernstein's djb2_32 hash:

use std::num::Wrapping;

fn djb2_32(bytes: &[u8]) -> u32 {
    let mut hash = Wrapping(5381);

    for &b in bytes {
        hash = (hash << 5) + hash + Wrapping(b as u32);
    }

    hash.0
}

This forms the relation:

$$h_m = \left( 33 \; h_{m-1} + b_{m-1}\right) \mod 2^{32}, \; h_0 = 5381$$

While extremely simple, this hash function is actually quite clever. It takes the form of a linear-congruential generator (LCG), applying an affine transformation followed by a modulus. We can view this in two ways. Say we consume $m$ bytes; our sequence will take the form:

$$h_m = (33^m h_0 \mod 2^{32} + 33^{m-1} b_0 \mod 2^{32} + \cdots + 33 b_{m-2} \mod 2^{32} + b_{m-1} \mod 2^{32}) \mod 2^{32}$$

Which, in effect, means that $h_m$ is the sum of $m$ different LCGs modulo $2^{32}$. From another perspective, say we consume a sequence of fixed bytes $b_m = B$. In this scenario, if $B$ is not 0, 1, 2, or 33, the recurrence would satisfy the Hull-Dobell Theorem and would form an LCG with a period greater than $2^{32}$. (Doesn't your linear-congruential generator satisfy the Hull-Dobell Theorem?) I imagine this was the reasoning behind choosing these specific coefficients.

While there are many non-cryptographic hash functions available, a number of fast non-cryptographic hash functions have been designed in the past decade. Some recent examples include Murmur, CityHash, XXHash, and t1hash. These functions claim superior performance in terms of both speed and quality, although they come with a complexity trade-off. This is because (1) they are often architecture-specific, (2) some perform unaligned accesses, and (3) they often require language-specific FFI bindings. Many benchmarks claim that simple hash functions, like fnv1a, have "serious quality issues." However, many of these benchmarks are also unrealistic, involving the construction of worst-case key pairs and ensuring there are no patterns in the hash outputs. So, I wanted to answer the question: Do any of these simple hash functions break down on real-world datasets? If so, what are their failure modes? To do this, I designed two tests that simulate real-world use cases and tested a number of hash functions across three datasets.

Hash Functions Under Test

I gathered some simple "low quality" hash functions as well as some "high quality" hash functions. These include:

Low quality hash functions:

• adler32: Mark Adler's version of the Fletcher checksum. adler32 is considered unreliable for short inputs, as per RFC 3309.
• djb2_32: A simple non-cryptographic hash devised by Daniel J. Bernstein.
• fnv1a32: A widely-used hash designed by Glenn Fowler, Phong Vo, and Landon Noll.

High quality hash functions:

• spooky32: A hash function designed by Bob Jenkins.
• murmur3: A hash function designed by Austin Appleby in 2008.
• city32: A fast hash function developed by Google.
• xx32: Claims to be the fastest x86 non-cryptographic hashing algorithm.

Some of these hashes also have 64-bit variants including city, xx, spooky, fnv1a, and djb2.

Datasets

I wanted to test each hash function on a variety of large datasets. These provide sample scenarios from networking, bioinformatics, and natural language processing. These include:

• All words in the English, German, and French languages as provided by the GNU ASpell dictionary version 2.1.
  • The French dictionary contains 221,377 words.
  • The American English dictionary contains 123,985 words.
  • The German dictionary contains 304,736 words.
• All possible private IPv4 addresses, as unsigned bytes in network byte order.
  • This includes 17,891,328 addresses.
  • Most addresses are continuous, differing in a single bit.
• All unique 12-mers (or 12-length substrings) in the human genome (e.g. all contigs from GRCh38.)
  • For example, 'AAGAGTCAGTTATT' is a 12-mer.
  • Comprising 203,091,438 unique 12-length substrings from the human genome. These sequences cover most of the possible combinations in the genome's four-character alphabet (A, T, C, G).
  • This dataset is challenging to hash because the differentiating information is contained in a small subset of the input bits.
  • I did not canonicalize these k-mers so they could be drawn from the 5' 3' or 3' 5' strands.

Multinomial Non-Uniformity Test

In practice, most hash functions are used to associate an item with a specific 'slot' in memory, and many algorithms depend on the premise that the distribution of items across these slots is no worse than that that could be produced by a uniform random distribution. This test is unique in modeling real-world behavior of the hash function rather than the behavior under a synthetic benchmark. Since the ranges of the hash functions are large (i.e., either $2^{32}$ or $2^{64}$), we need to choose a function that maps these to our slot index, $b_i$. This is most commonly accomplished by taking the modulus of the output with the number of slots. After hashing $k$ items, the resulting distribution should be modeled by a multinomial across the $n$ slots. On average, $\frac{k}{n}$ values should hash to each slot and we can use the $\Chi^2$ distribution to test if the distribution differs significantly from the distribution that would be produced by a random hash function. By assuming $n$ is sufficiently large, we can compute the test statistic using the following formula:

$$\Chi^2 = \sum_{i = 0}^{n-1} \frac{(b_i - \frac{k}{n})^2}{\frac{k}{n}}$$

Then, compute the $p$ value using the chi-squared CDF. I performed around 60 of these tests across all the datasets, so I'll only list a few here.

German Word List

$|b| = 1024$

$|b| = 1031$

$|b| = 435337 \approx 0.7 n$

With the exception of adler32, all the hash functions hold up well against these ASCII inputs. When the number of slots is prime and the table size is small, adler32 performs at its best. I think that's likely because the sum wraps around the modulus, creating something closer to a uniform distribution, though this does not necessarily mean it should be used.

Private IP Ranges

$|b| = 65536$

$|b| = 25559057 \approx 0.7n$

This is likely the most challenging test of the three due to the fact many of these IPs are differentiated by single bits. Both adler32 and djb2_32 fail. In particular, adler32 hash function only distributes hashes amongst 1% of the allocated buckets! Interestingly enough, for $|b| = 65536$, fnv1a seems to distribute the values very uniformly. In expectation, the 99th percentile should approach 312; interestingly, this doesn't happen for fnv1a. (I could probably run the 17712414th order statistic to find if this is significant, but that seems like a bit of a nightmare.)

All k-mers in GRC H38

$|b| = 65536$

$|b| = 290130625 \approx 0.7n$

The k-mer test seemed to induce failures in all the 32-bit values. While we can say these statistically differ from the uniform distribution, this does not mean it will impact the performance of our application significantly. It actually seems to be fairly well distributed, at least from the 50th, 75th, and 99th percentiles.

Sparse Collisions Test

While the non-uniformity test is simple to administer, interpreting its results can be challenging due to the fact you have to compare across distributions. This motivated me to develop a test to characterize the likelihood of observing a certain number of hash collisions throughout the entire data set. The "Sparse Collisions Test" is simple, and it operates by hashing all the keys (for example, all the words in the German language) and counting the number of collisions. The real challenge lies in determining whether the number of collisions we measure is significant. Finding the likelihood of observing $q$ collisions when $k$ values are hashed is a variation on the famously unintuitive Birthday Problem.

Characterizing the full distribution for each scenario proved difficult, and I believe there might not be a closed-form formula without approximation. After considerable effort, I was able to develop a formula to compute the likelihood of a specific number of collisions. This operated by summing a combinatorial formula over all partitions of the input space. Using dynamic programming, the exact distribution can be computed in $O\left( k^k \right)$ time and $O\left( k \right)$ space. This is only practical for small inputs. Fortunately, by limiting the space of partitions considered and eliminating those which would almost certainly would not occur, I was able to make more progress. In the end, I was able to categorize the expected number of collisions within the private IP address space and German word list for the 32-bit variants with an error on the order of $10^{-8}$. Further information about how the distribution was derived will be included in an appendix.

German Word List

The expected probability distribution can be computed using from the partitions formula:

For 32-bit hash functions, we should expect fewer than 22 collisions at $p=0.001$, a criterion that only adler32 fails to meet. The 64-bit hash functions can be bounded by the Birthday Problem, accordingly we expect that no collisions occur and any number of collisions are statistically significant at the 0.001 level. Thus, we can say djb2_64 also differs significantly from a random hash function.

All Private IP Addresses

The expected probability distribution can be computed using from the partitions formula:

For the 32-bit hash function, we would expect fewer than 37,812 collisions at $p=0.001$. As in the previous test, any collisions for the 64-bit hashes are significant at the 0.001 level. So for this test, djb2_32, adler32, and djb2_64 perform significantly worse than what would be expected from a random hash function. On the other hand, fnv1a_32, xx32, and murmur3 actually perform significantly better than what would be expected from a random hash function. city32 and spooky32 perform in line with our expectations.

Unique K-mers in GRCh38

With my current methods, I can't compute for the expected probability distribution $k=203091438$. It's too computationally expensive. I ran the tests anyway so I could list the results.

Conclusion

After conducting all these experiments, my biggest takeaway is that hash benchmarking suites are probably not measuring real hashing performance. In these tests, fnv1a, a simple hash function from the early 90s, held up remarkably well. While I think measuring the randomness of hash functions is interesting both theoretically and as a fun engineering exercise, I believe these hyper-optimized hash functions offer very marginal benefits. Of course, I am open to changing my mind. This would happen if I am presented with a real-world dataset that elicits bad behavior from a simple hash function like fnv1a. There might be some dataset for which city and spooky outperform their simpler predecessors. You can't really prove that these hash functions are "good"; you can only show that under certain situations they perform poorly.

Many early hash functions like adler32 and djb2 were designed in an era when hashing performance was an important consideration, and they were typically used for specific applications. adler32 was used in gzip, where entropy was abundant. This accounts for its significant shortcomings with short string inputs. I believe djb2 was designed for ASCII strings. ASCII data, like German and English words, contains a lot of inherent entropy, meaning that weaker hash functions like djb2 perform quite well. The main issue with djb2 is that the prime does not provide avalanching over the entire output space. Replacing 33 with a better prime, like 22695477, considerably boosts its performance. I think the reason Bernstein used 33 is that he designed it in the 90s when computing resources were limited. The multiplication operation could be replaced with a bit shift and addition.

Appendix

Deriving the Expected Collision Distribution

In order to characterize the collision distribution, we want to obtain the probability that $q$ collisions occur, $P(Q=q)$, for an idealized random hash function.

Let us define the hash function over some alphabet, $\Sigma$. This hash function maps an arbitrary input to one of the $n$ slots, that is, $f: \Sigma^\mathbb{N} \to [1, n]$. Each input has a $\frac{1}{n}$ probability of mapping to each output. We are interested in the probability that $q$ collisions occur within a set of $k$ values.

The distributions of hashes over the slots are a multinomial distribution since the number of trials is fixed, the trials are independent, and there is a fixed probability $p_i = \frac{1}{n}$ that they hash within each bucket. Therefore, the probability that a specific distribution of slot counts $b_0, b_1, \cdots, b_{n-1}$ is given by $\frac{k!}{\prod_{i=0}^{n-1} b_i!}$ where $\sum_{i=0}^{n-1} b_i = k$. We could evaluate this by considering all possible distributions of values in the buckets, summing the probabilities of each distribution that contributes to $q$ collisions. However, many of these are duplicative. For example, if $n = 2$ and $k = 3$, $b_0 = 1, b_1 = 3$ and $b_0 = 3, b_1 = 1$ occur with equal likelihood. Thus, we can compute the probability of achieving any set of bucket counts $c_0, c_1, \cdots, c_k$ by multiplying the probability of this outcome by the number of ways in which it can occur:

$$P\left(c_0, c_1, \cdots, c_k\right) = \frac{ \left( \sum_{j = 0}^k c_j \right) ! }{ \prod_{i = 0}^{k} c_i! } \frac{k!}{n^k \prod_{i=0}^{i=k} i! ^{c_i}}$$

To calculate the probability that $q$ collisions occur, this needs to be summed over all partitions of $k$. That is, all natural numbered coefficients $c_1, c_2, \cdots c_k$ which satisfy $k = \sum_{i=1}^k i \cdot c_i$. The number of collisions is given by $q = \sum_{j=2}^{k} c_j (j - 1)$.

I would have liked to obtain a closed-form formula, even via an approximation. But, there is no known closed-form formula for partitions. If anyone knows of an appropriate approximation, let me know.

Computing the Expected Collision Distribution

The equation given above can be computed efficiently using a few approximations. First, factorials can be approximated through the use of the log gamma function with 16-bit floating point accuracy. This is provided by the Lanczos Gamma Approximation. The log gamma values for $[0, k] \cup [n-k, n]$ can be cached to make these calls $O(1)$. This expected collision distribution can be computed through a depth-first search over the partition space. Unfortunately, partitions grow exponentially. For example, there are around $10^{60}$ possible partitions for the German Word List dataset. Many possible outcomes have near-zero likelihoods of occurring. For example, the likelihood that all $k$ values hash to the same bucket is $(\frac{1}{n})^{k-1}$. Near-perfect approximations can be obtained by limiting the depth of the search and the number of partitions at a given depth.

Additional Results

I have made all my results, as well a the program I used to compute the collision distributions, available in a Git repository.

Is Ubuntu Withholding Security Patches for Some Software?

Update 2023-11-14: While I still support my previous statements, they proved to be a bit controversial. Therefore, I wanted to elaborate.

My main intention was to highlight the fact that many users are unaware of the security differences between the "main" and "universe" repository components in Ubuntu. When I performed a clean install, I found that the "universe" component is enabled by default. This default setting means that users can install software not supported by Ubuntu through a simple apt install and no warning is provided by Canonical. It is inevitable that a vulnerability will be discovered in one of the "universe" packages, which, by policy, would leave these end-users vulnerable. If these users encounter the potentially confusing "Get more security updates" message during an apt update, they are unlikely to understand its implications. Furthermore, the fact that the "universe" component is enabled by default suggests it is an integral part of the operating system on which most people rely.

There is further commentary at the end of the post.

Note: I am not affiliated with Canonical.

Recently, IntelTechniques, an organization known for publishing books and training materials on "Privacy, Security, and Open Source Intelligence," released a post cautioning against subscribing to Ubuntu Pro service. Their experiments with Ubuntu led them to falsely believe that software installed using Apt was 'up-to-date,' despite producing a security warning.

Quoting from their post:

The [Ubuntu packages] "Available version" is the exact same product as the currently installed software. The update does nothing.

This statement reflects broader misunderstandings about package management. Given the post's prominence on the front page of Hacker News, I decided I would take some time to clarify.

Are Security Patches Withheld from Mainstream Ubuntu?

Officially supported software is found in the "main" repository and receives automatic security updates. For example, nginx, a widely-used web server, and python3, a popular programming language runtime, are part of this repository. In contrast, the "universe" component relies on community support. This means that anyone who joins the packaging mailing lists can submit packages to this repository. These submissions are overseen by community maintainers rather than by Canonical directly. The quality of community contributions varies, and sometimes security updates in point releases are delayed or missed. Often, security updates are intertwined with feature updates, which can lead to issues in downstream applications.

Ubuntu Pro, therefore, aims to provide security updates specifically for the "universe" component, extracting and applying these patches to the affected software. A cursory review indicates that the most critical packages, particularly those that are web-facing and crucial for businesses, are in the "main" repository. As of this writing, some notable packages in the "universe" repository include ffmpeg (a command-line tool and its libraries), kodi, and lighttpd. For example, the current version of ffmpeg in use has known vulnerabilities. In my estimation, the most likely way that this could be exploited is that a web application, such as a file converter running on Ubuntu, uses the community-maintained ffmpeg library

ffmpeg: A Case Study

In their original post, IntelTechniques highlighted ffmpeg as an example. To understand the changes brought by Ubuntu Pro, I first installed ffmpeg, then upgraded to Ubuntu Pro and updated my system. This update demonstrated that ffmpeg had indeed received ESM patches.

Initially, the command installed version of ffmpeg is as follows:

[#] sudo apt show ffmpeg
Package: ffmpeg
Version: 7:4.4.2-0ubuntu0.22.04.1
...

After upgrading to Ubuntu Pro:

sudo apt show ffmpeg
Package: ffmpeg
Version: 7:4.4.2-0ubuntu0.22.04.1+esm2
...

Ubuntu Pro introduces additional sources to Apt, deriving from their ESM repository. To access the source code used for building this ESM version, one can uncomment the source repository in deb-src located in /etc/apt/sources.list.d/ubuntu-esm-apps.list. Following this, we can obtain the ffmpeg source:

sudo apt update
sudo apt install dpkg-dev
sudo apt-get source ffmpeg

Ubuntu and Debian utilize a source management system named quilt. Patches applied to the package, which are not from the original authors, are found under ffmpeg-4.4.2/debian/patches/series. Examining this file reveals the CVEs addressed:

cat ffmpeg-4.4.2/debian/patches/series

0001-avcodec-arm-sbcenc-avoid-callee-preserved-vfp-regist.patch
0002-configure-arm-Don-t-add-march-to-the-compiler-if-no-.patch
CVE-2022-3109.patch
CVE-2022-3341.patch
CVE-2022-3964.patch
CVE-2022-48434.patch

Each file represents a commit addressing a specific vulnerability. (At least this is what Canonical intends, I haven't reviewed these fixes.) Referring to the security section of FFmpeg's website reveals that these are the CVEs identified since the release of 4.4.2. Consequently, it appears that Canonical is actively patching these universe packages and offering them to enterprises.

A Scare Tactic?

The author originally included this image in their post writing:

This warning appears concerning as it insinuates that some updates are being withheld from your machine unless you subscribe to the Pro service.

If you encounter a warning like this, it indicates that there are universe packages on your system which have received security updates through Ubuntu Pro. Clearly, this message is designed to encourage users to upgrade to Ubuntu Pro. For non-enterprise users, upgrading is a sensible choice as it is free and offers significant security enhancements. Business users encountering this message should evaluate whether any of this software is web-facing or processes untrusted inputs. If that's the case, it's likely that they need to apply some of these fixes.

Is This Fair?

Canonical essentially repackages security patches developed by the open-source community and markets them as a part of their support services, which can be a point of contention for some. However, it's important to maintain perspective on this issue. Package management is a complex task, requiring maintainers to learn specific tools and navigate what is often referred to as dependency hell. This process can involve creating bespoke patches for a particular system, supporting older language runtimes, or dealing with outdated compilers, all while ensuring not to disrupt an LTS release. At times, it necessitates collaboration with upstream maintainers. The open-source landscape is diverse, with different organizations releasing updates in various ways. Updating a package can inadvertently break dependencies for downstream users, even if the change seems benign. Canonical plays a crucial role in mitigating these issues for users of open-source software. Business users encountering this message should assess whether the software in question is web-facing or handles untrusted inputs. It's my belief that this approach will make open-source a safer choice for businesses, encouraging them in turn to contribute fixes and features back to these projects.

Could Rolling Releases Be More Secure?

In the area of package management, two predominant approaches exist: rolling releases and point releases. Rolling releases deliver continuous updates to packages as soon as they become available from upstream maintainers. Point releases, on the other hand, focus on stabilizing major package versions to prevent incompatible updates that could disrupt user code or other tools. While rolling releases are often criticized for potentially breaking the user experience — a rare but genuine concern in my view — there's an overlooked aspect in this debate. Stabilizing a release necessitates that the packaging team selectively extracts security fixes from the version control history, a process inherently prone to errors. This also presupposes that maintainers are adept at identifying and isolating security vulnerabilities. Although many of these issues are cataloged in the CVE database, numerous security vulnerabilities initially emerge as simple crashes. It's plausible that many fixes escape CVE reporting and remain unnoticed.

The lack of backporting and the management of multiple releases make rolling releases considerably easier to maintain. Taking ffmpeg as an example again, Debian's unstable version is currently free of known vulnerabilities, thanks to its upgrade to the latest release. However, I haven't seen any formal research comparing the two approaches.

Update 2023-11-14:

Regarding the more contentious issue: to some people, Ubuntu Pro simply adds additional security by patching known CVEs in community repository. Conversely, some view this as Ubuntu putting crucial security patches behind a paywall, obliging enterprises to financially support their product. These aren't mutually exclusive views. In my view, this situation creates an inherent conflict of interest. Canonical's value proposition rests on the fact some community software is vulnerable and that they can pluck the patches from upstream for this software. By providing these patches to non-enterprise users, they also support the broader community, thus mitigating this conflict. Additionally, by supporting Ubuntu's development, the revenue generated might benefit the open-source ecosystem as a whole.

Changed "Yes, Ubuntu Withholding Security Patches for Some Software" to "Is Ubuntu Withholding Security Patches for Some Software?" Although this is indeed true, and it is important for people to understand that the 'secure by default' principle does not apply here, my phrasing was interpreted by many as suggesting malice, which was not my intention.

Choosing an Authenticator: Who Can You Trust?

Recently, I found myself on the hunt for a 2FA authenticator app. I realized that in-depth reviews were scarce, if not nonexistent. So, I did some digging and was surprised at what I found.

Background

Two truisms persist in password security: humans are bad at generating strong passwords, and corporations are equally bad at safeguarding these fragile human creations. In an era where data breaches have become daily news items, mistakes in companies' products expose consumer information, including passwords, to hackers and sleuths. Many companies "hash" their passwords — a method by which passwords are scrambled before they are stored to make them computationally impracticable to reverse. Now, if companies adhered to strong hashing practices only those with particularly weak passwords would be at risk. However, the effectiveness of hashing relies on both the user's password strength and the available computational power. As technology advances, what was once considered unbreakable can become vulnerable.

Fortunately, ingenious cryptographers have been hard at work developing stronger hash functions to counterbalance increasing computational power. Unfortunately, the vast majority of websites have yet to adopt these fortified mechanisms, leaving their hashed passwords susceptible to cracking. The situation is exacerbated by the common practice of reusing passwords across multiple platforms, allowing attackers to target victims' banks, credit accounts, and online stores.

In light of this fragile ecosystem — characterized by poor memorization, reuse, and frequent leaks — a secondary authentication layer has become essential to combat fraud and identity theft. Enter OTPs, or One-Time Passwords. These codes, either generated by an app on your phone or sent via text message, serve to confirm that you are, indeed, not a hacker located on a different continent. Most internet users will have received these codes via text message. While text messages are the default delivery method for many companies, this method has its drawbacks. Skilled hackers can circumvent it by calling your cellular service provider, impersonating you, and having your service redirected. Therefore, the more secure option is to use authenticator apps that typically employ TOTP, or Time-Based One-Time Passwords.

How TOTP Works

Here's a non-technical summary: every 30 seconds, your phone generates a unique code, usually six to eight digits long. When logging into a site, you provide this code. If entered correctly, you gain access; input a series of incorrect codes, and you'll find yourself temporarily locked out. Ideally, the lockout period increases with each successive incorrect attempt to deter brute-force attacks.

The beauty of the TOTP protocol lies in its simplicity, which is advantageous from a cryptographic perspective. Simplicity allows for easier scrutiny of the protocol's security features and straightforward implementation. A developer could grab their nearest crypto library and whip up a basic app within a few hours. However, this ease of creation is a double-edged sword. It enables anyone to develop an authenticator app, sometimes incorporating features that inadvertently weaken the overall security — either by mistake or by design.

The Lineup

So, which authenticator should you pick? I was hunting for an authenticator app that withstood scrutiny. Oddly enough, most reviews tended to omit two crucial details:

1. The actual effectiveness of the authentication process
2. The privacy profile of the application

Starting with the first point, not all authenticators are created equal. For instance, Latch stores all your authentication codes on a remote server, unencrypted. A single breach could jeopardize all of Latch's tokens and, by extension, your accounts. While any TOTP app offers an improvement over mere passwords, such practices expose users to unnecessary risk.

Secondly, TOTP apps function as repositories of sensitive information, including usernames, affiliated websites, and device data.

For this review, I've scrutinized four leading contenders: FreeOTP, Google Authenticator, Authy, and Duo. These apps were selected based on their cross-platform compatibility, popularity, and security merits, to my knowledge. The evaluations aim for broad appeal but will delve into technical nuances for those who are technically adept. Most insights on the security of these apps originate from the comprehensive paper "Security and Privacy Failures in Popular 2FA Apps" by Conor Gilsenan, Fuzail Shakir, and Noura Alomar, all of whom are affiliated with UC Berkeley. Independently, I also intercepted the communications between the apps and their servers to identify if they were collecting unnecessary information.

FreeOTP: Trading Features for Privacy and Security

You may be surprised to learn that, in principle, there's absolutely no reason that an authenticator needs to contact a centralized server. The TOTP protocol merely requires that the authenticator and server share a master code, which forms the basis for generating the rolling codes. Technically, you could even manually calculate these codes on paper, although this would not be practical.

Developed by Red Hat, FreeOTP is an open-source, cross-platform solution. The motivation behind its creation is somewhat opaque, as Red Hat primarily serves enterprise-level clients. Nonetheless, they've developed an app that forgoes bells and whistles, opting instead for maximal security and privacy. FreeOTP doesn't back up or sync tokens across devices; its minimalist approach makes it arguably the most secure software-based option, short of using dedicated hardware.

Using FreeOTP is largely intuitive, albeit with some awkward design choices — for instance, requiring a right-swipe to delete an old token. If you decide to adopt it, make sure you have a fallback 2FA strategy, such as storing printed recovery codes in a secure location or keeping a secondary 2FA device that you don't typically carry.

You might wonder how a 2FA app can run offline if it requires scanning a QR code. While the specification doesn't provide any guidance on how these codes are communicated with the client, Google Authenticator introduced the practice of encoding the master secret in QR codes. Scanning the code doesn't reach out to a website; instead, it uses the URL protocol to auto-redirect the QR scanner to the authenticator app. Even though QR codes weren't part of the original TOTP specs, they have become practically ubiquitous today.

Offline Backup and Syncing

TOTP (Time-based One-time Password) QR codes are inherently versatile; they don't store device-specific information. This universal feature allows you to pair multiple authenticators with the same QR code, bypassing the need for cross-device syncing. However, capturing screenshots or printing out these QR codes for backup isn't without risks. If you go this route, make sure to store them in a secure location, such as a locked drawer, wallet, or safe — never alongside your passwords. Should you lose or damage your phone, a quick rescan of these QR codes will restore your access. But proceed with caution: print these codes before scanning to ensure legibility. For the tech-savvy who can confidently remember a robust password, storing these screenshots in an encrypted VeraCrypt container is also a viable option.

Note: This approach is applicable only for TOTP tokens, not for the older HOTP (HMAC-based One-time Passwords) technology. HOTP tokens don't refresh automatically every 30 seconds but rather change upon use. Although most authenticators support HOTP, it's a fading standard due to the risk of de-synchronization when misused.

Google Authenticator: Convenient Syncing, Worse Security, Questionable Privacy

Until 2023, Google Authenticator served as a minimalist, offline solution for two-factor authentication. It was considered inherently privacy-friendly due to its lack of server communication and was even open-source. However, the introduction of an automatic cloud-syncing feature has dramatically undercut its security stance. This new feature syncs authentication codes across devices but crucially lacks end-to-end encryption.

The convenience is alluring, but it also represents a significant security flaw. Anyone who gains access to your Google Account can subsequently access all your authentication tokens. The point of vulnerability shifts from hacking your individual device to impersonating you to Google. This isn't mere speculation; as journalist Dan Goodin recently reported, Google Authenticator has already been exploited to amplify a security breach. Worse still, if Google's servers were ever compromised, all your tokens would be up for grabs. I suspect Google is aware of this Faustian bargain and recognizes that using OTP at all would be an advancement in security for most of their users.

Privacy is another casualty. With your account information and secret keys stored on Google's servers, you're entrusting Google not to misuse this data. Google could also be required to provide this data to authorities if served with a warrant. While in my limited testing I didn't detect any logging, the application is not open-source, so there may be data collection I didn't identify. Privacy-wise, I believe there are better options.

"Two-factor" or "One-factor" Authentication?

Google Authenticator and Microsoft Authenticator dominate the authenticator install base, comprising upwards of 80% of Android installs. Both of these authenticators offer automatic syncing features, which impose inherent risks. Access to your Google or Microsoft account essentially grants an intruder access to your underlying TOTP tokens. What's worrisome is that many websites offer account recovery through OTP tokens and email access, often linked to the same Google or Microsoft account. Users may believe they've employed two separate security layers, but in reality, they're relying on a single point of vulnerability — providing only a facade of enhanced security.

Duo: Cloud Backups Done (Mostly) Right

Duo, a product from corporate security company Duo Security, unsurprisingly gets a lot right — especially when it comes to cryptographic primitives for its cloud backups. The app strikes a balance between user-friendliness and robust features. It offers cloud backup capabilities, enabling your 2FA tokens to be securely stored in iCloud on iOS devices and Google Drive on Android. The 2FA tokens are strongly encrypted and require a password to recover them from the backup. Lost your password? Tough luck. This app doesn't use a child lock. Choose a strong and unique password. It should go without saying, but if you use 2FA with a password that is shared across other websites, it's not true 2FA since this password can be cracked following a data breach and used to break into your Duo backup.

Duo's approach to privacy is not perfect, but I came away believing it offers the best privacy protections of any app with cloud backups, especially if it's used as a TOTP authenticator. First, the company's primary allegiance is to its business clients, who use the Duo authenticator to protect business resources like VPNs and login portals. In this context, these organizations have legitimate interests in information about the authenticating devices, such as its IP address, operating system version, and the identity of the account holder. When used as a standalone TOTP authenticator, all indications suggest that it is private. Duo has a clear privacy policy and states they do not sell your personal data. The app doesn't require a user account to function as an authenticator. When I disabled analytics, I did not detect any surreptitious server communication. There is one minor caveat: the encryption does not extend to the site usernames and identities; it only encrypts the secret tokens. Any party with access to your backups could identify what sites you access and your usernames on those sites. It is worth noting that unlike Google Authenticator, Duo uses third-party storage to facilitate cloud backups. Since they are not in control of these backups, you are entrusting Apple or Google with this data; Duo doesn't collect it. I find it highly unlikely that Apple or Google is associating this data with your account in a systematic way.

Ideally, an authenticator app would embrace a "zero-trust" architecture, encrypting all backup data, from site usernames and identities to secret codes. Unfortunately, there seems to be no commercial entity that offers this service. Duo seems to be the next best thing.

Authy: End-to-End Encryption with a Mixed Record on Security and Privacy

The feature that distinguishes Authy from other authenticators is its ability to sync tokens across multiple devices with end-to-end encryption. At first glance, this might seem like an ideal solution. However, Authy, a company focused on security, has shown inconsistent performance in this domain.

In August 2022, Authy suffered a data breach due to a phishing campaign. While Twilio's subsequent investigation indicated that the impact was limited to around 100 Authy users, it is concerning that their production environment was accessible via a phishing attack. Even more worrisome, a paper by Gilsenan et al. revealed that they had alerted Authy two years before the breach about a weakness in the encryption protecting their cloud backups. Astonishingly, Authy didn't address these concerns until October, after the breach had already occurred, and it remains uncertain whether the issue has been fully resolved. Furthermore, Authy discourages security researchers from publicly disclosing reported issues. Open disclosure, a standard practice in the industry, encourages transparency and accountability.

Despite these security concerns, it's worth noting that Authy's encrypted backups could be rendered more secure with some straightforward adjustments. After all, weak encryption is still preferable to Google Authenticator's lack of encryption.

When it comes to privacy, Authy leaves much to be desired. The app requires both an email and phone number to set up an account, ostensibly linking it to your real identity. Like Duo, Authy stores site identity and issuer information in an unencrypted format. Unlike Duo, this information is backed up to servers they control, meaning they could potentially use the information if they so desired. In my own testing, I discovered that Authy employs extensive, non-discretionary event-based logging. The app logs nearly every transaction and relays this data to remote servers. This includes when a user taps on an authentication code, and the corresponding event logs the transaction with a unique tag that identifies the token. This means that Authy has the technical capability to record all authentication attempts. They state that they do not share or sell information with companies for advertising purposes, but they do share information with companies that use their API. This can include limited location data.

The Details

What information can these apps really access?

The TOTP specification RFC 6238 and its predecessor RFC 4226 only define how the one-time passwords are derived from the shared secret. They do not define how the secrets should be shared. Fortunately, this process has been de facto standardized to use special URI encoding, as laid out by Google. These URIs contain three pieces of personal information:

• Label: Typically a combination of site and username
• Secret: The key used to generate the TOTPs
• Issuer: The site which issued the token

Duo Security - Backup

According to Gilsenan et al., Duo uses the argon2i Password-Based Key Derivation Function. Argon2 won the 2015 Password Hashing Competition and offers significantly higher resistance against brute-force attacks on low-entropy passwords. The parameters also seem to have been conservatively chosen. I could find no exact references validating these parameter choices, but RFC9106 offers:

The best-known attack on the 1-pass and 2-pass Argon2i is the low- storage attack described in [CBS16], which reduces the time-area product (using the peak memory value) by the factor of 5. The best attack on Argon2i with 3 passes or more is described in [AB16], with the reduction factor being a function of memory size and the number of passes (e.g., for 1 gibibyte of memory, a reduction factor of 3 for 3 passes, 2.5 for 4 passes, 2 for 6 passes). The reduction factor grows by about 0.5 with every doubling of the memory size. To completely prevent time-space trade-offs from [AB16], the number of passes MUST exceed the binary logarithm of memory minus 26.

With six passes, Duo is well above this threshold. They also set a high memory parameter of 128 MB. For encryption, they use XSalsa20-Poly1305. From what I've read, the XChaCha20-Poly1305 variant is preferred, but both are secure and widely adopted.

Authy Security - Backup

Authy uses the PBKDF2 Password-Based Key Derivation Function. As of 2022, the iteration count was set at 10,000, and I could find no reference indicating they've increased it since. The use of PBKDF2 is no longer considered best practice. Even if it has to be used due to outdated compliance requirements, OWASP recommends that the iteration count should be set above 600,000. Twilio uses the AES-256-CBC cipher to encrypt the data, which is considered secure and is recognized by NIST.

Logging and Data Collection

In order to observe apps' data collection practices, I used the Waydroid Android emulator and performed a man-in-the-middle attack, sniffing the traffic between the client application and the server. This was accomplished with mitmproxy running in transparent mode. I chose three apps to test: Authy, Duo, and Google Authenticator, since they were all closed source. Waydroid is based on Lineage OS and does not have Google Services, which may have affected these tests. I did not log into an account if it was not required. On all authenticators, I created two tokens, entered secrets I generated manually, provided a fake and unique account label, and then requested OTP codes from these two tokens.

I found that Duo did not make any requests to the server with logging disabled. Google Authenticator also did not contact the server. Authy, on the other hand, had extensive event-based logging. Most UI events sent data to Amazon's Kinesis service, in my case kinesis.us-east-1.amazonaws.com. These were sent in the form of HTTP POST requests. For instance, when the app is opened, it fires off a message to the server. A typical message looked like this:

{"Records":[{"Data":"ewogICJldmVudCI6ICJhcHBfc2Vzc2lvbl9pbml0aWFsaXplZCIsCiAgImV4dHJhIjogewogICAgImF1dGhlbnRpY2F0aW9uX2xvZ190b2tlbiI6ICJleUpoYkdjaU9pSklVekkxTmlKOS5leUpoZFhSb2VWOXBaQ0k2T0Rjd09UZzJOREF3TENKbGVIQWlPakUyT1RVeU9ERXdOVElzSW5abGNuTnBiMjRpT2lJeElpd2lZM1Z6ZEc5dFpYSWlPaUpoZFhSb2VTSXNJbUZqZEdsMmFYUjVJam9pYlc5aWFXeGxYMnh2WjE5MGIydGxiaUo5LjdwSlJEeUNtdUFJcG50RkVPbml3bVRfV25uYzdmQkljRlRRV3VueENXTGciCiAgfSwKICAibGV2ZWwiOiAiaW5mbyIsCiAgIm1lc3NhZ2UiOiAiVXNlciBvcGVucyB0aGUgYXBwIiwKICAib2JqZWN0cyI6IHsKICAgICJkZXZpY2UiOiB7CiAgICAgICJzX29zX3ZlcnNpb24iOiAiMzAiLAogICAgICAic19hY2Nlc3NpYmlsaXR5X3NlcnZpY2UiOiAibm9uZSIsCiAgICAgICJzX2RldmljZV9hcHAiOiAiYXV0aHkiLAogICAgICAic19hcHBfdmVyc2lvbiI6ICIyNC4xMy40IiwKICAgICAgInNfcHJvY2Vzc29yX2FyY2hpdGVjdHVyZSI6ICJ4ODZfNjQiLAogICAgICAiaV9udW1iZXJfb2ZfYXV0aGVudGljYXRvcl90b2tlbnNfZW5jcnlwdGVkIjogMCwKICAgICAgImJfYmFja3Vwc19lbmFibGVkIjogZmFsc2UsCiAgICAgICJzX2J1aWxkX3ZlcnNpb24iOiAiMTAxMSIsCiAgICAgICJpX2RlY3J5cHRpb25fYXR0ZW1wdCI6IDAsCiAgICAgICJzX2Fub255bW91c19pZCI6ICI5YzU1YWI0Yi04NWEzLTQxYWEtODY2Yy1jNjAwZTg0Yjk1Y2UiLAogICAgICAic191dWlkIjogImF1dGh5OjoxY2QxM2VjM2Q4ZmIzOWM0IiwKICAgICAgInNfZW5hYmxlZF9mZWF0dXJlX2ZsYWdzIjogInJlcG9ydF9hdXRoeV9hcHBzLCB2YWxpZGF0ZV93aGF0c19hcHBfaW5zdGFsbGVkLCBtaWdyYXRlX3Bpbl9hbmRyb2lkLCBjYW1lcmF4X3NjYW5uZXJfYW5kcm9pZCwgbmV3X3Rva2Vuc19saXN0aW5nLCBpbl9hcHBfdXBkYXRlX2FuZHJvaWQsIGRldmljZV9pbnZhbGlkYXRpb25fYW5kcm9pZCwgYmFja3VwX3Bhc3N3b3JkX2Zsb3dfYW5kcm9pZCwgZW1haWxfdmFsaWRhdGlvbl9hbmRyb2lkIiwKICAgICAgInNfZmlyZWJhc2VfaW5zdGFuY2VfaWQiOiAiZXlKaGJHY2lPaUpGdXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SmhjaEJKWkNJNklqRTZPREV5TWpJM01UTXlPREl4T21GdVpISnZhV1E2WVRRM1pETTVOREZsWkdReE0yUTNPQ0lzSW1WNGNDSTZNVFk1TlRnNE16a3lNQ3dpWm1sa0lqb2laa0ZUVEhkVlZqTlRRbUUzVTBsTVFWbHJTMHd3V2lJc0luQnliMnBsWTNST2RXMWlaWElpb2pneE1qSXlOekV6TWpneU14MC5BQjJsUFY4d1JRSWdUaGtLeGxaVXE5WnRVakdNYXkzWDNYZXdsam5DTV9VSjlVSEpBUl91bHZjQ0lRRDFlUHAwRFdSM1JWMjNEMF9sZ3MxVThsQkR3Z21ObGNVRUwyazlwQmc4Y3ciLAogICAgICAiaV9nb29nbGVfcGxheV9zZXJ2aWNlc192ZXJzaW9uIjogMTI0NTEwMDAsCiAgICAgICJzX2lkIjogIjg3MDk4NjU5MCIsCiAgICAgICJiX2RhcmtfbW9kZSI6IHRydWUsCiAgICAgICJzX2RldmljZV9tYW51ZmFjdHVyZXIiOiAiV2F5ZHJvaWQiLAogICAgICAic19tb2RlbF9uYW1lIjogIldheURyb2lkIHg4Nl82NCBEZXZpY2UiLAogICAgICAiYl9tdWx0aWRldmljZSI6IHRydWUsCiAgICAgICJub3RpZmljYXRpb25fY2hhbm5lbHMiOiB7CiAgICAgICAgInNfcHJpb3JpdHlfYXBwcm92YWxfcmVxdWVzdCI6ICJIaWdoIiwKICAgICAgICAic19wcmlvcml0eV9kZXZpY2VzIjogIkhpZ2giLAogICAgICAgICJzX3ByaW9yaXR5X21lc3NhZ2UiOiAiSGlnaCIsCiAgICAgICAgInNfcHJpb3JpdHlfbmV3X2RldmljZV9yZXF1ZXN0IjogIkhpZ2giLAogICAgICAgICJzX3ByaW9yaXR5X3N1cHBvcnQiOiAiSGlnaCIsCiAgICAgICAgInNfcHJpb3JpdHlfdG9rZW5zIjogIkhpZ2giCiAgICAgIH0sCiAgICAgICJpX251bWJlcl9vZl9hdXRoZW50aWNhdG9yX2FjY291bnRzIjogNCwKICAgICAgImlfbnVtYmVyX29mX3Zpc2libGVfYWNjb3VudHMiOiAwLAogICAgICAic19vcGVyYXRpbmdfc3lzdGVtIjogIkFuZHJvaWQiLAogICAgICAic19kZXZpY2VfdHlwZSI6ICJhbmRyb2lkIiwKICAgICAgInNfdXNlcl9hZ2VudCI6ICI8YXV0aHktYW5kcm9pZD4gPGFwcF92ZXJzaW9uXzI0LjEzLjQ+IDxvc192ZXJzaW9uXzMwPiA8cHJvY2Vzc29yX2FyY2hpdGVjdHVyZV94ODZfNjQ+IgogICAgfSwKICAgICJ1c2VyIjogewogICAgICAic19hdXRoeV9pZCI6ICI4NzA5OTY1MDAiLAogICAgICAic19jb3VudHJ5X2NvZGUiOiAiMSIsCiAgICAgICJzX2xvY2FsZSI6ICJlbiIsCiAgICAgICJpX251bWJlcl9vZl9hY2NvdW50cyI6IDQsCiAgICAgICJpX251bWJlcl9vZl9hdXRoeV9hY2NvdW50cyI6IDAsCiAgICAgICJpX251bWJlcl9vZl9kZXZpY2VzIjogMQogICAgfQogIH0sCiAgInByb2R1Y3QiOiAiYXV0aHktYW5kcm9pZCIsCiAgInJlcXVlc3QiOiB7CiAgICAiaWQiOiAiMDRmZmE3NDEtMmMyZS00ZjI3LWI2MGQtMWYzNDQ5YWY1Y2JiIgogIH0sCiAgInRpbWUiOiAiMjAyMy0wOS0yMVQwNjo1MTo1OS45NThaIgp9Cgo=","PartitionKey":"97c1a917-cb2f-43a9-baa8-954786931fde"}],"StreamName":"authy-coresdk-production"}

The base64 in the Data section is a second JSON object containing the structured log entry:

{
  "event": "app_session_initialized",
  "extra": {
    "authentication_log_token": "eyJhbGciOiJIUzI1NiJ9.eyJhdXRoeV9pZCI6ODcwOTg2NDAwLCJleHAiOjE2OTUyODEwNTIsInZlcnNpb24iOiIxIiwiY3VzdG9tZXIiOiJhdXRoeSIsImFjdGl2aXR5IjoibW9iaWxlX2xvZ190b2tlbiJ9.7pJRDyCmuAIpntFEOniwmT_Wnnc7fBIcFTQWunxCWLg"
  },
  "level": "info",
  "message": "User opens the app",
  "objects": {
    "device": {
      "s_os_version": "30",
      "s_accessibility_service": "none",
      "s_device_app": "authy",
      "s_app_version": "24.13.4",
      "s_processor_architecture": "x86_64",
      "i_number_of_authenticator_tokens_encrypted": 0,
      "b_backups_enabled": false,
      "s_build_version": "1011",
      "i_decryption_attempt": 0,
      "s_anonymous_id": "9c55ab4b-85a3-41aa-866c-c600e84b95ce",
      "s_uuid": "authy::1cd13ec3d8fb39c4",
      "s_enabled_feature_flags": "report_authy_apps, validate_whats_app_installed, migrate_pin_android, camerax_scanner_android, new_tokens_listing, in_app_update_android, device_invalidation_android, backup_password_flow_android, email_validation_android",
      "s_firebase_instance_id": "eyJhbGciOiJFuzI1NiIsInR5cCI6IkpXVCJ9.eyJhchBJZCI6IjE6ODEyMjI3MTMyODIxOmFuZHJvaWQ6YTQ3ZDM5NDFlZGQxM2Q3OCIsImV4cCI6MTY5NTg4MzkyMCwiZmlkIjoiZkFTTHdVVjNTQmE3U0lMQVlrS0wwWiIsInByb2plY3ROdW1iZXIiojgxMjIyNzEzMjgyMx0.AB2lPV8wRQIgThkKxlZUq9ZtUjGMay3X3XewljnCM_UJ9UHJAR_ulvcCIQD1ePp0DWR3RV23D0_lgs1U8lBDwgmNlcUEL2k9pBg8cw",
      "i_google_play_services_version": 12451000,
      "s_id": "870986590",
      "b_dark_mode": true,
      "s_device_manufacturer": "Waydroid",
      "s_model_name": "WayDroid x86_64 Device",
      "b_multidevice": true,
      "notification_channels": {
        "s_priority_approval_request": "High",
        "s_priority_devices": "High",
        "s_priority_message": "High",
        "s_priority_new_device_request": "High",
        "s_priority_support": "High",
        "s_priority_tokens": "High"
      },
      "i_number_of_authenticator_accounts": 4,
      "i_number_of_visible_accounts": 0,
      "s_operating_system": "Android",
      "s_device_type": "android",
      "s_user_agent": "<authy-android> <app_version_24.13.4> <os_version_30> <processor_architecture_x86_64>"
    },
    "user": {
      "s_authy_id": "870996200",
      "s_country_code": "1",
      "s_locale": "en",
      "i_number_of_accounts": 4,
      "i_number_of_authy_accounts": 0,
      "i_number_of_devices": 1
    }
  },
  "product": "authy-android",
  "request": {
    "id": "04ffa741-2c2e-4f27-b60d-1f3449af5cbb"
  },
  "time": "2023-09-21T06:51:59.958Z"
}

Many events triggered logs. For instance when an account is added, a token log item is sent back to the server:

{
  "event": "account_added",
  "extra": {
    "authentication_log_token": "eyJhbGciOiJIUzI1NiJ9.eyJhdXRoeV9pZCI6ODcwOTg2NDAwLCJleHAiOjE2OTUyODEwNTIsInZlcnNpb24iOiIxIiwiY3VzdG9tZXIiOiJhdXRoeSIsImFjdGl2aXR5IjoibW9iaWxlX2xvZ190b2tlbiJ9.7pJRDyCmuAIpntFEOniwmT_Wnnc7fBIcFTQWunxCWLg"
  },
  "level": "info",
  "message": "When users add an account",
  "objects": {
    "app": {
      "i_account_add_time_in_seconds": 453,
      "s_account_type": "authenticator",
      "s_logo": "authenticator_blue",
      "s_logo_type": "authenticator",
      "s_token_id": "1695240000"
    },
    "device": {
      "s_os_version": "30",
      "s_accessibility_service": "none",
      "s_device_app": "authy",
      "s_app_version": "24.13.4",
      "s_processor_architecture": "x86_64",
      "i_number_of_authenticator_tokens_encrypted": 0,
      "b_backups_enabled": false,
      "s_build_version": "1011",
      "i_decryption_attempt": 0,
      "s_anonymous_id": "9c55ab4b-85a3-41aa-866c-c600e84b95ce",
      "s_uuid": "authy::1cd13ec3d8fb39c4",
      "s_enabled_feature_flags": "report_authy_apps, validate_whats_app_installed, migrate_pin_android, camerax_scanner_android, new_tokens_listing, in_app_update_android, device_invalidation_android, backup_password_flow_android, email_validation_android",
      "i_google_play_services_version": 12451000,
      "s_id": "870986493",
      "s_device_manufacturer": "Waydroid",
      "s_model_name": "WayDroid x86_64 Device",
      "b_multidevice": true,
      "notification_channels": {
        "s_priority_approval_request": "High",
        "s_priority_devices": "High",
        "s_priority_message": "High",
        "s_priority_new_device_request": "High",
        "s_priority_support": "High",
        "s_priority_tokens": "High"
      },
      "i_number_of_authenticator_accounts": 5,
      "i_number_of_visible_accounts": 1,
      "s_operating_system": "Android",
      "s_device_type": "android",
      "s_user_agent": "<authy-android> <app_version_24.13.4> <os_version_30> <processor_architecture_x86_64>"
    },
    "user": {
      "s_authy_id": "870986200",
      "s_country_code": "1",
      "s_locale": "en",
      "i_number_of_accounts": 5,
      "i_number_of_authy_accounts": 0,
      "i_number_of_devices": 1
    }
  },
  "product": "authy-android",
  "request": {
    "id": "04ffa741-2c2e-4f27-b60d-1f3449af5cbb"
  },
  "time": "2023-09-21T07:06:42.990Z"
}

This log entry contains a token_id. When an account is selected, they also send the token id back to the server along with the associated user:

{
  "event": "account_selected",
  "extra": {
    "authentication_log_token": "eyJhbGciOiJIUzI1NiJ9.eyJhdXRoeV9pZCI6ODcwOTg2NDAwLCJleHAiOjE2OTUyODEwNTIsInZlcnNpb24iOiIxIiwiY3VzdG9tZXIiOiJhdXRoeSIsImFjdGl2aXR5IjoibW9iaWxlX2xvZ190b2tlbiJ9.7pJRDyCmuAIpntFEOniwmT_Wnnc7fBIcFTQWunxCWLg"
  },
  "level": "info",
  "message": "Users selecting account from account list/grid in app",
  "objects": {
    "app": {
      "s_account_column": "0",
      "s_account_row": "0",
      "s_account_type": "authenticator",
      "s_logo": "authenticator_blue",
      "s_logo_type": "authenticator",
      "s_token_id": "1695240000",
      "s_view_mode": "list"
    },
    "device": {
      "s_os_version": "30",
      "s_accessibility_service": "none",
      "s_device_app": "authy",
      "s_app_version": "24.13.4",
      "s_processor_architecture": "x86_64",
      "i_number_of_authenticator_tokens_encrypted": 0,
      "b_backups_enabled": false,
      "s_build_version": "1011",
      "i_decryption_attempt": 0,
      "s_anonymous_id": "9c55ab4b-85a3-41aa-866c-c600e84b95ce",
      "s_uuid": "authy::1cd13ec3d8fb39c4",
      "s_enabled_feature_flags": "report_authy_apps, validate_whats_app_installed, migrate_pin_android, camerax_scanner_android, new_tokens_listing, in_app_update_android, device_invalidation_android, backup_password_flow_android, email_validation_android",
      "i_google_play_services_version": 12451000,
      "s_id": "870986493",
      "s_device_manufacturer": "Waydroid",
      "s_model_name": "WayDroid x86_64 Device",
      "b_multidevice": true,
      "notification_channels": {
        "s_priority_approval_request": "High",
        "s_priority_devices": "High",
        "s_priority_message": "High",
        "s_priority_new_device_request": "High",
        "s_priority_support": "High",
        "s_priority_tokens": "High"
      },
      "i_number_of_authenticator_accounts": 5,
      "i_number_of_visible_accounts": 1,
      "s_operating_system": "Android",
      "s_device_type": "android",
      "s_user_agent": "<authy-android> <app_version_24.13.4> <os_version_30> <processor_architecture_x86_64>"
    },
    "user": {
      "s_authy_id": "870986200",
      "s_country_code": "1",
      "s_locale": "en",
      "i_number_of_accounts": 5,
      "i_number_of_authy_accounts": 0,
      "i_number_of_devices": 1
    }
  },
  "product": "authy-android",
  "request": {
    "id": "04ffa741-2c2e-4f27-b60d-1f3449af5cbb"
  },
  "time": "2023-09-21T07:15:33.378Z"
}

wg-quick(8) on Linux - a deep dive

Perhaps you have decided to secure your company's internal database or tunnel all your traffic through a VPN. You enter the command wg-quick up wg0, watch as commands scroll past your screen, and suddenly realize you can't access the network.

If this situation sounds familiar, you're not alone. WireGuard is a Layer 3 VPN that has become the de-facto standard for good reasons — it's fast, simple, and secure. Anecdotally, it's known to be easier to configure than its bulkier and more convoluted predecessor, OpenVPN. However, configuring or debugging WireGuard networks requires a robust understanding of networking, something the thorough WireGuard documentation can help most users with.

I recently reviewed the source for wg-quick(8) and discovered that it was not entirely documented. Some of the documented parts assumed an in-depth understanding of networking. This research turned out to be a surprisingly instructive exercise in Linux networking, and I'm sharing my results here. Hopefully, it can fill in some gaps in the existing documentation.

In particular, this guide:

1. Provides additional exposition on Linux networking
2. Details default-route handling
3. Explains the default firewall configuration
4. Describes the multicast limitations of WireGuard tunnels

The Basics

WireGuard adheres to the Unix philosophy of "doing one thing well," fostering a design of modular systems. Accordingly, it integrates into the Linux networking stack as a network interface, configured through the user space tool wg(8). However, to enable traffic flow, IP addresses must be assigned, and routes need to be configured. While this is a relatively simple procedure, it can become tedious and requires automation to ensure reliability. This is where wg-quick(8) steps in.

This is an extremely simple script for easily bringing up a WireGuard interface, suitable for a few common use cases. ... Generally speaking, this utility is just a simple script that wraps invocations to wg(8) and ip(8) in order to set up a WireGuard interface. It is designed for users with simple needs, and users with more advanced needs are highly encouraged to use a more specific tool, a more complete network manager, or otherwise just use wg(8) and ip(8), as usual.

- wg-quick(8)

wg-quick serves as a straightforward orchestration tool for creating and removing WireGuard tunnels. It is the de-facto standard for configuring tunnels since it is cross-platform, supported by all official clients. Its strength lies in the fact that the configuration provides a complete description of the tunnel, allowing tunnels to be brought up or down with a single command. Additionally, the interface can be initiated at boot through systemd, although - oddly - this feature remains undocumented. Without additional tooling, wg-quick can configure either a static server endpoint or a roaming peer.

A "static server endpoint" refers to a computer with a static IP, typically provided by cloud computing vendors, which functions as a node connecting peers. Often, it may facilitate access to protected resources, like a database. A roaming peer, on the other hand, is a machine that connects using the static IP of the server and can access the server or other peers. Peers typically lack a fixed endpoint and "roam" from one IP to another.

There's also a special provision for cases where a peer routes all their internet traffic through the tunnel, aligning with how most consumers conceive of a VPN. However, VPNs can also selectively transmit traffic bound for specific computers. Unfortunately, the server configuration for this setup cannot be handled with wg-quick alone since it necessitates Network Address Translation (NAT). This limitation likely stands as an exception, enabling VPN providers to distribute wg-quick configurations instead of writing their own tools to route traffic.

wg-quick is by no means the sole configuration tool for WireGuard. It's supported alongside other network management systems such as networkd and NetworkManager. Specifically, NetworkManager is a prevalent tool for managing WiFi networks on desktop environments, and it has the capability to import WireGuard tunnels using the .conf format, just like wg-quick. (The extent of feature compatibility between them, however, is something I'm not fully aware of at the moment.) On the other hand, networkd is more commonly leveraged for network management on servers. In the near future, I'll be releasing a tool that can "import" (or more accurately, transpile) wg-quick files into networkd configurations. When choosing between these options, it would be wise to defer to your system's network manager. Most users configure WireGuard tunnels within the initial network namespace. Multiple tools can end up tripping over one another while managing the same network resources.

It is also worth stating that wg-quick is not the only configuration tool. WireGuard is supported both by networkd and NetworkManager. NetworkManager is commonly used to manage WiFi networks on the desktop and can import WireGuard tunnels using the .conf format shared with wg-quick. (Although I am unaware of the feature compatibility.) networkd is commonly used to manage networks on the server. Soon, I will be releasing a tool I wrote which allows wg-quick files to be "imported" (or more accurately transpiled) to networkd. When considering which option to use, I would default to your system's network manager. WireGuard tunnels affect resources in the global network namespace, so it makes sense for them to be managed by an entity responsible for managing network resources globally.

The process of manually configuring a WireGuard tunnel is thoroughly outlined on the quickstart page. If you've set up a network with static IP addresses, you might already be acquainted with this process. However, many enthusiasts and developers may have avoided this route, since the aptly named Dynamic Host Configuration Protocol (DHCP) takes care of automating IP management. In the following section, I'll delve into configuring network interfaces on Linux, assuming that you possess a fundamental understanding of networking concepts. Should you be new to setting up a static network, or if you find yourself unfamiliar with terms like private IP address space, subnets, public IPs, routers, subnet masks, and interfaces, I strongly recommend taking the time to acquaint yourself with these concepts before proceeding.

Revisiting a Simple Network

WireGuard seamlessly integrates with the Linux networking stack, functioning as a networking interface. The Linux networking system is indeed robust, supporting a wide array of features. However newcomers should be warned, the documentation often appears to lag behind the pace of feature development. In this article, we'll explore the fundamentals of a simple Local Area Network (LAN) and shed light on how packets are routed on contemporary Linux systems. This foundational knowledge is essential for understanding how wg-quick interfaces with these routing constructs.

Assigning a static IP address to a networking interface can be accomplished using the following command. This sets up a LAN with the subnet 192.168.0.0-127 and assigns the networking interface the IP address 192.168.0.22:

[#] ip -4 addr add 192.168.0.22/25 dev eth0

Should you attempt to transmit data to a peer at 192.168.0.11 (e.g., using a UDP socket), how does the kernel determine where to route these packets? Upon executing the command above, the kernel modifies a structure known as the routing table. Routing tables function as a database, directing traffic to a specific interface by matching the destination IP address. Most routes are automatically appended to the main routing table, which can be viewed with the command ip route show:

[#] ip -4 route show table main

192.168.0.0/25 dev eth0 proto kernel scope link src 192.168.0.22

In essence, packets destined for our subnet will be sent using eth0 with the source IP address 192.168.0.22. The kernel automatically adds this route for a subnet assigned to a link, known as a prefix route. If multiple competing routes exist in a routing table, the route is selected using the longest prefix match algorithm. Essentially, the most specific route, characterized by the longest subnet mask, is chosen. If you add a custom route specifying that 192.168.0.11/32 should be sent using eth1, traffic will flow through this interface instead, since the 32-bit subnet mask is longer than the 25-bit subnet mask:

[#] ip -4 route add 192.168.0.11/32 dev eth1
[#] ip -4 route get 192.168.0.11
192.168.0.11 dev eth1 src 192.168.0.12 uid 0
    cache

This brings up an intriguing question: If we send traffic to the current host at 192.168.0.22, how does the routing algorithm determine that the packets shouldn't leave the computer but instead be handled by local programs? The answer lies in another table that precedes the main table, known as the local table. The local table is consulted before the main table, and if a match is found, the packet is routed accordingly. The kernel also manipulated this table in response to our earlier ip addr add command.

[#] ip -4 route show table local

local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
local 192.168.0.22 dev eth0 proto kernel scope host src 192.168.0.22
broadcast 192.168.0.127 dev eth0 proto kernel scope link src 192.168.0.22

The original ip addr add command added both a local route and a broadcast route. Specifically, the entry local 192.168.0.22 dev eth0 stipulates that packets addressed to 192.168.0.22 should be "looped back" and delivered to sockets listening locally on eth0, or bound to the address 192.168.0.22. Typically, IP traffic is unicast, meaning it originates with one host and is directed at another. The broadcast rule informs the kernel that 192.168.0.127 is a broadcast address. By default, the kernel utilizes the highest IP address in the subnet as the broadcast address.

The above overview offers almost a complete picture, but there's another crucial layer of indirection in play: the Routing Policy Database (RPDB). The RPDB acts as a guide, specifying how routing tables are selected and ordered. This can be queried with the ip rule command, and the initial state of the RPDB might look something like this:

[#] ip rule

0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

Quoting directly from the manual provides a comprehensive explanation:

Each policy routing rule consists of a selector and an action predicate. The RPDB is scanned in order of decreasing priority (note that a lower number means higher priority, see the description of PREFERENCE below). The selector of each rule is applied to {source address, destination address, incoming interface, tos, fwmark} and, if the selector matches the packet, the action is performed. The action predicate may return with success. In this case, it will either give a route or failure indication and the RPDB lookup is terminated. Otherwise, the RPDB program continues with the next rule.

Semantically, the natural action is to select the nexthop and the output device.

At startup time the kernel configures the default RPDB consisting of three rules:

1. Priority: 0, Selector: match anything, Action: lookup routing table local (ID 255). The local table is a special routing table containing high priority control routes for local and broadcast addresses.
2. Priority: 32766, Selector: match anything, Action: lookup routing table main (ID 254). The main table is the normal routing table containing all non- policy routes. This rule may be deleted and/or overridden with other ones by the administrator.
3. Priority: 32767, Selector: match anything, Action: lookup routing table default (ID 253). The default table is empty. It is reserved for some post-pro‐ cessing if no previous default rules selected the packet. This rule may also be deleted.

- ip-rule(8)

If you've been following along, you now understand that when a socket binds or connects to an address, the kernel selects an appropriate route by querying the routing tables. This query is guided by the rules specified in the routing policy database (RPDB). When a table contains multiple routes, the longest prefix match algorithm is used to determine the most specific route, and that is the one selected.

Note: Policy routing can be bypassed using the SO_BINDTODEVICE option. This allows you to bind a socket directly to a specific device, effectively sidestepping the standard routing process.

Bringing Up a Site-to-Site-Style Tunnel with wg-quick(8)

Use up to add and set up an interface, and use down to tear down and remove an interface. Running up adds a WireGuard interface, brings up the interface with the supplied IP addresses, sets up mtu and routes, and optionally runs pre/post up scripts.

- wg-quick(8)

The command wg-quick first searches for a configuration file. If the first argument to wg-quick up matches the pattern [a-zA-Z0-9_=+.-]{1,15}, which represents a valid interface name on Linux, the file is assumed to exist in /etc/wireguard with the provided name and a .conf extension. Once a suitable file is found, its contents are read. For example, it might read the following file, wg0.conf:

[Interface]
PrivateKey = yDdqzxdE66e64xy5Qu1PshT0ybQJLHbU9N+91PS1Dng=
Address = 192.168.42.1/24

[Peer]
PublicKey = o837llPmQ4t9cN0rmiLasp6SF54dAzS0Ea1p71c1jFA=
AllowedIPs = 192.168.42.0/32
Endpoint = 19.216.242.139:16262

[Peer]
PublicKey = YIUKiCiw9+6an3HnDn7t3CwlF30ERQkhEQ6f3jRBUnk=
AllowedIPs = 10.1.0.0/16
Endpoint = 19.216.242.138:16263

The process begins identically to the steps taken in the quick start documentation. A WireGuard interface is added to the system with the interface name from the corresponding file:

ip link add dev wg0 type wireguard

Then, the WireGuard configuration is obtained by stripping out the wg-quick-specific sections, passing the rest to the wg command. This configuration can be produced using the wg-quick strip command. Running wg-quick strip on the above example removes the Address section, and wg parses the configuration to pass it on to the kernelspace driver [1].

[1] It can also communicate this information to the userspace implementation, if available.

[#] wg-quick strip wg0.conf
[Interface]
PrivateKey = yDdqzxdE66e64xy5Qu1PshT0ybQJLHbU9N+91PS1Dng=

[Peer]
PublicKey = o837llPmQ4t9cN0rmiLasp6SF54dAzS0Ea1p71c1jFA=
AllowedIPs = 192.168.42.0/32
Endpoint = 19.216.242.139:16262

[Peer]
PublicKey = YIUKiCiw9+6an3HnDn7t3CwlF30ERQkhEQ6f3jRBUnk=
AllowedIPs = 10.1.0.0/16
Endpoint = 19.216.242.138:16263

As the official documentation states:

The configuration file adds a few extra configuration values to the format understood by wg(8) in order to configure additional attributes of an interface. It handles the values that it understands, and then it passes the remaining ones directly to wg(8) for further processing.

- wg-quick(8)

The way the interface configuration affects packet routing is well-covered by the original documentation. If you haven't read it, you should explore the "Simple Network Interface" and "Cryptokey Routing" sections before continuing.

[#] wg-quick strip wg0.conf > wg0-stripped.conf
[#] ip setconf wg0 wg0-stripped.conf

Next, the IP addresses specified in Addresses are added to the interface. For the above configuration, it assigns the IP address 192.168.42.1 to the WireGuard interface. This becomes the source IP address for packets traveling to the peer VPN endpoint [3]. The system also automatically creates a prefix route directing traffic from any IP address matching 192.168.42.0/24 to the interface wg0, and a local route for 192.168.42.1.

[3] The source IP address, as defined in the Addresses section of the configuration, serves as the originating address for packets sent to the peer VPN endpoint. However, this can be overridden using a "raw socket," in which an application has the ability to define all fields of a packet, including the source IP. While this introduces significant security considerations, WireGuard's Cryptorouting scheme substantially mitigates the risks. Malicious peers are restricted to spoofing only those addresses listed in the AllowedIPs section of the [Peer] configuration, and all unauthorized packets are promptly dropped. This built-in mechanism aids in containing potential threats.

[#] ip -4 addr add 192.168.42.1/24 dev wg0

Afterward, the link is brought up, and the MTU (Maximum Transmission Unit) is set. The MTU represents the maximum size of a packet that can be communicated on a link. If unspecified, the MTU of the underlying link is obtained, and the MTU of the WireGuard tunnel is reduced by 80 bytes to exclude the overhead of the WireGuard encapsulation packets. I looked into this and am still unsure as to why an 80 byte MTU reduction was chosen. My working theory is that this value was chosen to cover both IPv4 and IPv6 with a factor of safety [4].

[4] The encapsulation method depends on whether the tunnel endpoint is an IPv4 address or an IPv6 address. The maximum size of an IPv4 packet is 60 bytes. IPv6 packets are more challenging to quantify since they can have any number of arbitrary header extensions. The base IPv6 header is only 40 bytes but additional headers can add up. Accordingly, 80 bytes would cover an IPv6 header with a few extensions.

[#] ip link set mtu up 1420 dev wg0

Finally, routes are added for all the entries of AllowedIPs to the main routing table:

[#] ip -4 route add 192.168.42.0/32 dev wg0
[#] ip -4 route add 10.1.0.0/16 dev wg0

Default-Route Handling

The above example illustrates how WireGuard establishes static routes for specific segments of the IP space, which we specified in AllowedIPs. However, these previous examples assumed that the peer endpoint falls outside any of the AllowedIP ranges. If this were not true, the routes would create a routing loop, with packets exiting the WireGuard interface being routed back into it.

One common use case for a VPN is to route all traffic to an endpoint which functions as a NAT router. (If you are unfamiliar with NAT, picture your home router home router.) This can enhance privacy, reduce the specificity of a user's public IP address in fingerprinting systems, prevent ISPs from selling data to advertisers, or obscure the regional ISP an individual is using. To route all traffic, we need a default route 0.0.0.0/0 and ::0/0 which match all destination IPs. This creates an issue since our endpoint's IP will fall into this range. Fortunately, there are methods to make default routes work as intended.

Linux policy routing can be used in various ways to solve this problem, and the Wireguard Routing and Namespace Documentation outlines a few potential solutions. wg-quick elegantly accomplishes this with the following rules, which require some explanation:

[#] wg set wg0 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0

The Linux kernel routing system features fwmark, an integer marker ranging from 0 to 2^32 - 1 which can be attached to a packet. It designates that the packet should be routed or filtered according to special rules. Packets with this mark can either be routed to a specific table using policy routing (ip-rule) or filtered with the nftables framework. The fwmark can be set either within the netfilter subsystem or by a program when making a network connection [4].

[4] setsockopt can be used with SO_MARK to set the outgoing mark as so long as the process has the CAP_NET_ADMIN capability.

In this instance, wg-quick marks all packets leaving the tunnel with the fwmark 51820:

[#] wg set wg0 fwmark 51820

Recall our previous discussion of routing tables. wg-quick creates a new routing table within which the default route directs all traffic to the wg0 interface, effectively serving as a "default gateway." With none of the mask bits set, all traffic is routed to wg0:

[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820

Traffic must then be directed to this routing table. Here, all traffic that has not exited the VPN interface (and thus does not have this fwmark) is sent to the table with the default route:

[#] ip -4 rule add not fwmark 51820 table 51820

By default, the IP command assigns this rule a higher priority than the rule querying the main table.

[#] ip rule
0:	from all lookup local
32765:	not from all fwmark 0xca6c lookup 51820
32766:	from all lookup main
32767:	from all lookup default

We have established that traffic will be directed through the Wireguard interface if it does not match any local IP ranges. It will then be sent through wg0 before being routed according to the main table (and likely exit through a default gateway.)

The ip rule command prints the mark in hex as 0xca6c.

Handling Local Network Traffic

While VPN traffic will flow as expected, this configuration may have unintended consequences. All local LAN traffic will be directed to the tunnel, preventing access to the local network (except for encrypted Wireguard packets directed to the default gateway). Policy routing has one more trick:

[#] ip -4 rule add table main suppress_prefixlength 0

The command suppress_prefixlength rejects all routes with prefixes equal to or less than the specified length. So, suppress_prefixlength 0 rejects all default gateways, causing the routing algorithm to query the main table first, which typically contains LAN routes but can also contain routes setup by virtualisation software.

These rules are set whenever the default IPv4 0.0.0.0/0 or IPv6 ::/0 routes appear in the AllowedIPs. In the case of IPv6, ip -6 rules are added as well. Finally, if FwMark is not specified, wg-quick searches for the next available mark, stopping when an empty routing table is found. Finally, the FwMark is set to the routing table number.

The Default Firewall

wg-quick configures a firewall if nftables is installed and a default route is specified. The following firewall rules are setup when an IPv4 endpoint is specified:

table ip wg-quick-wg0 {
    chain preraw {
        type filter hook prerouting priority raw; policy accept;
        iifname != "wg0" ip daddr 192.168.42.1 fib saddr type != local drop
    }

    chain premangle {
        type filter hook prerouting priority mangle; policy accept;
        meta l4proto udp meta mark set ct mark
    }

    chain postmangle {
        type filter hook postrouting priority mangle; policy accept;
        meta l4proto udp meta mark 0x0000ca6c ct mark set meta mark
    }
}

The first rule in the preraw chain mitigates a potential vulnerability that could allow a network attacker to access servers listening on wg0. It's easy to assume that local services listening on the WireGuard IP address are safe since they're only accessible to those connected to the tunnel. However, this is not the case.

Imagine an attacker connected on eth0, an adjacent ethernet interface, attempting to access the address assigned to our WireGuard interface, 192.168.42.1. Since a route for this address is in the local table, the attacker can send packets to 192.168.42.1 and receive responses, even if the requests originate from another subnet. This could enable the attacker to exfiltrate sensitive data or inject malicious packets. Therefore, the first rule enforces that services on wg0 can only be accessed through addresses on the same interface.

Reverse Path Forwarding

The last two rules in the code snippet above pertain to reverse path forwarding, a technique used to prevent spoofed packets from entering a network. Consider a router with forwarding enabled and two links:

172.30.1.0/31 dev eth-01 proto kernel scope link src 172.30.1.0
172.30.2.0/31 dev eth-02 proto kernel scope link src 172.30.2.0

Now, think about a scenario where a router connected to eth-02 with IP 172.30.2.1 and an attacker connected to this router attempt to perform a denial-of-service attack against 1.1.1.1 by spoofing network ICMP packets with random destination IPs and the source address of their victim. Classically, if one of these spoofed packets "from" 1.1.1.1 destined for 172.30.2.1 arrives on eth-01, it would be forwarded. However, this would be suspicious since a packet from 172.30.2.1 destined for 1.1.1.1 would be dropped, not forwarded through eth-02. By considering if a routable path exists in the reverse direction, a router can automatically drop spoofed packets. This concept is known as reverse path forwarding, as defined by RFC 3704, known as "strict reverse path forwarding."

To enable strict reverse path filtering in the Linux kernel, you can use the following sysctl switch:

[#] sysctl net.ipv4.conf.INTERFACE_NAME.rp_filter=1

Most Linux distributions default to Loose Reverse Path Forwarding. In strict mode, traffic from 172.30.1.1 on eth-02 would be dropped, as it should have routed through eth-01. This enforces a symmetrical routing policy but can disrupt asymmetric routing configurations. Loose Reverse Path Forwarding, conversely, processes traffic if routable through any interface.

Why does this matter for WireGuard? A firewall's secondary set of rules ensures that strict reverse path forwarding operates accurately. The fwmark is set by the traffic traversing the reverse path, working in collaboration with Linux's conntrack subsystem, which persistently associates incoming traffic with its corresponding outgoing traffic [5]. The second rule sets the "connection mark" labeled ct mark as the fwmark during connection establishment, while the third rule copies it back when packets are received.

wg-quick then instructs the kernel to use the fwmark for reverse path forwarding, since this is not enabled by default:

[#] sysctl -q net.ipv4.conf.all.src_valid_mark=1

[5] For example, in tracking a UDP stream, both IPs (source and destination) and L4 headers (source and destination ports) can be tracked to identify a connection.

Additional Thoughts

When researching WireGuard and reading through the documentation, I found there were a few non-obvious technical details that seemed undocumented, although potentially important.

UDP Socket Parameters

WireGuard permits users to modify the UDP port on which an interface listens using ListenPort. Although, it always binds its internal UDP socket to all available interfaces using INADDR_ANY [6]. This is an important consideration reasoning about packet flows and writing netfilter rules.

Broadcast and Multicast Traffic

Broadcast and multicast traffic are used in standard protocols, notably service autodiscovery. When considering how to configure effective firewalls, I was led to ask whether WireGuard carried multicast traffic. Short answer: yes and no. If the multicast ip range 224.0.0.0/4 or ff00::/8 appear in the AllowedIPs for the tunnel, multicast traffic could theoretically be broadcast from the other peer. Although, this works with at most two peers. Internally, WireGuard uses a single prefix tree to select the endpoint to which a packet is sent. If the same AllowedIPs appear within the configuration of multiple peers, the last peer configuration overwrites the routes configured on other peers [7]. Thus Mutlicast traffic can be directed to, at most, one peer. Thus, if you want to pass broadcast or multicast traffic, a secondary encapsulation method must be used on top of WireGuard. This does not mean peers with default routes in their AllowedIPs may leak Multicast traffic. This seems to be technically possible.

In practice, the interface flags are used by programs like avahi to determine on which interfaces they should broadcast [8]. WireGuard is not started with the IFF_MULTICAST or IFF_BROADCAST flags [8]. According to comments in the Linux kernel, links without IFF_MULTICAST can perform multicast but point-to-point devices cannot broadcast [9]. The exact meaning of these flags seem ambiguous from the limited documentation, and I wouldn't be surprised if the kernel and userspace developers were on different pages as to their exact technical meaning.

IFF_MULTICAST means that this media uses special encapsulation for multicast frames. Apparently, all IFF_POINTOPOINT and IFF_BROADCAST devices are able to use multicasts too.

- [9]

[6] Kernel cGit Github

[7] Github

[8] Kernel cGit Github

[9] Kernel cGit Github

Announcing wg2nd: Migrate WireGuard Configurations to networkd

Today, I am excited to release wg2nd, a tool specifically engineered to convert WireGuard configurations from the wg-quick(8) format to systemd-networkd compatible configurations.

• wg2nd - Source Code
• wg2nd-web - Web Port (contains some limitations)

Purpose

wg2nd serves as a bridge to translate wg-quick configurations into networkd configurations without requiring additional setup. networkd is a feature-complete network manager, allowing users greater control over WireGuard tunnels. This tool also addresses potential reliability issues that may arise when networkd interferes with tunnels it doesn't manage. Moreover, wg2nd can batch-convert wg-quick configurations to networkd.

Goals of the Project

1. Compatibility: wg2nd supports all wg-quick configurations except those that involve PreUp, PostUp, PreDown, and PostDown scripts, which are omitted.

2. Security: Private and symmetric keys are stored in keyfiles with restricted access permissions. wg2nd leverages the same formally-verified Curve25519 implementation employed in WireGuard. All operations involving private keys are executed in constant-time. Additionally, the web port operates entirely on the client-side. It does not transmit or store any sensitive data.

3. Reproducibility: wg2nd generates configurations deterministically with respect to the input WireGuard configuration. When updates are made to the WireGuard source configurations, only the corresponding elements in the output will be altered. This ensures that configurations from a VPN provider can be batch-converted without generating unnecessary files or inducing unexpected behavioral changes.

   Keyfiles for both private and symmetric keys are named according to the public key of the relevant interface or peer. These keyfiles are encoded in base32 rather than base64 to avoid issues with the Unix path separator present in base64 encoding. The public key corresponding to a keyfile can be obtained using the following command:

   echo $KEY | sed -E 's/\.(priv|sym)key//' | base32 -d | base64

   This approach effectively ensures that if two interfaces share the same private key, a single shared keyfile will be generated. The fwmark field employs a SipHash of the interface name, enabling the generation of identical network and netdev files across separate program invocations, while minimizing the risk of fwmark collisions.

Compatibility and Limitations

wg2nd is designed for high compatibility but comes with some caveats:

1. Dynamic Firewall Installation: Unlike wg-quick, which installs a firewall by default when a default route is specified, wg2nd does not. However, an equivalent firewall can be generated if desired.

2. Pre/Post Interface Setup Scripts: wg2nd does not handle PreUp, PostUp, PreDown, and PostDown script snippets, which wg-quick does recognize.

3. FwMark and Table Handling: wg2nd uses a deterministic method for generating fwmark based on the interface name. This contrasts with wg-quick, which dynamically checks availability. This deterministic approach is necessary because a static value must be chosen for configuration. However, this could result in a birthday collision if a large number of interfaces are ported. (Such a scenario becomes only remotely probable after porting around 500 interfaces.)

Web Port

The web port has been developed by converting the C / C++ implementation into WebAssembly (WASM). It offers an entirely browser-based experience, converting your WireGuard configurations into a series of Bash commands to configure the interface. This allows you to experiment within your browser.

The code is dual-licensed under the GPL-2.0 and MIT licenses. Feel free to send me patches via email or submit pull requests through GitHub.

For further details, including installation instructions, please consult the project README.

Happy networking!

Building the A1 Differential Drive Robot

Recently I embarked on a project to build a differential drive robot from commercial parts. I intend to eventually use this platform for testing sensor fusion, localization, and mapping techniques. Initially, I built a platform to accomplish a simpler goal; to navigate along a user selected path.

System Design

Motor Selection and Mounting

The robot was designed to navigate through an indoors environment at a speed of 40 cm/s which seemed reasonable. I was also concerned with selecting motors to achieve a smooth drive, especially when navigating over high friction surfaces like carpet. I searched for motors which could sustain around half a newton of force tangent to the wheel continuously. Often, one would consider continuous rotation servos in this case since they provide a gear motor with built-in closed-loop control. Continuous rotation servos which operate in this range can be quite expensive so I opted for a 110 rpm 5 kg cm DC gear motor. The motor came with a quadrature encoder that I used to provide feedback for a closed-loop control algorithm.

To mount the motor to the drive base, I created two mounting plates with a motor cage. This cage mounted to the bottom of the base plate with M3 screws. I also attached a passive caster to the base plate through a 3D printed offset. The base plate was made of 2 mm polycarbonate.

Electronics

To control the motors, I ended up using two Arduino Nanos because each motor requires two interrupt pins for each quadrature signal. A single Arduino Mega could be used to trigger interrupts but I had Arduino Nanos on hand. The Nanos interfaced with a TB6612FNG H-Bridge to provide speed control from a 12 V supply. A RPi 3B+ was used to perform the path calculations. The Nanos only have 2.5 kB of SRAM so the paths are stored on the RPi and fed over the I2C bus. Or at least, that was the idea. The current version stores the paths in flash. More on that later.

To power the robot, I used a three cell LiPo battery. This was connected to a BMS which provided over current and over discharge protection. The BMS output distributed power to each motor and a 5V buck-boost converter. Each was protected by a fuse.

Control Algorithms

The motion pipeline are composed of three stages:

1. Trajectories are generated on the RPi. These are provided to the motor controllers over the I2C bus.
2. The encoder signals are decoded and the position estimation is updated.
3. The trajectory and current motor position are used to calculate the input voltage for the motors.

Trajectory Generation

The paths are specified parametrically in the form <x(k), y(k)> This is transformed into a trajectory <x(t), y(t)> by time parametrizing it. This is a non-trivial problem since the rotational and forward velocities of a differential robot are intertwined: if motors are operating at their maximum velocity, an increase in the rotational velocity requires a decrease in the forward velocity. To plan a trajectory along a path, the maximum forward (tangent) velocity was calculated at each position k along the tangent path. This velocity limit varies with the curvature; the higher the curvature, the slower the robot can navigate along the path. Numerically, the forward velocity limit imposed by a single wheel (left or right) is proportional to the derivative of the tangent arc length with respect to the wheel arc length where the constant of proportionality is the max motor velocity. This provides a ceiling on the tangent velocity. The initial and final velocities along the path are known. This same process can be used to bound acceleration. The exact forward velocity transitions can be determined by a motion profile tuned to stay within the boundaries of these constraints. In my case, I used a simple trapezoidal profile. The tangent velocity function can be used to identify the position trajectories of each wheel. (In terms of path length.) These wheel position trajectories were fed to each motor controller.

Encoder Feedback

In order to provide accurate motion control, the system monitors the position of the motor and uses this information to make more informed estimates of the input voltage required to reach the target position. Quadrature encoders emit square waves on two channels A and B. Transitions in the signals A and B  encode changes in the motor position. For instance, when A transitions from low to high while B is low, this indicates that the motor has moved one section of an arc in the forward direction. If B made the transition before A, the encoder would move in the opposite direction. To decode the signal, the algorithm keeps a running tally of the number of arcs recorded. Each signal state is encoded in two bits. Following each state transition, the two bits representing the prior state and the two bits representing the current state query a lookup table containing the eight possible states. The counter is incremented or decremented according to the table entry. This maintains an accurate record of the encoder position. I've seen similar techniques in use elsewhere. In my case, this routine was triggered by a hardware interrupt. Triggering on interrupts ensures the algorithm doesn't miss a state transition while carrying out other control tasks.

Position Control

Armed with the trajectories, each motor controller was tasked with providing the correct input voltages to reach the designated positions. To accomplish this, it used feed forward motion control. Using this technique, the algorithm makes a crude initial guess at the input voltage. Then, it uses the known position, as obtained by the encoder, to correct this initial guess. A PID controller is used to make this correction. PID controllers are used commonly in industrial applications. Feed forward techniques, while less common, increase the responsiveness of the system to changes in the input position.

voltage = k_vf * v_setpoint + k_fa * a_setpoint + k_p * err + k_d * derr/dt

Known Issues

There are two main challenges with the current design. The first is that the 2 mm polycarbonate is flexible causing distortions in the width of the drive base. To mitigate this while testing, I added additional support to prevent the base board from flexing. A simple fix would be to combine both motor mounts into a single 3D print to add additional support. The second more significant issue is that the motors cause EMI on the I2C bus. I find it highly likely that this is due to high ground currents. I am currently experimenting with bus isolators to prevent the noise from affecting the bus.

Results

The result is a robot which can follow an input trajectory with surprising accuracy. I tested the robot against cosine, ellipse, and figure eight trajectories. In my testing, the robot generally deviated less than a centimeter along a five meter path.

Packaging Nebula for Debian

I am close to concluding a multi-week endevor to package Nebula, a VPN-style network mesh networking overlay. If all goes well, it will be uploaded to debian/experimental within the next few days. This would also mean the package would be pulled into Ubuntu during the next merge window.

Timeline

Unfortunately, Debian does not adhere to a constant release cycle. This means the timeline is uncertain. It will likely be uploaded to experimental within a few days. See the new queue. It will stay in experimental for the next three months or so until the next release occurs. (It is incompatible with the version of protobuf in unstable. This prevents it from moving into unstable until the next version release.)

{upload queue} -> [experimental] -> [unstable (sid)] -> [testing] -> [next release]

Preemptively, I'm going to write up a set of install instructions specific to debian derivatives and briefly a few of the decisions made during the packaging process.

Installation

Step one will currently fail. See installing from experimental

For the sake of simplicity, I'm going to assume that you're setting up a network with two nodes -- one lighthouse node and a node on your laptop. Once you understand the process, it easily scales to as many nodes as you wish. Pick your favorite virtualization provider in order to set up the lighthouse. The lighthouse requires minimal resources because it functions as a mutually-reachable node which synchronizes the address mappings. You could use a home server provided that you have a static ip (unlikely) or setup dynamic DNS. The latter may introduce some instability. I'm also assuming both clients are debian derivatives and have access to apt.

If this is not the case, please consult the upstream instructions which will guide you through the processing of installing the binaries directly.

1. Install Nebula through Aptitude

You'll need to install Nebula on both endpoints.

sudo apt install nebula

2. Creating a certificate authority

The certificate authority is to "root of trust" for a Nebula network. Compromising the certificate authority's key file would compromise the integrity and security of the entire network. The upstream instructions recommend that you store the key file in a location with strong encryption [^1].

You can generate a ca.key and ca.cert file with the following command:

nebula-cert ca -name "Myorganization, Inc"

You will copy the ca.crt file to all the hosts. The ca.key file should remain secret.

3. Nebula host keys and certificates generated from that certificate authority

With your ca.key file in hand, generate keys for each node.

nebula-cert sign -name "lighthouse" -ip "192.168.100.1/24"
nebula-cert sign -name "laptop" -ip "192.168.100.2/24"

Repeate this process for each node. It is important that each is issued a unique internal ip. The IPs are specified in CIDR notation [^2]. This internal ip will be used to configure Nebula later.

4. Copy the configuration files to each host

Each host requires the host.key, host.crt, and ca.crt files to be present on the system. By convention, these are located in the /etc/nebula directory. Make sure to copy them into this directory.

For example, to copy the credentials to a lighthouse with ip 203.0.113.11 as user you may use sftp and ssh as follows:

sftp user@203.0.113.11 <<EOF
put lighthouse.key
put lighthouse.crt
put ca.crt
EOF

ssh user@203.0.113.11
sudo install -m 600 -o root lighthouse.{key,crt} /etc/nebula
sudo install -m 600 -o root ca.crt /etc/nebula
rm ca.crt lighthouse.{key,crt}

5. Configure your network

The upstream recommends that you start from an example configuration file:

cp /usr/share/doc/nebula/examples/config.yml /etc/nebula/my_network.yml

• On your lighthouse, you'll want to change the cert and key sections to the paths /etc/nebula/lighthouse.crt and /etc/nebula/ligthouse.key. Change am_lighthoue: true. Remove the lighthouse ip from the hosts section under lighthouse.

• On the host, change the cert and key sections to the paths /etc/nebula/laptop.crt and /etc/nebula/laptop.key. Ensure the lighthouse is added to the static_host_map and the hosts section.

Once you're done, you can test whether your configuration is valid with nebula-service -test -config /etc/nebula/my_network.yml.

6. Bringing up the tunnel

To start the tunnel, you can use the templated systemd service packaged alongside Nebula [^3][^4].

sudo systemctl start nebula@my_network

There is also a means by which a Nebula lighthouse can be run by a unprivileged user but further configuration is required [^5].

Once both ends of the tunnel have been started, you should be able to ping the lighthouse from the laptop node and vice versa.

ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=5.67 ms

7. Additional Configuration

Nebula has built-in default deny firewall. The default configuration file allows network traffic outbound. (That is, any node is permitted to initiate a connection.) In order for a node to provide services, the port mapping needs to be added to the inbound section. For instance, to permit ssh to the lighthouse:

inbound:
   - port: 22
     proto: tcp
     host: lighthouse

Now, an ssh connection should be able to be initiated via Nebula's internal ip:

ssh 192.168.100.1

Once you're happy with the setup, you can automatically start Nebula when the laptop / server boots:

sudo systemctl enable nebula@my_network

For more information on usage and configuration, you may want to take a look at nebula.yml(5), nebula(1), and nebula-cert(1).

Enjoy.

  *  . . *    * .          . *    .      *   .   .   * .
. * .     .  *      .   *    .     *   .      *    .    *
.    .   *   .    *   .      *    .     *    .      *   .
. *   *   .   *   .   *   .    .   *   .     .  *    *   .
  .     .  *   .         *    .      *   .  *    *  .   .
 *   . *      .   *   .  *   . *    .    *   .   .    *  

Installation Footnotes

[^1] If you're in need of a technology to provide strong encryption, LUKS is a popular choice on Linux. Veracrypt is a venerable cross-platform encryption application. Some password managers, like KeePassXC, also allow you to attach files.

[^2] Effectively, this means all nodes will receive an ip in the form "192.168.y.x". The y part is a value in the range [0, 255] and is specific to the network. (Thus, all nodes should have the same "y" value.) "x" should be a unique ip for each node and be in the range [0, 244].

[^3] The launcher I wrote will detect the my_network.yml and my_network.yaml files. Do not specify the extension when launching the service. The launcher has no way to discriminate between my_network.yml and my_network.yaml extension so pick a distinct name for each network.

[^4] If Nebula is misconfigured, the service will fail without warning. You can check the status of the unit with systemctl status nebula@my_network. Nebula can also be started within the terminal using nebula -conifg /etc/nebula/my_network.yml.

[^5] The systemd unit that is packaged with Nebula runs the interface as root. This is what I expect most users will want. If the lighthouse doesn't need to be connected to the network, you can sudo systemd edit nebula@.serivce and simply change the User section to the user you wish to use to launch Nebula. The user will also need read access to the configuration file, key, and cert files.

Packaging Notes

I am going to create a brief summary of the changes made while packaging. I suspect other distros might benefit from some of the work done to package Debian [^6].

The Debian package differs from the packaging done on Arch. There's also a package created for NixOS but Nix is its own beast.

Templating the Unit File

If we were to use the unit file provided by the upstream project, it would fail without warning until the user fully setup the service because (1) the path of to Nebula configuration was hard coded as /etc/nebula/config.yml and (2) the user needed to change the configuration file in order for Nebula to function.

To make the relationship between the user configuration and the systemd unit clear, the systemd unit was templated. This also means that there is a clear and simple way to connect one machine to multiple Nebula networks. To accomplish this and support both .yml and .yaml extension, the systemd file executes a shell script under /usr/lib/nebula/bin/nebula-systemd-launcher passing the "instance variable" as the first argument. This script then identifies the proper configuration and launches Nebula with this configuration. The script was installed user /usr/lib so that it would not autocomplete in the user's shell.

Doc and examples

• Man pages were generated from the nebula help flag to create nebula(1) and nebula-cert(1).
• nebula.yml(5) man page was created to describe the configuration process. It was derived from the comments in the example configuration.
• The config.yml example configuration was installed under /usr/share/doc/nebula/examples/ so users could copy it into /etc/nebula if they wished to use it as a starting point.

Patching for Go 1.13

Debian packages all go dependencies to maintain tight control over the versions used while building go binaries. It also packages go itself. Currently, go 1.16 is not in the debian repos [^7]. The following patch was applied since net.ErrClosed is not available in older versions of go.

--- nebula.orig/sshd/server.go
+++ nebula/sshd/server.go
@@ -1,7 +1,7 @@
package sshd
import (
-   "errors"
+   "strings"
   "fmt"
   "net"
   "sync"
@@ -116,7 +116,8 @@ func (s *SSHServer) run() {
   for {
       c, err := s.listener.Accept()
       if err != nil {
-           if !errors.Is(err, net.ErrClosed) {
+           str := err.Error()
+           if !strings.Contains(str, "use of closed network connection"){
               s.l.WithError(err).Warn("Error in listener, shutting down")
           }
           return

Dependencies

• I packaged golang-github-nbrownus-go-metrics-prometheus since changes were made to go-metrics-prometheus that were not backwards-compatible
• I also packaged golang-github-flynn-noise since it was not in the debian repositories

Packaging footnotes

[^6] All files used to create the package are located in Salsa (Debian's VCS). All external configuration and build rules are located in the debian directory.

[^7] Actually, it is packaged individually but not under the golang-go moniker. I initially compiled it by preloading the PATH with go 1.16 to force dh-golang to use those build tools. Thus caused dh-golang to misbehave and not harden or strip the binaries. Since the changes required to adapt Nebula to go 1.13 were minimal, I opted to create a patch.

Installing from experimental

This is a temporary aside. As mentioned above, the package is currently bouncing around Debian's packaging infrastructure. I'm assuming at the time of reading that it is in experimental. This is an internal Debian repository which allows maintainers, developers, or the curious to test the newest version of software before it enters the next Debian unstable.

If you are running buster, you cannot install it directly using apt. If you would like to test the package while it is experimental, I will offer some instructions here. All the usual disclaimers apply. This is fairly safe since Nebula is a binary package (and doesn't have any runtime dependencies other than glibc).

There is a remote chance it will segfault due to binary incompatibilities with the version of glibc. If so, run sudo apt purge nebula and try installing from source. Building it from sources would require you to pull in a plethora of experimental build dependencies.

1. Add experimental to your sources.list file

sudo sh -c "
sudo cat >/etc/apt/sources.list.d/99-tmp-nebula-overrides.list <<EOF
# Temporary pull in packages from the experimental distribution
 
deb https://deb.debian.org/debian experimental main
EOF
"

2. Demote experimental in your apt preferences

sudo sh -c "
sudo cat >/etc/apt/preferences.d/99-tmp-nebula-prefer-stable <<EOF
Package: *
Pin: release o=Debian,a=experimental
Pin-Priority: -10
"

3. Update

sudo apt update

If they above steps succeed, you should see:

All packages are up to date.

3. Force APT to install the package from experimental

sudo apt install -t experimental nebula

After installing, you can continue to creating a certificate authority. Just ensure to remove nebula when you're finished testing.

4. When you're done testing

sudo rm /etc/apt/sources.list.d/99-tmp-nebula-overrides.list \
       /etc/apt/preferences.d/99-tmp-nebula-prefer-stable
sudo apt purge nebula

Installing cGit behind NGINX on Ubuntu

cGit is a fast web interface based on the CGI specification. It is lightweight and doesn't require a database or web authentication system.

It's easy to configure. For some reason, all the online guides for Ubuntu decided they needed to compile it from scratch and write their own start scripts in a mix of perl and bash. You don't need superhero sysadmin skills from the late 90s. All components are packaged with systemd units... there is a better way...

1. Install cgit and fcgiwrap.

fcgiwrap will create a socket NGINX can use to pass the CGI variables to cGit:

sudo apt install fcgiwrap
sudo apt install cgit

2. Modify the cgitrc file under /etc/cgitrc to your liking:

# See cgitrc(5)
# prepend this string to every url
virtual-root=/
enable-index-links=1
enable-commit-graph=1

root-title=My Git Repos
root-desc=I exclusivly write code in Smalltalk-71
logo=/assets/my_custom_logo.png

# Add site-specific configuration
# ...

3. Optionally create an assets directory and add your custom logo / css:

mkdir /var/www/html/assets
cp my_custom_logo.png /var/www/html/assets

4. Configure NGINX

Add the site to NGINX. This launches the cgit.cgi executable passing it to the fcgiwrap socket:

echo >/etc/nginx/sites-available/cgit.conf <<EOF
server {
    listen 80;

    server_name  git.domain.com;
    server_name  www.git.domain.com;

    root /usr/share/cgit;

    # Maintainer overridden assets will live in /assets
    # This allows you to change add a custom logo or modified CSS
    # See cgitrc(5)
    location ~* /assets {
        root /var/www/html;
        expires 30d;
    }

    # Fallback to static assets included by cGit 
    location ~* ^.+\.(css|png|ico)$ {
        root /usr/share/cgit;
        expires 30d;
    }

    try_files $uri @cgit;

    location @cgit {
        fastcgi_param   SCRIPT_FILENAME /usr/lib/cgit/cgit.cgi;
        fastcgi_param   PATH_INFO       $uri;
        fastcgi_param   QUERY_STRING    $args;
        fastcgi_param   HTTP_HOST       $server_name;
        fastcgi_pass    unix:/run/fcgiwrap.socket;
    }

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log warn;
}
EOF

4. Enable the site:

ln -s /etc/nginx/sites-available/git.conf /etc/nginx/sites-enabled/cgit.conf

Note: all files in sites-enabled should be included in nginx.conf's http section:

include /etc/nginx/sites-enabled/*;

5. Restart NGINX

sudo systemctl restart nginx