in Linux

What’s In Amazon Linux & Why Might You Use It?

Just before Christmas, Amazon announced the release of Amazon Linux 2. In addition to the usual userland modernization (switching to systemd, providing a newer glibc ABI) and the Linux 4.9 kernel, you can now run it on premise; images are provided for VMware, Hyper-V and KVM. Sometime in 2018, Amazon will also start providing long-term support for Amazon Linux, similar to RedHat Enterprise Linux (RHEL), by promising to support ABI compatibility for at least five years.

I’ve always been somewhat puzzled by Amazon offering their own Linux distribution to customers. No other cloud provider does this, although Google runs their own Debian derivative on non-customer-accessible systems. (For more on that, the slides from a talk they gave in 2013 makes for fascinating reading, in part because live-upgrading a 10-year-old base OS to a newer one while changing from .rpm to .deb is… terrifying yet awesome.) I decided to take a pretty deep dive into Amazon Linux 2’s feature set to see if there’s a compelling reason I could see for customers to adopt it.

What’s In the Distribution Itself?

A Linux distribution consists of a kernel version plus a userland — all the user-facing utilities, and system services (such as a boot loader, init system, package manager) needed to support applications. Amazon Linux 2’s userland appears to be a lightly-modified version of CentOS/Fedora; all the packages have amzn2 in their names (instead of, say, el7 or f24) but the changelogs show that the upstreams are RedHat/Fedora.

Unless Amazon’s engineers are omitting changes they make from their changelogs, very little of the userland deviates from CentOS/Fedora. It’s purely a recompilation against glibc 2.25 and binutils 2.27 using a newer GCC. They do install several of their own packages, like the amazon-ssm-agent package and aws-cfn-bootstrap to make interoperability with AWS services like SSM and CloudFormation possible by default, although you can easily install these packages on any generic Linux distribution yourself.

Amazon Linux does configure a few system services to work optimally against Amazon services. Again, these are nice-to-haves & you could easily do this for any Linux distribution using a configuration management system like Chef, but it’s good to see that Chrony is configured by default to use the new Amazon Time Sync Service. cloud-init is installed by default, but this is true for any modern Linux distro cloud image.

Like RHEL7, the default filesystem is xfs.

Extras Library

Amazon Linux 2 includes an “Extras Library”, which is the equivalent of Software Collections on RHEL or Fedora. These types of systems allow for newer versions of application runtimes like PHP or Python to be installed on a base OS that is locked to an older version.

I’m not entirely sure why Amazon decided to write their own system rather than just using Software Collections. I haven’t yet been able to find any documentation providing a rationale.

At the time of writing, there are only 14 extras available:

 0  ansible2  disabled  [ =2.4.2 ]
 1  emacs   disabled  [ =25.3 ]
 2  memcached1.5   disabled  [ =1.5.1 ]
 3  nginx1.12   disabled  [ =1.12.2 ]
 4  postgresql9.6   disabled  [ =9.6.6 ]
 5  python3   disabled  [ =3.6.2 ]
 6  redis4.0   disabled  [ =4.0.5 ]
 7  R3.4   disabled  [ =3.4.3 ]
 8  rust1   disabled  [ =1.22.1 ]
 9  vim   disabled  [ =8.0 ]
10  golang1.9   disabled  [ =1.9.2 ]
11  ruby2.4   disabled  [ =2.4.2 ]
12  nano   disabled  [ =2.9.1 ]
13  php7.2   disabled  [ =7.2.0 ]
14  lamp-mariadb10.2-php7.2   disabled  [ =10.2.10_7.2.0 ]

Personally, though, I find systems like Extras or Software Collections to not be widely adopted or frankly, that useful. The industry trend is for application developers to ship their software with everything built in, including the runtime interpreter and all of the application’s dependencies, hence the increasing popularity of languages like Go or Rust where everything is built into a portable binary that can run anywhere with no runtime dependencies. The fundamental problem of application developers being locked to some ancient version of PHP, Ruby, Python, etc. is not solved by Software Collections, so these days most developers take matters into their own hands in order to move at speed, regardless of the level of control that traditional infrastructure administrators would prefer — and developers win in the end on this front.

The Kernel

Amazon Linux 2 ships with Linux kernel 4.9.62. Released in December 2016, the 4.9-series is Linux’s LTS kernel and at the time of its release, was the “biggest cycle ever” for the kernel project. The biggest feature in 4.9 was Greybus but that doesn’t have much relevance to VMs running in the cloud. Nevertheless, 4.9 includes lots of improvements for filesystems (btrfs, xfs, ext4) as well as the usual plethora of device driver changes.

Amazon Linux carries a set of patches to the upstream distribution kernel and I thought it would be interesting to examine what they are & what they do, so I downloaded Amazon’s kernel source RPM and read all 56 patches they apply. For the most part, they are unremarkable: performance optimizations to Xen drivers like xen-blkfront, as well as to work around a variety of bugs in PM suspend/resume and so on. Despite their purported reputation for consuming open source and not contributing back, the Amazon kernel developer team seems to be pretty good about sending most of their patches upstream, with the exception of a few items that they consider to not ready for public consumption because they’re currently hacky or AWS-specific.

Amazon also patches the kernel’s driver tree to ship some additional drivers. First is a newer version of the ixgbevf driver, which is the virtual function driver for SR-IOV Intel 10 Gbps PCIe Ethernet cards. ixgbevf exists in the distribution kernel tree, but the newer version presumably allows Amazon to work around bugs in the upstream. Not much to see here. Note that SR-IOV is only available on certain Amazon instance types; Amazon calls this feature “Enhanced Networking”.

AWS later introduced their own Elastic Network Adapter (ena) still under the branding of Enhanced Networking. This is still an SR-IOV device but providing up to 25 Gbps network speeds. It’s supported by a different set of instance types. Since this is Amazon’s own hardware, they had to write their own driver for it, which isn’t included in the upstream kernel tree, so they package it with their kernel distribution. (The source code is freely available for other distributions.)

Before I dispense with the kernel, there is one last patch of note that gives a hint to Amazon’s future plans: providing a serial port whose output will show up in the EC2 console log, and can be retrieved with GetConsoleOutput. Cool!

Conclusion: Should You Use Amazon Linux?

I admit that reading and understanding kernel patches satisfied my inherent nerdiness for a few minutes. Putting my product marketing hat back on: should you use Amazon Linux?

For general-purpose workloads, Amazon Linux offers no substantial advantages over a more mainstream distribution like CentOS, Ubuntu or RHEL, and a significant downside in that its userland deviates from CentOS and Fedora. Out-of-the-box vendor software built for, say, RHEL is unlikely to run because of the differences in glibc and binutils from RHEL6 or RHEL7. And, at present, until Amazon works out all the kinks in its on-premise versions of the Amazon Linux image, it’s still harder to have non-cloud environments for development purposes.

The one area where Amazon Linux is beneficial is if you are using high-performance instances i.e. the ones that have Enhanced Networking. Compiling and installing custom kernel drivers onto cloud machines is a pain in the ass; particularly if you are using the Elastic Network Adapter on a generic RHEL system, RedHat will get really annoyed at you if you load a “non-certified” kernel module into their kernel.

Why, then, is Amazon investing in Amazon Linux? I continue to be perplexed by this question for which I don’t have a good hypothesis. There’s not sufficient user benefit to justify it, and the downsides are large. My only theory at this point is that Amazon continues to offer it to customers and maintain it in preparation for deeper integration with their new Nitro hypervisor (currently available on C5 instances), which is a fork of KVM. It’s possible they will need driver support or patches to KVM that the upstream kernel won’t ever carry because of Nitro’s proprietary nature.

If you’re using Amazon Linux, I’d love to know why. And if you have any thoughts on it and/or any information or even rampant speculation about why Amazon invests in it, please feel free to comment on this post.