Arch’s Dirty Little Not-So-Secret

UPDATE: Please note that this article is several years old and should be viewed as history at this point. While I still have reservations about the handling of security in Arch Linux, and I think this article is worth reading if you use Arch, they do now use package signing.

A reader of my blog recently made a comment about Arch’s lack of package signing, and this got me looking into the issue more carefully. What I found has left me deeply concerned with a number of aspects of Arch.

Most distributions, even Windows, sign their packages so that when the computer downloads and installs them, it can check the signature to make sure the package is authentic – it hasn’t been tampered with on the server, or anywhere between the server and the local system. This mechanism has been around for many years and works well – the tools to implement it are available and simple to use. Yet for some reason I can’t understand, Arch Linux has never had package signing. Arch packages are simple tarballs – they can be opened, modified, and retarred, and the updating system has no way to detect this. This tampering can take place on one of the many mirrors that host Arch packages, yet it can also take place elsewhere – in network proxies and misdirection, in intranet caches, and on local systems. Package signing gives admins a way to verify that the packages they’re using to update their system are authentic, regardless of how those packages have been delivered or stored, or who has access to the data.

Package signing is well understood in security circles as vital. Without it, a single mirror server can be compromised, which allows data on that server to be changed. Perhaps a malicious version of a program could be compiled and placed on the server. Any systems using that mirror for updates will download and install that compromised program, with no way to detect that it was modified. The malicious program may never be detected, especially on Linux where there is rarely virus protection in place. The system is now infected at the root level.

For example, if you use the mirror at ftp.tku.edu.tw, you are downloading your Arch updates from an Apache FTP server running on an Ubuntu machine. Thus your system becomes vulnerable to security problems in Ubuntu or caused by misconfigurations on that server.

As another example, a network can be compromised such that an admin downloads updates from a false repository. This doesn’t require any tampering with Arch’s official mirrors. The system may attempt to download from ftp.archlinux.org, but the requests will be redirected to another server with tampered packages. Without package signing, there is no way for the system to know what server it has actually connected to, or that the packages are not authentic. It then installs compromised packages or even a kernel that can open the server to any form of attack. One network vulnerability has now translated into an undetected infection at the root level. While this may sound like too much trouble, it is a real threat to server admins, and setting up a false repository is trivial. All the data on Arch’s mirrors are sent without SSL – meaning this data is vulnerable along its entire network route. There is a trivial method to stop this kind of man in the middle attack: SIGN YOUR PACKAGES.

Arch Linux grew from a small circle of developers, where package signing was probably not seen as necessary. Yet with the growth of Arch they have failed to update their security model, and like Sourceforge’s recent attack, I think Arch is a sitting duck for a major security compromise, if it hasn’t silently occurred already, or occurred in local incidents. Because of the lack of package signing, any mirror intrusion will propagate to all Arch systems silently. There are currently over 150 Arch mirrors located all over the world – at universities, private companies, and other sites.

Users on the Arch Linux forum were contentiously divided on this issue, to the point where it became impossible to discuss it there. No one likes to have their vulnerabilities pointed out, which leads to denial and anger. While some saw the importance of it, and have been discussing it on and off for months, others reacted with bizarre fury that it was being discussed. Finally the moderators closed the thread and tucked it away in a section of the forum that only subscribers can access. I have seen this behavior before on the Arch forums – it is not a very open forum for discussing anything even slightly contentious. But hiding and censoring important security discussion was a new low even for them. As a result, many Arch users will simply not be exposed to the fact that their distribution has a major security problem.

There is a very slow, low priority effort underway to add package signing. For many users, it seemed enough to know that ‘somebody is working on it’. But it was obvious from the dates that little was actually being done. I then spoke with some of the developers, and what I found there disturbed me even more.

Several developers had a cavalier and careless attitude about their users’ security in general, and I find attitude is very important when it comes to developing security mechanisms. I like to know the devs behind the system I use take that responsibility seriously. This was not what I was expecting to see. Many of the developers also displayed either ignorance or a poor understanding of the importance of package signing, and even other security practices in general [bold below mine]:

Alf Gaida: “For me it makes not a big difference if a package is signed or not. It’s a nice to have feature and I would be glad if someone would implement it. But for me it has a very low priority.”

Loui Chang: “Archers deserve to die! But really I’m not convinced by this hyper-paranoia trash. There will always be ways to compromise your machine… You will never be safe.”

Allan McRae: “I am responsible for nothing. I only choose to pull together the package signing patches in my spare time… Contracting my services would actually motivate me to implement this. My standard consultancy rate is USD1000 per day or part thereof… Minor performance issues interest me a hell of a lot more than package signing.”

Keep in mind these are the people allegedly responsible for system security in Arch, in that they are the developers of package distribution and other systems. Allan Rae in particular has done most of the work on the package manager’s signature checking code thus far, still under lax development. Needless to say his attitude does not fill me with confidence that he takes his users’ security seriously in the slightest.

There were a few developers that rose above that level and seemed to understand the importance of package signing, and good security practices in general. Denis A. Altoé Falqueto and Pierre Schmitz took the concerns I raised seriously, and immediately went to work on makepkg’s signature code.

Daniel Mendler: “The mail by IgnorantGuru is very much what I was going to write. There is no problem in adding signatures to the Arch repositories immediately… [To Allan McRae:] You seem to have a working implementation but you don’t integrate these into master. Instead you work on minor performance issues… even though we have a very serious security problem.” Elsewhere he writes: “It makes a big difference if your system is compromised. And then you will care about it. I don’t understand this naive and short-sighted opinion… I am a bit disappointed with your opinion that you want to implement only features that you care about. I think there is also a reponsibility if you are one of the main developers of the package manager of a popular distribution. And you don’t even have to implement the features yourself – there are people who are willing to help. But those people should also get some support by you.”

Denis A. Altoé Falqueto: “This is a standard open source community, there’s just momentum when there is personal interest. In the other hand, I kind of enjoy your stimulus… Sometime I need some push over to make me do things.” (He immediately resumed working on his contributions to package signing.)

Pierre Schmitz: “Once there is a version of pacman which supports signed packages I can
start implementing these ideas. And last but not least we need to think about key management which is less technical but very important.”

Xavier Chantry was rude, but did concede, “it would be nice to have signatures on the mirrors already, regardless of what pacman and repo-add support. And well, we agree…”

My main point to the developers was that even before the VERY slow-moving changes to automated pacman signature checking is complete, it would be very valuable and technically trivial to add signatures on the mirrors. This would allow simple scripts to be written which could check the signatures, at least until pacman can do so. There is simply no reason not to add the signatures immediately, and I didn’t encounter any argument to the contrary. Yet there was some debate about how that can get done. It seems many of the primary Arch devs are against package signing, or at least are resistant to adding this code to the final releases. Thus even the small changes the developers above make tend to be held in limbo on git servers, never seeing real use. Meanwhile, the data on the mirrors sits completely vulnerable.

Overall, it was encouraging to see some work being done on this now in Arch, and the developers did seem to be open to the suggestion of getting the signatures onto the servers ASAP – hopefully that will see reality. Yet I was disappointed, to put it mildly, with the careless attitude of many of the developers. This does not bode well for the work they will be doing, if they even do it. I am also unimpressed with the response of the Arch community in general to this issue – there seemed to be more energy toward ridiculing the messenger and hiding the problem from their users than addressing it in a responsible and positive way. Often Arch users and admins show a high degree of knowledge, but in this case the people who understand the problem seemed virtually silent on it, and the ones who don’t understand seemed very angry that it was being discussed.

I will indeed keep an eye on this. If they take the wise and painless step of adding the signatures to the mirrors, I will seriously consider writing a script to check the signatures in a users cache, such that they can download packages in pacman, run a script to check them, then install them with pacman. It’s not so difficult, but it does require the signatures.

In the meantime, I’m considering writing a script that will grab the db files containing the (unprotected) MD5 sums from several servers, compare them, and use them to check packages. This polling is far from an ideal security solution, yet even it will improve security drastically. Unfortunately, from what I understand, even the MD5 files are poorly done, such that package versions are not tabulated with them (meaning that if an MD5 sum mismatches, it’s hard to tell whether the mirror is simply out of sync or has been tampered with). But I’ll see what I can find in that area to use. UPDATE: paccheck is now available for this purpose.

Nevertheless, this starts entering an area like Windows, where half measures and fighting against the OS is required to ensure reasonable security, instead of the developers of the OS acting in their users’ best interests. I am reminded of the Microsoft mentality where reports of security problems are ignored, until finally a very destructive virus compels them to take action – too late. Usually when things reach this point I reach for a different OS (suggestions welcome), which this issue now has me considering. Yet there are many things done well in Arch, so it is my hope they begin to address this vulnerability responsibly, and place responsible, knowledgeable developers in charge of their security mechanisms.

UPDATE: My Move From Arch To Aptosid