IgnorantGuru's Blog

Linux software, news, and tips

Arch’s Dirty Little Not-So-Secret

UPDATE: Please note that this article is several years old and should be viewed as history at this point. While I still have reservations about the handling of security in Arch Linux, and I think this article is worth reading if you use Arch, they do now use package signing.

A reader of my blog recently made a comment about Arch’s lack of package signing, and this got me looking into the issue more carefully. What I found has left me deeply concerned with a number of aspects of Arch.

Most distributions, even Windows, sign their packages so that when the computer downloads and installs them, it can check the signature to make sure the package is authentic – it hasn’t been tampered with on the server, or anywhere between the server and the local system. This mechanism has been around for many years and works well – the tools to implement it are available and simple to use. Yet for some reason I can’t understand, Arch Linux has never had package signing. Arch packages are simple tarballs – they can be opened, modified, and retarred, and the updating system has no way to detect this. This tampering can take place on one of the many mirrors that host Arch packages, yet it can also take place elsewhere – in network proxies and misdirection, in intranet caches, and on local systems. Package signing gives admins a way to verify that the packages they’re using to update their system are authentic, regardless of how those packages have been delivered or stored, or who has access to the data.

Package signing is well understood in security circles as vital. Without it, a single mirror server can be compromised, which allows data on that server to be changed. Perhaps a malicious version of a program could be compiled and placed on the server. Any systems using that mirror for updates will download and install that compromised program, with no way to detect that it was modified. The malicious program may never be detected, especially on Linux where there is rarely virus protection in place. The system is now infected at the root level.

For example, if you use the mirror at ftp.tku.edu.tw, you are downloading your Arch updates from an Apache FTP server running on an Ubuntu machine. Thus your system becomes vulnerable to security problems in Ubuntu or caused by misconfigurations on that server.

As another example, a network can be compromised such that an admin downloads updates from a false repository. This doesn’t require any tampering with Arch’s official mirrors. The system may attempt to download from ftp.archlinux.org, but the requests will be redirected to another server with tampered packages. Without package signing, there is no way for the system to know what server it has actually connected to, or that the packages are not authentic. It then installs compromised packages or even a kernel that can open the server to any form of attack. One network vulnerability has now translated into an undetected infection at the root level. While this may sound like too much trouble, it is a real threat to server admins, and setting up a false repository is trivial. All the data on Arch’s mirrors are sent without SSL – meaning this data is vulnerable along its entire network route. There is a trivial method to stop this kind of man in the middle attack: SIGN YOUR PACKAGES.

Arch Linux grew from a small circle of developers, where package signing was probably not seen as necessary. Yet with the growth of Arch they have failed to update their security model, and like Sourceforge’s recent attack, I think Arch is a sitting duck for a major security compromise, if it hasn’t silently occurred already, or occurred in local incidents. Because of the lack of package signing, any mirror intrusion will propagate to all Arch systems silently. There are currently over 150 Arch mirrors located all over the world – at universities, private companies, and other sites.

Users on the Arch Linux forum were contentiously divided on this issue, to the point where it became impossible to discuss it there. No one likes to have their vulnerabilities pointed out, which leads to denial and anger. While some saw the importance of it, and have been discussing it on and off for months, others reacted with bizarre fury that it was being discussed. Finally the moderators closed the thread and tucked it away in a section of the forum that only subscribers can access. I have seen this behavior before on the Arch forums – it is not a very open forum for discussing anything even slightly contentious. But hiding and censoring important security discussion was a new low even for them. As a result, many Arch users will simply not be exposed to the fact that their distribution has a major security problem.

There is a very slow, low priority effort underway to add package signing. For many users, it seemed enough to know that ‘somebody is working on it’. But it was obvious from the dates that little was actually being done. I then spoke with some of the developers, and what I found there disturbed me even more.

Several developers had a cavalier and careless attitude about their users’ security in general, and I find attitude is very important when it comes to developing security mechanisms. I like to know the devs behind the system I use take that responsibility seriously. This was not what I was expecting to see. Many of the developers also displayed either ignorance or a poor understanding of the importance of package signing, and even other security practices in general [bold below mine]:

Alf Gaida: “For me it makes not a big difference if a package is signed or not. It’s a nice to have feature and I would be glad if someone would implement it. But for me it has a very low priority.”

Loui Chang: “Archers deserve to die! But really I’m not convinced by this hyper-paranoia trash. There will always be ways to compromise your machine… You will never be safe.

Allan McRae: “I am responsible for nothing. I only choose to pull together the package signing patches in my spare time… Contracting my services would actually motivate me to implement this. My standard consultancy rate is USD1000 per day or part thereof… Minor performance issues interest me a hell of a lot more than package signing.”

Keep in mind these are the people allegedly responsible for system security in Arch, in that they are the developers of package distribution and other systems. Allan Rae in particular has done most of the work on the package manager’s signature checking code thus far, still under lax development. Needless to say his attitude does not fill me with confidence that he takes his users’ security seriously in the slightest.

There were a few developers that rose above that level and seemed to understand the importance of package signing, and good security practices in general. Denis A. Altoé Falqueto and Pierre Schmitz took the concerns I raised seriously, and immediately went to work on makepkg’s signature code.

Daniel Mendler: “The mail by IgnorantGuru is very much what I was going to write. There is no problem in adding signatures to the Arch repositories immediately… [To Allan McRae:] You seem to have a working implementation but you don’t integrate these into master. Instead you work on minor performance issues… even though we have a very serious security problem.” Elsewhere he writes: “It makes a big difference if your system is compromised. And then you will care about it. I don’t understand this naive and short-sighted opinion… I am a bit disappointed with your opinion that you want to implement only features that you care about. I think there is also a reponsibility if you are one of the main developers of the package manager of a popular distribution. And you don’t even have to implement the features yourself – there are people who are willing to help. But those people should also get some support by you.”

Denis A. Altoé Falqueto: “This is a standard open source community, there’s just momentum when there is personal interest. In the other hand, I kind of enjoy your stimulus… Sometime I need some push over to make me do things.” (He immediately resumed working on his contributions to package signing.)

Pierre Schmitz: “Once there is a version of pacman which supports signed packages I can
start implementing these ideas. And last but not least we need to think about key management which is less technical but very important.”

Xavier Chantry was rude, but did concede, “it would be nice to have signatures on the mirrors already, regardless of what pacman and repo-add support. And well, we agree…”

My main point to the developers was that even before the VERY slow-moving changes to automated pacman signature checking is complete, it would be very valuable and technically trivial to add signatures on the mirrors. This would allow simple scripts to be written which could check the signatures, at least until pacman can do so. There is simply no reason not to add the signatures immediately, and I didn’t encounter any argument to the contrary. Yet there was some debate about how that can get done. It seems many of the primary Arch devs are against package signing, or at least are resistant to adding this code to the final releases. Thus even the small changes the developers above make tend to be held in limbo on git servers, never seeing real use. Meanwhile, the data on the mirrors sits completely vulnerable.

Overall, it was encouraging to see some work being done on this now in Arch, and the developers did seem to be open to the suggestion of getting the signatures onto the servers ASAP – hopefully that will see reality. Yet I was disappointed, to put it mildly, with the careless attitude of many of the developers. This does not bode well for the work they will be doing, if they even do it. I am also unimpressed with the response of the Arch community in general to this issue – there seemed to be more energy toward ridiculing the messenger and hiding the problem from their users than addressing it in a responsible and positive way. Often Arch users and admins show a high degree of knowledge, but in this case the people who understand the problem seemed virtually silent on it, and the ones who don’t understand seemed very angry that it was being discussed.

I will indeed keep an eye on this. If they take the wise and painless step of adding the signatures to the mirrors, I will seriously consider writing a script to check the signatures in a users cache, such that they can download packages in pacman, run a script to check them, then install them with pacman. It’s not so difficult, but it does require the signatures.

In the meantime, I’m considering writing a script that will grab the db files containing the (unprotected) MD5 sums from several servers, compare them, and use them to check packages. This polling is far from an ideal security solution, yet even it will improve security drastically. Unfortunately, from what I understand, even the MD5 files are poorly done, such that package versions are not tabulated with them (meaning that if an MD5 sum mismatches, it’s hard to tell whether the mirror is simply out of sync or has been tampered with). But I’ll see what I can find in that area to use. UPDATE: paccheck is now available for this purpose.

Nevertheless, this starts entering an area like Windows, where half measures and fighting against the OS is required to ensure reasonable security, instead of the developers of the OS acting in their users’ best interests. I am reminded of the Microsoft mentality where reports of security problems are ignored, until finally a very destructive virus compels them to take action – too late. Usually when things reach this point I reach for a different OS (suggestions welcome), which this issue now has me considering. Yet there are many things done well in Arch, so it is my hope they begin to address this vulnerability responsibly, and place responsible, knowledgeable developers in charge of their security mechanisms.

UPDATE: My Move From Arch To Aptosid

February 19, 2011 - Posted by | Tips


  1. Nobody’s forcing anyone to use Arch. If I don’t like something, I leave it.
    As you said yourself, the lack of package signing, it’s not really a secret.

    Patches welcome, as always.

    Comment by karol | February 19, 2011 | Reply

    • What a pathetic reply. If your packages aren’t signed, your distro is nothing more than a toy. It’s great that you little hobbyist freaks have a nice little hobby distro.

      Comment by Ryan | October 26, 2012 | Reply

  2. I just want to leave my thoughts on this (and a few other things)

    First, over the years there have been _many_ people complaining over the lack of package signing. So far _none_ of them have actually cared enough to do something about it, and that is the real problem. Ranting doesn’t help fixing problems, contributing does.

    If there had been less ranting and more contributing, package signing would have been implemented years ago. If anyone had offered to become a dev and impliment/maintain package signing, then there would be a fairly high chance that they would have been accepted.

    Which brings me to my second tought:
    Archlinux and pacman is developed entierly by people deciding to spend their spare time on a project they like, doing things they want to do. No one is getting paid for the work, and no one is forcing anyone to do something that they don’t want to do. If tomorrow there was a new management that said: “you must impliment foo” or “you must do bar”, then I’m fairly sure that a good portion of the developers quickly would loose the motivation to work on arch and pacman, and go off to something else that intrests them.

    Archlinux was not made for you, neither was it made for me. It was made by the developers for the developers, that quite a few other people happens to use it is just a happy coincidence.

    And then some sidetoughts.

    I’ve been using arch for quite a few years. I started using it as my main distro back in the 0.6/0.7 days, that is back in 2003/2004 or something like that.
    My general observation is that, sadly, as arch has become more popular, and have gotten a larger userbase, sadly there seems to have been a decline in the (relative) amount of people willing to actually contribute. Arch has always been driven forward by the community, one of the best examples of this is the x86_64 port. It started off pretty much as a repo set up and maintained by a few users, and was later intigrated officially into the archlinux project.
    But these days it looks like a larger portion of the community, specially among the newer converts, that seems to think that they have a right to use arch, and that the developers should be at their bidding. And the worst thing of all is that I have not only seen this in arch, but in the whole society as a whole, across the whole world.

    Finally I want to say this:
    I’m not pointing fingers at anyone specific, and I’m certainly not the perfect user who always offers everything he/she/it has for the greater good. But I want to get this of my heart: “If you want something done, do it yourself” and “stop whining and do something about it”. The world would be a much better place if the majority of us wasn’t quite so egoistic and self-centered. (and we would have had package signing implimented years ago :p)

    Oh and as a final final comment. You do have some twisted truths and halftruths, bad out of context quoting and twisted viewpoint in the blogpost, but it’s 03:06 here and I’m too tired to go trough and respond to/correct it all. (as I said, I’m far from perfect)(and yes, my discussion skills just plain suck, like many of us I’m not good at putting my words down on paper/bits/stone tablets)

    Comment by Mr.Elendig | February 19, 2011 | Reply

    • But these days it looks like a larger portion of the community, specially among the newer converts, that seems to think that they have a right to use arch, and that the developers should be at their bidding. And the worst thing of all is that I have not only seen this in arch, but in the whole society as a whole, across the whole world.

      Come on, have you ever heard about the division of labour? If someone doesn’t wish to spend time polishing your favourite distribution it soesn’t mean that this person is sitting on his ass, doing nothing but parasitizing on a sickly body of human society.

      Your altruism is great, I honestly think so. Just don’t oversimlify things.

      Comment by pipy | February 20, 2011 | Reply

    • Mr. Elendig:

      I also see the decline in Arch you mention, but from what I’ve experienced I think it goes deeper than a lack of contributors. There seems to be a lack of a general plan, and a security plan in particular. Simply put, they just don’t have it together, and the attitude of many of the developers is not what I like to see.

      I disagree regarding your specious comment on twisting the truth or quoting out of context. I’m not trying to fool anyone. I made an effort (what you claim is needed more in Arch) to look into this issue and even improve it, and I shared what I found. I included a few quotes to characterize the overall conversations, and I believe I did well with that. Yet these conversations are public in their entirety, so those interested can read them. If you doubt my analysis of Arch’s lack of package signing and the implications, do your own research.

      I believe I will be leaving Arch with all due haste, aiming for Gentoo, Slackware, or FreeBSD. So this is likely my last contribution to Arch, and I do believe bringing security issues to light and open discussion is important.

      Comment by igurublog | February 20, 2011 | Reply

      • Then stop ranting and offer to work on it.

        When you go to a restaurant you PAY. When you cook for yourself you DO THE WORK. There is no way I know of to get a free lunch by ranting and pointing out the obvious. Let me know if you find one.

        Comment by Craig | June 18, 2012 | Reply

    • Debian is also all-volunteer and they have package signing. It’s important!

      Comment by Eric Mesa | March 1, 2011 | Reply

    • It must be frustrating for developers to hear criticism from people who are not actively contributing to the development of the distro. I think that the users really *do* appreciate the work the developers have done. However, as the Arch userbase gets larger it will also grow more diverse. There are people who cannot contribute to the development, but can contribute in other ways. If an open discussion were encouraged on the forums, I don’t think it would devolve into a 10 page b*tchfest, but rather a resource for overcoming what is obviously a major flaw in Arch. There are solutions in the AUR, as mentioned above, to temporarily fix the security hole and it would be nice if there were a sticky or something explaining how to implement a stop-gap solution to tide users over until the package signing is implemented. Quickly censoring the discussion yields no benefits however.

      It is late here also so I might be a bit incoherent, but you get the idea :P

      Comment by MycoRunner | March 7, 2011 | Reply

  3. According to your Arch forum sidebar, you’ve been using Arch for 1-1/2 years. I, on the other hand, have been using Arch for four months. How is it that I knew about this before ever installing Arch and you just found out about it? And as I mentioned on the forums already, how many problems has it actually caused for you? This happens to be your blog, so I don’t necessarily have to abide by any community standard. Therefore, I submit to you that you, sir, are a fuckin’ jackass, and you’ve got nothing better to do with your time than complain about a nonexistent problem that you have the power to fix but won’t because you like the sensation of a clenched assholoe way too much. ‘Nuff said.

    Comment by ANOKNUSA | February 19, 2011 | Reply

    • you’ve got nothing better to do with your time than complain about a nonexistent problem that you have the power to fix

      Raising the problem is a first step to solve it. On the other hand, pointless swearing really solves nothing. I hope you’ve had a good time pumping up your mentally unbalanced ego. Because that’s the only thing you’ve did here.

      Comment by pipy | February 20, 2011 | Reply

      • The problem has already been raised; that’s why your posts keep getting closed. You’re not helping a damn thing, only bitching about a hitherto minor problem while failing to put forth any effort in finding a solution.

        Comment by ANOKNUSA | February 20, 2011 | Reply

        • The problem has already been raised; that’s why your posts keep getting closed. You’re not helping a damn thing, only bitching about a hitherto minor problem while failing to put forth any effort in finding a solution.

          I’m a detached observer, neither I use Arch nor I post to it’s mailing lists. I need secure and stable systems, that’s why I use FreeBSD/Debian on my servers and Ubuntu LTS on desktops.

          Comment by pipy | February 21, 2011 | Reply

          • “I need secure and stable systems, that’s why I use FreeBSD/Debian on my servers and Ubuntu LTS on desktops.”

            The Debian that gets constantly owned for heavily patching by clueless packagers? Like OpenSSH?.

            Comment by pffff | February 28, 2011 | Reply

          • I think you mean OpenSSL, not OpenSSH and I’m not sure if Debian folks were the only ones at fault:

            Comment by karol | February 28, 2011 | Reply

          • You mean the only one. That’s not constantly, that’s once. If it would happen every week or month, and major breakage would happen everytime, then your point would be valid. But this is just a generalisation.

            On the other side there is no metric for how many times Archlinux got owned because of a lack of package signing.

            Comment by Anonymous | March 6, 2011 | Reply

      • “hitherto minor problem”

        Sorry, but lack of signing is *not* a “minor problem”, nor a recent one. Security and trust are important, and if that’s the general attitude of the Arch developers, that’s a very bad sign, and not at all professional.

        Take a look at how other distributions, who all generally take these things seriously, handle it. Gentoo and Debian are good examples. But Fedora and everyone else all sign their stuff, and have done for over a decade.

        Every Debian developer has a key used to sign *all* uploaded files for distribution. I’m using an older 1024-bit DSA/ElG key, but I am switching to a 4096-bit RSA key soon, like most others. (This is to ensure older, weaker keys, are phased out a long time before they become insecure.) These are used to verify the origin and integrity of *all* uploads. There are then central release keys used to sign Release files which contain the MD5, SHA1 and SHA256 digests of all the Contents/Packages/Sources files, which in turn contain the MD5, SHA1 and SHA256 digests of individual package files. [MD5 will be dropped in the future and newer, more secure, hashes will be adopted.] Anyone downloading a package from a mirror can use this to guarantee the integrity and origin of any package. And the standard tools all check package signatures, so all end users are protected against tampering *by default*. And this isn’t restricted to end user systems; the automated build infrastructure checks source package signatures and package signatures as well to prevent compromises via this route.

        To not care about this is akin to selling a car without working brakes, in the full knowledge that the brakes are faulty. You can’t then tell the driver it was their fault when they came to a hill and crashed [had their system rooted due to a compromised mirror]. Distributors have a responsibility to their users, whether they wish to acknowledge it or not. You’re by definition distributing that software, and have a responsibility to make sure it’s not tampered with between you and the end user’s system. Not having *any* means of allowing users to verify the integrity of downloaded files, let alone not doing this automatically, is the height of irresponsibility. How can I (or anyone) trust Arch as a distribution without this?

        The tools to implement secure signing are readily available (GPG) and are well understood and used by everyone else. The effort required to start using the tools is minimal.


        Comment by Roger Leigh | March 1, 2011 | Reply

        • I’d say that Debian decades-old signing practices need some facelift too. I have already posted a link to an extensive analysis of package management vulnerabilities. This is an especially nice part:

          To give an example of how easy it is for a malicious party to obtain a mirror, we ran an experiment where we created a fake administrator and company name and leased a server from a hosting provider. We were able to get our mirror listed on every distribution we tried (Ubuntu, Fedora, OpenSuSE, CentOS, and Debian) and our mirrors were contacted by thousands of clients, even including military and government computers!

          Comment by pipy | March 1, 2011 | Reply

          • There are absolutely things Debian can do better WRT signing and security–things are always evolving as our understanding of what requirements we need to maintain a secure system and infrastructure changes. Metadata expiration and replay attacks are a very good example. However, I think such features are now implemented, or at the very least on the way to being fully implemented. From the current unstable InRelease file:

            Hash: SHA1

            Origin: Debian
            Label: Debian
            Suite: unstable
            Codename: sid
            Date: Tue, 01 Mar 2011 08:22:01 UTC
            Valid-Until: Tue, 08 Mar 2011 08:22:01 UTC

            So the root metadata comes with both a creation date and expiration date (one week later) after which it is no longer considered valid. I’m not sure if the tools use the information yet, and if they do if it’s turned on by default, but the information is now there for tools to make use of it. This prevents replay attacks after a period of one week.

            One key thing to mention is that this sort of security infrastructure evolved over time, and was not done overnight. Uploads have been signed since forever, but release (download) signing is more recent (but still quite old now). There was also a delay between the signing being done, and the tools making use of it, and then the tools making use of it by default. Today, a new Debian install will be verifying everything from the start, but that was not always the case–you used to have to install the signing keys by hand and turn on verification; now that’s all done for you.

            From the point of view of implementing things in Arch, getting detached signatures on the distributed files is the first step. This is easy (e.g. signed sha256sums). Getting tool support is the next step, and it does not have to be done all at once. Going from no trust and no verification to a fully trusted, fully verified setup can be done one step at a time, making sure the infrastructure is working at each step.


            Comment by Roger Leigh | March 1, 2011 | Reply

          • @Roger Leigh:

            No. As this discussion clearly shows us, the whole idea of mirroring is clearly out of date.

            What we need is several SSL-secured torrent trackers for each distribution. They should be the ones to serve packages themselves and metadata.

            Ironically, as I am writing this, apt-get is complaining about the fact that it has just failed to get current package metadata from the mirror.

            E: Some index files failed to download, they have been ignored, or old ones used instead.

            Comment by pipy | March 2, 2011 | Reply

          • Sorry, torrent trackers were a not very good idea.

            I think that the most secure way to update your Debian system today is to use a good mirror and stream updates over Tor.

            Comment by pipy | March 2, 2011 | Reply

  4. ANOKNUSA, hello, I see you have the free time to read the article and post a flame. Can you summarize it for those of us who don’t? Dumbass.

    Mr.Elendig, spot on.

    IgnorantGuru, I’m not sure what you intend on accomplishing with these articles. You’re not adding anything to the discussian other than trivial details and gossip. It would be more useful to have some sort of assessment on how good the existing proposed patches are.

    Comment by fsckd | February 19, 2011 | Reply

    • Uh… what?

      Comment by ANOKNUSA | February 20, 2011 | Reply

  5. Don’t worry! It is opensource and there is space for everyone. Maybe Arch community isn’t really concerned about security. Don’t blame maintainers for their reluctance to deal with this issue — they’ve got their personal lives and interests. The best thing that they can do at this point is to put a big disclamer in the official installation guide. They need to openly acclaim that security is not their concern (there is nothing wrong with that, they just need to highlight it for Arch beginners). People who are concerned about security probably won’t install Arch at this point, but that’s OK, since there is a lot of other places to go.

    Probably, if you carefully select your download mirrors and don’t care about explicitly targeted attacks, you are pretty secure at this point. Arch configurations are really diverce between different users and ant it’s overall popularity is not very high, this makes it’s user community an unlikely target for malware.

    I’m neither a Linux specialist nor do I mean to insult anyone, but Arch mailing list gave me an impression that it’s rather some kind of hobby tinkering project rather than a serious distribution targeted for a production use or dealing with sensitive information.

    As for your personal experience getting in touch with maintainers and forum moderators — let’s face it, dealing with peolpe require special skills and they don’t always correlate with ability to produce good code. Arch forum policy is crearly wrong and looks quite childish. Tactic of denial won’t bring them benefit in the long term.

    By the way, I’m currently installing Gentoo in the VirtualBox — this distribution looks very decent and doesn’t seem to be as comlicated as I thought. Compile times seem tolerable too, but they may be small due to the fact that I have plenty of ram and VM sits on SSD drive.

    Comment by pipy | February 20, 2011 | Reply

  6. Thanks for your comments.

    A few clarifications:

    This is not a case of lack of manpower to address a problem. Adding signed packages involves two very trivial steps:

    a) Packagers make keys for themselves and upload them to the key servers (this is not something I or anyone else can do for them)

    b) A single line can be added to the packaging script to implement signing. This is a trivial and completely secure approach. The developers themselves admitted this could be done, and in fact suggested it, but as a group they’re simply unwilling to do it, or to allow anyone to do it.

    That is all that is required to add signatures to the mirrors, so it is simply a matter of unwillingness to do so, not a lack of contributors. The situation is about as irrational as it could get.

    As for signature checking, that requires a little more work on someone’s part. Yet automated signature checking is not a prerequisite for package signing. I for one would be happy to write a script to check signatures, and this would be quite effective, because scripts provide transparency. As for the code added to pacman for this, someone familiar with their emerging security protocols and with pacman’s code should implement this carefully. Much of this code has already been written – it’s just not getting pushed through to final release.

    Further, there were a number of developers who expressed willingness to contribute. Yet more influential developers in their hierarchy are not being cooperative with their efforts.

    Beyond this, I find their lack of knowledge and attention to security very unwise and unprofessional (and yes, freeware developers can be quite professional). Even if this one serious problem is addressed, I have lost respect for the primary developers of Arch. It’s like finding a single worm in a package of food – you know there are more. So I think a change of distribution is the best solution for me.

    As for why I am addressing this now – I simply never looked into it in detail before. There are many aspects to a distribution, and frankly it didn’t occur to me that any major distribution would be this ignorant of something as basic as package signing or some form of authentication. It was a poor assumption on my part.

    Gentoo looks promising – I’ll keep that on my list – thanks.

    Comment by igurublog | February 20, 2011 | Reply

    • Just as a matter of historical interest:
      E-mail about the state of an ebuild signing in Gentoo’s Portage package manager. Just notice an entirely opposite developer’s attitude!

      Here is an extensive article about security of distribution of Gentoo software — interesting reading.

      Comment by pipy | February 24, 2011 | Reply

      • Thanks. I’m knee-deep in freeBSD at the moment. I have Openbox and Firefox running – just trying to get my Brother laser printer working next, because if I can’t then its a show stopper. So far limited success with cups, but no printing yet. So far I like freeBSD and if the printer works I may go for it, but its not linux, so there are some differences, including more limited hardware support.

        From what I can gather, FreeBSD doesn’t sign their packages either, which surprised me, but I can’t find much info on it – maybe there is an authentication method I’m missing, and they seem to control the mirrors more tightly. The ports are signed, and that’s what most people use, but the packages are handy for large things.

        Gentoo looks interesting too, but the 4 days install time is a bit frightening. I look forward to your review of your experiences – drop me a link.

        As for Arch, they don’t need much security themselves – its up to users to secure their systems and choose software. So if they address package signing then I think they are in pretty good shape. paccheck does a decent job as well – gives me some peace of mind.

        Not sure which I’ll end up with.

        Comment by igurublog | February 24, 2011 | Reply

        • You’ve got an interesting question about pkg_add. ))

          As I understand, all core FreeBSD packages are updated via freebsd-update, which checks trusted keyprint. Other binary packages of your current OS version would not be updated, so after you get them from your install DVD, default way to upgrade them would be using ports system(and compiling the new versions).

          Gentoo installation took me about half of a day, I did it in a VM during work, and wasn’t actually concentrated on this process all the time. So, I suppose that few hours on a modern machine would be enough to get a base system(solid state drive usage might’ve also decreased compile time). Note, I didn’t bother to extensively configure kernel options and usage flags.

          I’ve got Xfce, base system apps and firefox(which I actually compiled instead of getting binaries.) Haven’t yet installed mplayer, Sun Java and my other work tools.

          I hate retyping commands, so before I’ve got X up and running I’ve did all installation over ssh to be able to cut&paste. After I’ve installed zsh-completion, portage became really intuitive to use, it is a great package manager. So, as of today, I didn’t have time to extensively play around, but system seems cute! When I’ve done configuring it, I will move it from VM to my laptop – it really needs some speed boost. I won’t install it as my base desktop system untill I’ll have plenty of free time on my hands and will be 100% sure that everything will go well! ))))

          After having some issues with drivers I buy only HP office-grade printers, they flawlessly work under Linux. If your printer works under Arch, it theoretically should work under FreeBSD, since they both use CUPS.

          Comment by pipy | February 24, 2011 | Reply

          • > Other binary packages of your current OS version would not be updated, so after you get them from your install DVD, default way to upgrade them would be using ports system(and compiling the new versions).

            Yet if you add a package later, you can use ports or you can use “pkg_add -r”, which downloads an unsigned package from a mirror.

            Brother’s linux printer drivers contain binary blobs which don’t necessarily work on FreeBSD, so I’m not sure if its possible, but there are reports of it partially working.

            Thanks for the notes – Gentoo sounds reasonable.

            Comment by igurublog | February 24, 2011 | Reply

        • I found an interesting site, that probably contains everything you might want to know about package management security.

          You were right about pkg_add. Here is an interesting and well-written paper about package management security practices, and it explicitly states that
          pkg add
          relies on users checking that detached signatures match the provided tarball. freebsd-update has improved security and signs root metadata but does not sign packages or use HTTPS making it similar to APT from a security standpoint.

          Comment by pipy | February 25, 2011 | Reply

          • In that case, at least they provide the detached signatures which allows the users to check them. That’s good to know. Interesting article too.

            Comment by igurublog | February 25, 2011 | Reply

        • So if they address package signing then I think they are in pretty good shape. paccheck does a decent job as well – gives me some peace of mind.

          It might be fun to pull and compare data from all of the official Arch mirrors. )))
          It may be done in client-server way: clients run comparison software and sync results with centralized database. Who knows, we might discover some interesting stuff.

          Comment by pipy | February 25, 2011 | Reply

          • As I mentioned in this reply, users examining paccheck’s results provide a kind of monitoring system of the mirrors. Detection of attack is also important, in addition to prevention. There’s no saying how many incidents have already occurred.

            And there are unofficial mirrors floating around as well.

            Comment by igurublog | February 25, 2011 | Reply

          • This time I’ll try to be more specific:

            You are talking about users individually examining their package update results.

            I’m talking about a centralized examination of all currently available Arch mirrors and making a report about their status.

            There is a server-side tool called mirmon, which compares mirrors by their timestamps. I am talking about something like this, but with client-server architecture and ability to perform additional checks. Client will check downloaded packages with paccheck and it will automatically upload information about them to a server.

            Why put anything on a server? Paccheck is a bandwidth hog and I am curious about running a full-blown check for as many mirrors as possible. There is a chance to get some interesting statistics.. or waste some time.. 50/50 ))).

            Your paccheck tool is client-based. If it had the ability to upload retrieved data to packcheck-server, we might be able to compose real-time reports about mirror tampering. If you are interested, I can write server-side software for this feature.

            Comment by pipy | February 25, 2011 | Reply

          • >You are talking about users individually examining their package update results.

            >I’m talking about a centralized examination of all currently available Arch mirrors and making a report about their status.

            I understand, and it’s a good idea, but I doubt I will invest time in that. Any work on paccheck will be rendered obsolete once package signatures are available, which hopefully won’t be too long.

            Comment by igurublog | February 25, 2011 | Reply

  7. As said by karol, if you don’t like it, no one is forcing you to use it.

    Here is where open source really shines, you have the choice to move on and use something else. Alternatively, complaints are best submitted in the form of patches.

    Ranting and expecting others to do the heavy lifting will achieve little. If it is something that is that important to you, roll up your sleeves and get dirty in the code.

    Comment by fukawi2 | February 20, 2011 | Reply

  8. Allan said it best: ‘I think I know every distribution using pacman as a package manager and (unless there is an enterprise level distro I am missing) if peoples lives depend on one of these distros, then I am sorry to say it but in my opinion they are stupid and deserve to die.’


    I got a good laugh from that. More serious, there is no way that I’ll use Arch for production work. But, even though I am mindful of its shortcomings, particularly this one, I still use it to tinker and see what is the latest and greatest in OSS.

    Comment by yfph | February 20, 2011 | Reply

  9. Don’t the checksums come from the same sever you download the packages from? So if that server is compromised they can just change the checksums, so this whole argument is moot?

    Comment by hellop2 | February 26, 2011 | Reply

    • Don’t the checksums come from the same sever you download the packages from? So if that server is compromised they can just change the checksums, so this whole argument is moot?

      You are confusing checksum with digital signature.

      Comment by pipy | February 26, 2011 | Reply

    • That’s why paccheck verifies the checksums against the checksums from other servers: they all should be the same.

      Comment by karol | February 26, 2011 | Reply

    • Yes, the md5 checksums (which can be altered easily) come from the same server, which is why signatures are valuable. If a signature or the signed file is changed, the signature will no longer be valid. Checksums (like md5) offer no such protection.

      You can think of a signature as a tamper-proof photograph of the package. If the package changes, it won’t match the photograph, and if the photograph is altered, the tamper-proof mechanism built into it will reveal that it was altered. Thus with signed packages you can distribute and store the packages and signatures however you like – no server security required. There are a few complexities, but they can be dealt with by a reasonable package management system.

      In the meantime, as karol said, you can use paccheck to download the checksums from multiple mirrors and compare them, or even have it download the packages themselves from multiple mirrors if you don’t trust the md5 algorithm (which is somewhat warranted as it is old and partially compromised).

      Comment by igurublog | February 26, 2011 | Reply

  10. Users on the Arch Linux forum were contentiously divided on this issue, to the point where it became impossible to discuss it there. No one likes to have their vulnerabilities pointed out, which leads to denial and anger. While some saw the importance of it, and have been discussing it on and off for months, others reacted with bizarre fury that it was being discussed. Finally the moderators closed the thread and tucked it away in a section of the forum that only subscribers can access. I have seen this behavior before on the Arch forums – it is not a very open forum for discussing anything even slightly contentious. But hiding and censoring important security discussion was a new low even for them. As a result, many Arch users will simply not be exposed to the fact that their distribution has a major security problem.

    This is wrong.

    There have been plenty of discussions on the topic both on the forum and the mailing list. The problem is related to the package architecture and the “proper” way to do the signing in a KISS model, coupled with the sheer manpower necessary to make it happen once it’s decided on. Threads on the forum get closed for bringing up the a topic that’s already been discussed a hundred times.

    Comment by B-Con | February 28, 2011 | Reply

    • KISS model, coupled with the sheer manpower

      Come on, if everyone interpreted KISS principle like this we would still be driving cars without seat belts. Arch devs have recently created a Hurd distribution. How does it relate to a lack of manpower?

      Comment by pipy | February 28, 2011 | Reply

      • I think he meant that nobody _wants to_ work on this, the devs have other priorities.

        It’s been discussed already, that’s why such threads are closed on sight on Arch forums.

        Comment by karol | February 28, 2011 | Reply

        • Arch is probably a very decent distribution, I’ve read lots of good reviews. In spite of that I won’t try it or recommend it to anyone in the foreseeable future.
          Not only because it didn’t implement package signing. In my opinion, core developers approach to their work is just unacceptable.

          Arch is for tinkerers and isn’t ment to be used in production; devs don’t care about security? — There is nothing wrong with that. Just be honest, write it on your front page. You are being deceitful in supposing that everyone should know it before even installing Arch.

          Even with tinkerers attitude, whats wrong with having some responsibility? If respected A. McRae is holding a post of a core developer, he definitely has a responsibility to Arch users; mere fact that core Arch developers find this optional is a deeply troubling sign. This is a childish if not sociopathic attitude. If I am right, there are only two possibilities: either Arch core team will change or it’s community will degrade.

          Comment by pipy | February 28, 2011 | Reply

          • > either Arch core team will change
            > or it’s community will degrade

            You may be right, time will tell. Maybe Allan will fix it for Christmas :-) You know, Duke Nukem Forever game is finally (almost) here, so anything is possible!

            Comment by karol | February 28, 2011 | Reply

      • Last I heard, no one had agreed on how to make it work properly with KISS.

        Also, this would primarily be development of pacman. The pacman team only has a couple strongly dedicated devs, so an architecture change is more of a man power issue than you might think. Arch development != pacman development.

        Comment by B-Con | February 28, 2011 | Reply

  11. I’v e been using Arch for for 9 months now and I’m very satisfied. I became aware of the package signing issue shortly after I installed arch. Would I have switched to Arch had I know about it before? Maybe not.

    The attitude of the quoted devs is shocking and so is the attitude of many commenters here. Pointing out a problem should be considered a great service not an excuse to heap abuse on the messenger and ask them to fix it themselves.

    Some people can code others can’t.
    Some who can code, don’t have the time right now. So they point to the problem.
    It takes all kinds to make a succesfull community.

    There is no excuse for Arch not to do this. Other distros are able to. Why not Arch?

    For those who are saying that the people pointing out the problem should fix it, how long would you use Arch if there were a major hole in it and the responsible devs just said the users can fix it or shut up?

    Perhaps this will take somebody cracking some mirrors and spreading around some tainted cheese before the community takes it seriously.

    just my 2 cents


    Comment by emk | February 28, 2011 | Reply

  12. This is maybe the hidden part of the reason for so much hate and hunger against Ubuntu. Linux users aren’t anymore all of them developers. It has become very easy to create and distribute remixed Distros. Many authors of these remixes as well as tens of thousands of the new Linux users, attracted by the Ubuntu ease of use, are not security conscientious people and early or later exploits may start spreading in a supposedly virus free environment. Everybody will then start looking around for explanations and the conclusion will be that new Linux users won’t be the only ones to be blamed.

    Comment by gielgy | February 28, 2011 | Reply

  13. I wanted to say that I appreciated this article. I for one did not know this but have not looked into the distribution much. I do use it and use others and then come back to it. I do like the fact that when updates come out they are posted quickly for the most part. Some take some time to show up but those are mainly programs and those I can wait for. I think that I will look else where as I have seen comments on the forums that basically tell users that Arch Linux is not for every one. I think most people realize this already but they can be nicer how they approach users.

    One thing that lead me to them was the rolling release and the programs that I wanted being released soon after they were released. I did not have to wait for these like with Ubuntu or Debian. I had tried Gentoo but found some steps missing at the time. I got to a certain point and either it was broken or did not know the next step as it has been a while as I forget what happened. One thing that I liked about Arch Linux is that some one wrote an easy install guide which made it pretty easy to install. I for one am not an advanced Linux user and thus find some things hard to do in Arch Linux such as setting up desktops but know that many guides exist for this.

    One thing I have tried is Sabayon which was a fine way to go for me since I did not have much luck installing Gentoo from scratch. This was an OK release but I think that I broke it and went back to Arch Linux. I think that I was doing a clean up and some how deleted more then I was supposed to.

    One thing you might want to try out is PC-BSD as it is an easy install. They have different one’s based on different versions of FreeBSD. I have never installed FreeBSD from scratch. I have seen articles comparing FreeBSD to Linux and Linux usually comes out on top as far as speed goes. I keep trying out the PC-BSD versions but lately I am using Arch Linux as like I said I like more recent updates and FreeBSD does not do them as quickly. One command that I was told about by the PC-BSD people is portmaster -a which will update your system but will give you all the questions at the beginning of the install. What I mean is that certain updates come with some items you can check depending on what you want on your system. Other commands such as portupgrade have these when it gets to that package. I found that updating the system took a very long time to compile each item. I guess another reason why I liked Arch Linux as the updates seem to be faster.

    Comment by John | March 1, 2011 | Reply

  14. I’m using Arch for a year now and only now through this article the importance of this issue is dawning on me.
    I think the attitude of some people here swearing at you is really the worst sort of fanboyism. I really think it is a pity you are leaving the Arch community. But there some strange informal hierarchies and social processes going on there, that I don’t really grasp and wanna grasp.
    I think the matter can be translated to this imaginary scene:
    *The roof is leaking…really serious
    #Yeah, I know, it has been leaking for years.
    *But it is really starting to rot now.
    #We have contacted a contractor and he said time and time again that he would come and fix it.
    *So, he is not fixing it, is he? We should find some one else to do it.
    #Why don’t you go live somewhere else if you don’t like it here.
    *But I care about the house.
    #You’re spoiling my day. Get lost.

    Comment by Paul | March 1, 2011 | Reply

    • Wrong.

      *But it is really starting to rot now.
      # So why don’t you fix it? Would be a great service.

      *No, I only complain. It is your responsibility to fix it.
      # Yeah, my daily consultancy charge is 1000 USD. Let us sign up a contract and start work!

      # You’re spoiling my day. Get lost.

      Comment by Jasim | March 24, 2011 | Reply

  15. Whew! I was thinking to move to arch from Debian…was always tempted by the tempress! (Latest n Greatest)… but this is a deal breaker.. Sorry Archie you are no good for me..

    Comment by tom | March 1, 2011 | Reply

  16. I want to solely comment on this sentence: “This mechanism has been around for many years and works well – the tools to implement it are available and simple to use.”

    The tools to sign a file with a key are there. But what else is needed? Key management (which keys should a user/admin trust), integration into the package manager, integration into the management tools on the servers, integration into the developer workflow.

    If any of these is done improperly and without thought, then all you get out of package signing is a false sense of security. After all, security is a concept, not a bunch of tools.

    It is easy to write a witchhunt rant like your above article, it is difficult to create a security concept and implement it right.

    None of that is an excuse for not having signed packages, but it is the reason. One thing is for sure: There will be signed packages, but it is currently unknown when that will be.

    Comment by Thomas Bächler | March 1, 2011 | Reply

    • > But what else is needed? Key management (which keys should a user/admin trust), integration into the package manager, integration into the management tools on the servers, integration into the developer workflow.

      I would say you are exactly wrong. Integration into the management tools is NOT needed for signatures to be provided. Users can check signatures on their own or using scripts, at least until your package management is complete. All that is required is that the packages include a signature somewhere. That can be as simple as adding a single line to the db creation script, and a single key for now. Arch users are very self-sufficient for the purposes of checking those signatures. The number of people using my paccheck script to attempt to verify the mirrors shows this – and that script is a horrible brute compared to what a single signature could provide.

      Your assessment that leaving the software on mirrors completely unprotected is better than providing a signature now is utterly without substance. Even a poorly implemented key management scheme would VASTLY improve the security situation. Instead, the Arch devs would rather sit on their hands and talk about package signing schemes for years while doing nothing, and while preventing anyone from doing anything.

      You are standing in the way of improving security on Arch – you and others in your ‘club’ won’t let anyone add that simple signing line to the db creation script. Not only would this improve security, but it would get the ball rolling in a positive way. It is a valuable beginning.

      And don’t talk about how easy it is to write an article such as this – try it some time, and listen to people like yourself attacking the messenger. Writing a few lines of code is far easier than dealing with your (collective) emotional and ego-based resistance to change. What’s easy is adding a signature to a package. Why you can’t see that or how it would vastly improve the current security situation, I can’t begin to imagine.


      Comment by igurublog | March 2, 2011 | Reply

      • After reading your post yesterday, I couldn’t help but keep my head busy with this topic again (I am after all a paranoid person).

        Let me add what came out of it:

        1) A quick’n’dirty “better than nothing” solution that could quickly be implemented is this:

        Sign the sync database on each update with a key that is stored on the server. This has many flaws, but it would guarantee that the sync db file originates on our server.

        The sync db contains md5sums of all package files – this is not optimal, but still better than nothing to verify authenticity. Generating another package that appears genuine, has been compromised and matches the md5sum is not impossible, but not easy either.

        2) Package signing is on the roadmap to pacman 3.6, and I think it will stay there. Pacman 3.5 is just about to come out, so the 3.6 development cycle is soon to start.

        Two more points:

        First, I don’t feel I attacked you. I don’t disagree with most of what you wrote, I just tried to put some perspective onto your very one-sided summary of the topic.

        Second, I will repeat the standard response – I also hate it, but it is the truth: If signed packages are so important to you, why don’t you take Allan’s package signing patches and continue the work, so it will be completed sooner?

        Comment by Thomas Bächler | March 2, 2011 | Reply

        • > Sign the sync database on each update with a key that is stored on the server. This has many flaws, but it would guarantee that the sync db file originates on our server.

          Yes, this is exactly what I have been asking them, and I also asked them to include a SHA256 sum in addition to MD5. I tried submitting feature requests on this very thing this morning, WITH A PATCH. As usual, Mr. McRae is doing everything in his power to subvert this improvement, while at the same time he claims signatures are of no interest to him. It makes me wonder who he may be really working for (you’re not the only paranoid one). Please vote…

          Their roadmap does not particularly encourage me, but it is better than nothing. They have been talking about this and doing nothing for YEARS.

          > If signed packages are so important to you, why don’t you take Allan’s package signing patches and continue the work, so it will be completed sooner?

          First, I have already given them a patch for repo-add – look how simple it is to do, and watch how they refuse it.

          I am not familiar with pacman’s code, or the signing system they have devised, nor am I a particularly experienced C developer. Before I invest the time in bringing myself up-to-date on their code and adding to it, I would want some reasonable assurance that my work would not be ignored. They seem to refuse contributions in this area – their own devs are complaining how Mr. McRae has blocked their code. I’m not wasting hours and hours of time only to have them say, “no, we don’t accept this contribution”, which will be the case based on their responses so far. I have written numerous useful scripts as well as pcmanfm-mod – none of them have ever been included in the Arch repos, even though similar tools with fewer votes have. It seems Arch is very much run as an insider’s club, and I can’t stand politics.

          So the answer to your question is, I AM contributing (look at paccheck), to the extent they seem to accept contributions, which is not at all thus far. Believe me, if I were permitted to code it, it would have been done in less time than these conversations have taken. Yet I’m not the best man for that job – the pacman devs are (they know their code, which is a big bonus).

          Yet may I say again – I AM NOT ASKING FOR PACMAN CHANGES. I am asking for signatures, and will gladly submit a patch for them if desired (which will consist of one line). I don’t care if pacman checks signatures or not – I can do that with gpg.

          Comment by igurublog | March 2, 2011 | Reply

          • It makes me wonder who he may be really working for (you’re not the only paranoid one).

            *PARANOIA PARSING ERROR*. Security agencies are probably bugging Linux kernel itself and its core security modules. Why would they want to bug some non-production distro?

            Imagine for a second that Alan McRae is a bad guy: he probably already has more than enough means to introduce vulnerabilities, with or without package signing.

            The famous Reflections on Trusting Trust (1984) paper says it all:

            The moral is obvious. You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code.

            You have to trust someone! Or just consider your systems compromised from the start, it’s not a bad approach.

            Comment by pipy | March 2, 2011 | Reply

          • > Security agencies are probably bugging Linux kernel itself and its core security modules. Why would they want to bug some non-production distro?

            Indeed. I was mostly joking, although his behavior is cryptic (doesn’t care about package signing but REALLY cares that it doesn’t get done). Mostly I think he’s just too ego-invested to allow any signing approach to proceed which beats pacman’s signing to the punch. That’s how ridiculous this is. Allan McRae has come to believe he IS Arch Linux – and most of the devs seem to allow that. Who knows? All I know, getting anything done in this area, with or without patches, is impossible.

            It’s one thing to reasonably trust the Arch developers and use their software. It’s another to trust an anonymous mirror operator, and all the people who have legitimate and illegitimate access to that mirror.

            Comment by igurublog | March 2, 2011 | Reply

    • This is not intended to be inflammatory, but just a suggestion.

      None of the above needs /designing/ from scratch. The problems and solutions are already found and solved by the distributions that are already using signing. You can just pick up the existing methodology (and tools, should you wish) used by others.

      I’m not suggesting that /implementation/ of all the functionality won’t require effort, but the hard design problems have already been solved.

      But, you can implement package signing well before you have tool support for signature verification. Generating and using a single archive signing key does not require the use of complex key management solutions, at least for an initial implementation.


      Comment by Roger Leigh | March 2, 2011 | Reply

  17. The Arch distro comes with two major unsympathetic traits: the tone in its fora and the aloof, cavalier disdain by the devs toward the distro users, as perfectly exemplified by the behaviour of this Mr. McRae.

    Responsibility is a rarely used word in the Arch ‘community’. As is security – try finding it on their front page. They prefer (talking about) eating tacos.

    Other than that, i.e. as long as you are lucky and don’t have to visit the fora, Arch Linux itself is a pleasure to work with. I have used it now for three years with only minor or short-lived issues.

    However, the Arch devs’ irresponsible handling of security concerns has serious potential implications for Linux, open source in general, not to mention end users.

    A big thank you to IgnorantGuru for literally turning a spotlight on the situation!

    Comment by SES | March 2, 2011 | Reply

    • @SES: Your comments exactly reflect my thoughts on Arch at this point. It’s a handy system to use, but every time I interact with the devs (or the forum moderators – in some cases the same people) I am left feeling like something is very wrong. They have been informed of the seriousness of this situation since prior to 2006 – that is a very long time in computer years. The good news is they now have package signing on the roadmap for pacman 3.7, so maybe it will finally happen (but I’m not betting money). And there are some devs there that have been asking for this for years, so some of them ‘get it’. Yet there are those in that circle that seem to work actively against users’ security.

      And as you said, it reflects poorly on OSS in general. I can just see thousands of Arch users’ systems getting rooted, and the headlines “Linux security compromised”, with Microsoft smiling with glee. For all we know, mirrors have already been silently compromised many times over the years – there isn’t a way to tell.

      Arch doesn’t need to do much for reasonable security. Mostly they just distribute the software as-is, handling the compiling and basic config (if that), so much of the security responsibility is the users’. Yet they certainly need to secure the basic distribution system or chaos is inevitable.

      For a distribution that is prideful and even arrogant, the emperor is stark naked.


      Comment by igurublog | March 3, 2011 | Reply

  18. Well, it seems that the Chakra devs (the ones who built KDEMod for Arch) think package signing warrants attention for their Arch-derived distro. They even made the thread a sticky.


    Comment by yfph | March 6, 2011 | Reply

  19. UPDATE

    Having now spoken with Aaron Griffin, Arch’s ‘owner’, I thought I would add a few comments.

    I contacted Aaron because after various attempts to get something done with Arch’s lack of mirror security, I felt that the problem was a lack of leadership. I was hoping he was merely unaware, but I also acknowledged to myself that the fish probably stunk from the head down. Yet I wanted to hear what he had to say on the subject. I already knew that the reason NO progress has been made on this issue for years is not due to lack of contributors. In fact, devs and others have brought this issue up continuously over the years, asking for and offering solutions, myself included. Contributions are BLOCKED. And this makes sense. Secure package management just isn’t THAT difficult.

    Speaking to Aaron, I came to see where the attitude of the devs comes from, and even where the forum rigidity comes from. I was quite shocked by Aaron’s utter disdain for Arch users, describing them as “thankless”. I very rarely see any Arch user who is thankless. On the contrary, they often profess their thanks to the devs of Arch, and if anything are too timid (in my opinion) in giving them constructive criticism. Beyond this, Aaron addressed none of the issues I raised with anything of substance, and expressly did not welcome contributions. I was left with the sense that Aaron resents anyone even using Arch, and thus seems to welcome the security problems being unaddressed for so long. In his mind, I think the “thankless” users don’t deserve security. He couldn’t care less.

    Now, for me at least, the puzzle has begun to fit together and the picture is emerging. The reason Arch’s security problems remain after years of occassional discussion comes from the top – the leadership of the Arch project. And the moderator hostility on the forums stems directly from the disdain the leadership holds for the users – that is why the moderators are not held to a higher standard, or any standard. There is just not a quality relationship in Arch between those who control its development (and lack thereof) and those who use it, and no technical solution is going to improve that. Now, not only am I convinced that the Arch devs have a poor attitude about their users’ security, but they have a poor attitude in general. This is not an attitude that I wish to be associated with in any way.

    On the subject of security, the resistance to improvement in this area is stunning, and I do not rule out it being deliberate. As pipy said, those who wish to compromise Linux can bug the kernel, yet there are many factions involved. For example, to me the entire Adobe Flash project is nothing but the world’s biggest trojan (except for Windows itself). Those who wish to weaken computer security do so from many angles. Yet it could also simply be carelessness and disdain for Arch users. I just don’t know the truth of it. Yet I do know that Arch has dropped the ball in a serious way. With the current security model in Arch, it doesn’t much matter if root exploits are addressed, or even if the root password file is published. Compromising large numbers of Arch systems is trivial, and may have already occurred countless times. Even those who don’t take security particularly seriously don’t like to have their systems compromised – it just isn’t any fun. A little basic security goes a long way, and the tools for it are well-developed. Arch just isn’t using them and doesn’t want to use them. Put simply, they don’t care.

    Open source/freeware developers deserve a lot of latitude, as most of what we do is volunteer. Yet if Linux is going to have any degree of quality to it, a certain level of responsibility and sincerity is required, even from those volunteering. The Arch developers don’t make the cut, at least for me. I know there are developers in the Linux community that do excellent work. I’ll seek their company.

    As others have said, Arch’s design has a lot of value. Yet with the larger picture I have now, it’s only skin deep. There are serious problems with Arch Linux under the hood – buyer beware. I did my best to change that, without success, and to inform others, with some success. Now the only responsible alternative is to leave it behind. While it may take me some time to find a good alternative and to migrate my systems, I’m on my way.

    Comment by igurublog | March 8, 2011 | Reply

  20. The fish IS rotting from the head.

    Well, well, well… I sympathize with your decision to abandon ship, but personally I will stay with Arch for now, mostly out of sheer laziness :(

    Good luck in finding a replacement, and thanks for this useful update. Things have become a lot clearer!

    Comment by SES | March 8, 2011 | Reply

    • It is a bit of work to change, but Arch wasn’t very challenging to me anymore, so I’m looking forward to learning something new. Arch was comfortable though – fit well in some ways. It’s hard when a distro has a lot going for it but misses the boat on a major issue – makes it hard to choose, but I have to set some standards. And frankly the devs have creeped me out. If you stick with Arch for now, just make good use of paccheck (it may be only a bash script, but it represents the biggest security improvement to Arch in at least 5 years).

      Someone mentioned that I might like aptosid, so going to look at that too – debian on wheels?

      Also, keep in mind that most tools I develop on this site are distro-non-specific, so don’t be a stranger.

      Comment by igurublog | March 9, 2011 | Reply

      • Please keep us posted on your quest for a new distro. There doesn’t seem many out there with Arch’s current features and up-to-date packages.

        Comment by LoyalArchUser | March 16, 2011 | Reply

        • Yeah, I’d like to switch too. What about Mint Debian?

          Comment by paulo | March 18, 2011 | Reply

        • Sure – I posted an update on my latest adventures here.

          Comment by igurublog | March 18, 2011 | Reply

  21. Thanks for a *very* timely warning.

    I *had* planned on using Arch on my production servers, but upon reading your article, I guess I’ll stay put with Gentoo.

    That said, trust me if I say that — with today’s computers — installing Gentoo from scratch is no longer a multi-day event. Unless you *really* like to tinker with kernel .config.

    Again, a heartfelt thanks from me.

    Comment by pepoluan | April 9, 2011 | Reply

    • Glad it served you well. The Arch devs basically said that Arch should not be used for anything requiring decent security.

      I agree Gentoo is pretty reasonable to install. I had an almost full install setup, and came close to using Gentoo, but ended up going with aptosid for now.

      One note on Gentoo – although not as careless as Arch, it does have some related issues. This came up in the comments to the LWN article:

      “What is true is that there seems to be no policy requiring Gentoo developers to sign manifests, and as a result, many developers never bother to do so and thousands of packages remain unsigned.” link

      Comment by IgnorantGuru | April 9, 2011 | Reply

  22. I’d also like to express my thanks for making this information public. I read a little of the methods of how this could be exploited and also the arguments from the arch devs.
    IMHO I thought the treatment you received was a little harsh. If someone points out your car window is open you don’t give them the key and tell them to go and close it.
    Similarly the downplay of the issue is pretty bad management. Arch is a great distro but for another analogy it doesn’t matter how good your “house” may look or feel if you knew people could potentially walk in without much trouble and take your hard worked code and family stuff pictures!
    Thanks again and good work IgnorantGuru.

    Comment by Todd | May 10, 2011 | Reply

  23. Thank you, IgnorantGuru.
    Your quality journalism brings a lot of value to the Free Software community, and in this case, personally to me.
    I have an advice for Arch Linux developers, their fanboys and Aaron Griffin:

    Linux has certain standards pertaining to openness, security, stability and freedom.
    If you can’t run a Linux distribution in accordance with them — don’t fucking do it.
    If you are a Free Software developer, prepare to be altruistic. You can’t be a good voluntary developer/maintainer/contributor and an asshole at the same time. These things just don’t go together, because of the way gift communities fundamentally work.

    In a gift community, you don’t share shit for free, then expect something from its users.
    You share shit for free and just feel glad when someone uses it.

    Comment by Anonymous | May 12, 2011 | Reply

  24. . .
    - , - ,
    , :
    . , .
    XXI .
    , , ,
    . , ,
    - , ... , , . , , - . , : , , , , . , . .
    , .
    , ,
    . ,

    Comment by vitalik06047913 | January 15, 2012 | Reply

  25. “News: pacman 4 moves to core
    2012-01-16 – Dave Reisner
    Pacman 4 has landed in core! Thanks to 24 contributors producing 893 commits, you’ll find many new features. The one explicitly worth calling out is gpg signing. However, until the last few details regarding database signing and keyring distribution are ironed out, this is disabled in pacman’s default config. If you’re interested trying out package verification, please refer to the documentation on the wiki about pacman-key or Allan’s blog post.

    As always, please make sure to merge your pacnew files!”

    Nuff Said.

    Comment by Bergers | January 17, 2012 | Reply

  26. Its nice to read from me that i’m a arch developer. Funny too. Only because i wrote my opinion in the dev-List, i’m be responsible. Fuck it. And delete the guru from your name. Ignorant as a name and program make more sense.

    For the records. I never ever wrote a line for any official arch package. Esp. not for pacman. For security reasons these are good news for arch.

    Comment by alfgaida | March 27, 2012 | Reply

  27. The content of this post is still valid?
    This ArchWiki page let me think the problem was finally addressed:

    [sorry for the (maybe) stupid question, but I really care of security while not being so expert in such field]

    Comment by easley | January 28, 2013 | Reply

    • Since this article was written, Arch Linux has implemented package signing. I’m no longer using Arch, but the few times I tested it the package signing had some problems, and many people were disabling it just to get pacman to work. Hopefully they have improved it since.

      Overall, I don’t trust Arch Linux, and I don’t recommend it for those wanting a fairly secure system. While they have implemented package signing to some degree of effectiveness, the attitudes of many of the Arch devs was and presumably is highly questionable in the area of security, and they lack an effective security team to review and announce potential problems (last I knew). When I dealt with them, they actively HID security problems from their users. I simply wouldn’t count on them to consider your system security as important, and that’s the first ingredient for good security. Arch is also becoming ever more user-hostile (see the recent forced migration to systemd for reference).

      So I would suggest debian sid, gentoo, etc if you’re looking for a distro that takes the security of their users seriously. But I also am not actively involved in the Arch community these days so you may want to seek more informed opinions on what has been happening there.

      Comment by IgnorantGuru | January 28, 2013 | Reply

      • Ok, clear, I understand.

        What a pity, it’ s a so sweet distro.
        Gentoo is too for me, I don’t want to spend my life compiling program.
        Maybe I’ll try Debian Sid…

        Just two questions about Debian Sid:
        1) can I customize my system like in Arch? I mean, I do not use a Desktop Environment, simply OpenBox, and the applications I use are fairly custom and come from different environment (from Gnome: gnome-system-monitor and file-roller, from LXDE: LXPanel, LXAppereance, from ROX: ROXTerm, from nothing: ARandR, pavucontrol, etc…)?
        2) are Debian Sid package stable? (I mean, are Debian Sid package at least as stable as Arch one?)

        Comment by easley | January 29, 2013 | Reply

        • I agree, Arch is a nice distro in some ways, but unfortunately the devs consider it a toy.

          You can read my somewhat dated review of moving from Arch to Aptosid, which is debian sid with a few refinements. There is also now Siduction as well as VSIDO (debian sid with spacefm as the default fm), but I haven’t personally tried them yet. I am still using aptosid for almost two years now and it works well for me with just Openbox (I start with the xfce or lxde installation and then simplify it down to just openbox), and custom components for most things. I have found it a decent replacement for Arch.

          I encounter less breakage in debian sid than I did in Arch, so I would describe it as more stable, and more thought goes into how packages are integrated, so more secure overall and less hassles. You can also try debian testing, which should be more stable than sid.

          Comment by IgnorantGuru | January 29, 2013 | Reply

  28. This article ist from February 19, 2011. Its completly outdated cause all Arch packages are signed these days, there are keys on the initial installation disk and signature checking is turned on by default. I would suggest that you put an itroduction chapter about this topic on top of this article cause otherwise people are getting confused.

    Comment by Philip May | June 23, 2013 | Reply

  29. Packages are now signed, FYI.

    Comment by Glados | September 29, 2013 | Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 94 other followers

%d bloggers like this: