IgnorantGuru's Blog

Linux software, news, and tips

LWN Picks Up On Package Signing

In their latest issue, LWN.net, one of the definitive news sites providing comprehensive coverage of development, legal, commercial, and security issues related to Linux, published their article Arch Linux and (the lack of) Package Signing:

The Arch Linux user and developer community has been engulfed in sharply divisive debate recently over the issue of package signing. It started when an Arch user blogged about the distribution’s lack of package signatures, the security risk it created, and his own frustrations with the core developers’ response to the issue. The ensuing argument has since spread to include Arch’s development model and a variety of leadership questions, but the root problem remains unsolved: there is still no mechanism to verify the authenticity of “official” Arch Linux packages.

To blogger IgnorantGuru, this constitutes an unacceptable security risk. In February, he blogged about his concerns, noting that without a method for Arch users to verify that a package is unaltered, packages can be replaced with Trojan-horse-laden code.

The author, Nathan Willis, contacted me earlier this week to ask some questions, and I feel his article provides a very comprehensive review of the core issues, including the problems with Arch’s devs refusing contributions in this area and stalemating Arch’s security improvements for years. I think it’s great that LWN is reporting to their subscribers so candidly and giving this issue much needed visibility. The article concludes in part:

In the final analysis, Arch users are exposed to a security threat both by the distribution’s lack of package signing and by the core developer’s resistance to adopting it. However much the Arch “philosophy” says each user is responsible for his or her own system, IgnorantGuru is correct in his first blog post when he observes that without signatures, the distribution’s infrastructure is vulnerable to every exploit found in every other system on the path between the main project server and the user’s PC…

Read Full Article Here

The ongoing discussion in the comments there is robust and also highlights some ways that Gentoo may still have vulnerabilities related to this (at least according to some), so I believe discussing these issues openly and without censorship is valuable.

March 24, 2011 - Posted by | Tips


  1. They’ve finally got the message and are merging signing support into pacman.



    Comment by psychedelicious | March 24, 2011

    • The prospects of getting signing in pacman eventually do look better, but I would be cautious – they’ve been semi-working on this for years and their pacman ‘roadmap’ is always subject to change.

      I also noted that Dan McGee, who originally mocked my SHA256 sums idea as ‘hilarious’ (great lead dev attitude toward contributors), has now allegedly implemented it for pacman 3.6 (two versions away). While this may seem like him being helpful, 3.6 is also the version they plan to have package signing in, so I don’t see the use of SHA256 sums then. IOW he is creating unnecessary delays in improving Arch’s security until it’s in pacman – just as I discussed earlier. Unbelievable resistance to get anything done. (The SHA256 sums could be put in pacman now with no hassle and would make paccheck’s testing more complete.) I guess he too views me as in competition with pacman. As for users security, that doesn’t seem to be on his radar at all.

      At any rate, this LWN article definitely did some good – the Arch devs are now attempting to explain themselves (apparently it’s all MY fault now – thanks for the promotion but I’m just a concerned user).

      Comment by igurublog | March 24, 2011

      • Pacman 3.5 is already out so I expect signing will be implemented in 3.6 as this is the next major version and signing is in the roadmap for that.

        I think possibly both parties here, the Arch devs and yourself have said and done good and bad things. The end result is though, that the problem looks like it will be fixed and your rather overt publicising of it has probably moved things forward substantially.

        Comment by psychedelicious | March 24, 2011

  2. for those who care about package signing, the real story is on Dan’s blog[1].

    you will all see that ignorantguru produced stories that in fact doesn’t exist.

    [1] http://www.toofishes.net/blog/real-story-behind-arch-linux-package-signing/

    Comment by Ionut | March 24, 2011

    • Here is the reply I posted to Dan on his blog. Given the Arch way of doing things it will probably be deleted, so here it is for your viewing pleasure. At this point I think this discussion has degraded into a blame game, but at least some response is now being made.

      Hash: SHA256

      Mr. McGee,

      I see I am now solely responsible for your failed leadership. Thanks for the promotion, but perhaps you should take responsibility.

      I see you’re once again pushing the ‘no contributors’ explanation for years of security neglect. As your own developer Pierre wrote in 2008, users security could be improved hugely by simply having the server sign the database. That was also what I suggested, and it is trivial to implement (and he was willing to do so). The only reason it wasn’t was because of lack of leadership and potential loss of face for pacman’s team. Users security doesn’t even appear to be on your radar.

      > Mr. IgnorantGuru filed this “flyspray”, FS#23103, asking to add sha256sums to our package databases… Surprise- no patch showed up in my inbox or on the bug report.

      No patch was provided because you never responded. I note that you didn’t include in your quote your calling my idea “hilarious” (great lead dev attitude toward contributors). I would be thankful that a couple of days ago you did implement my “hilarious” suggestion, except that you have pushed it all the way to pacman 3.6, when package signing is allegedly due. IOW you have rendered it useless. Once again saving face for pacman’s dev team is more important than users security. At any rate, when you inevitably delay package signing a few more years, please leave the SHA256 sums intact for the sake of users security (I know, “hilarious”, but some of us appreciate our systems not being rooted.)

      > For the first bug, FS#23101, the article is correct… However, suggestions don’t produce working software

      Pierre was willing to create the server key and have the server sign the database. It is trivial – once again no patch was required or requested. Apparently it was just politics and lack of leadership – seems to be a theme.

      LWN provided a comprehensive article. Nor are they the only ones – you’re just inattentive. Instead of attacking them, and me, and anyone else you can think of, take a look at your own lack of leadership and resistance to accepting contributions. These may not directly involve pacman, and thus not directly inflate your ego, but they would provide users with the ability to authenticate their downloads. LWN’s priorities are where they belong – they are INFORMING their subscribers of a serious security flaw which as yet your team has been unable to correct for years. Embarrassment is understandable, but that does not justify personal attacks or forum censorship of the issue. This is why I made the issue visible – to improve users security. Ponder that priority.

      Version: GnuPG v1.4.11 (GNU/Linux)


      Comment by igurublog | March 24, 2011

      • “Here is the reply I posted to Dan on his blog. Given the Arch way of doing things it will probably be deleted”

        When engaged in a battle of rhetoric, you should be careful about any assertions you make. In good faith, Dan has left every comment up on his blog, including yours. In my eyes, he already stands above you on the morality scale.

        You take a preemptive pot-shot at him before he’s had a chance to act, and this diminishes you (and your platform) in my eyes.

        If you truly believe in, and want to champion, this subject to the end, then you need to maintain some dignity throughout. No more pot-shots. I know Ghandi quotes gets thrown around like candy, but be the change you want to see. Ad hominem attacks are cancer to an important debate such as this, and from what I see here, you are contributing to that problem, while at the same time professing your desire for change. This comes off as hypocritical, to myself and likely to others as well.

        I support the idea of package signing in Pacman. I do not support your methods.

        Comment by anonybot | March 25, 2011

  3. “The real story” and “for those who really care”, the defamation of IG by Arch fanboys has become so default, that it has become a stale and acid formula. Read the comments of Dan McGee’s supporters in his original post and feel the horror they get of doubting Arch ( if you have criticism, you are not a friend = an enemy).

    What was but a small part of the LWN article and should have had more attention and concern is the smear campaign against IG, the way he is constantly attacked personally. They way IG is portrayed as the sole mad ranter. The censorship and lack of tolerance to dissent on the Arch forums seems so normal and accepted, that even in his defence Dan McGee chooses to ignore it. The discussion wasn’t open and IG was silenced in the Arch community and only because of the LWN article it is now on the http://planet.archlinux.org/

    Now of course LWN is the new enemy. Well the picture in the mirror is harsh and is getting harsher every day.

    Comment by Pablo | March 25, 2011

    • It seems like you harbour some ill will towards the Arch team, Pablo. Pray tell, do you use Arch? If so, why? Is it technically superior? Do you agree with its philosophy? It seems you have very little good to say about it, but you do sound invested in this topic.

      I’ve been an observer of the Arch development community for some time, and I too have an opinion on their actions and reactions. As a FOSS developer, I think our backs tend to go up when people are rude or demanding of us. It stems from the fact that we’re often tired, juggling a salary job and then our FOSS jobs in our so-called leisure time. Yes, this is our choice, and we like it for the most part.

      But the reception of a feature request is really quite dependent on how the request is made. If I walk into a room of volunteers and demand they do something, I will be met with cold stares, or I may just be ignored. The developers are admittedly understaffed, and

      Package signing is important, more to some people than others (as has been witnessed on here, LWN, Reddit, and Dan’s blog). So, in the true spirit of FOSS, does it not make sense that the feature could/should be implemented by those that value it most? This is how things happen in the bazaar model.

      As I understand it, IgnorantGuru has crafted at least a partial solution. So, while keeping emotion out of the equation, can we identify the reasons why it was not accepted or merged? Again, do this without ad hominem attacks. Don’t say things like “Dan eats babies and hates security and is trying to ruin Arch.” Let’s actually study the issue.

      I cannot answer for the devs, and I will admit that I am not fully educated on the timeline of events. But I can speculate.

      (1) Did IG’s implementation have parity with the existing Pacman architecture? Was it written in Bash and C, with minimal dependencies?
      (2) Was it delivered to the Arch devs in their desired format, likely a series of patch files against the Pacman trunk?
      (3) Was it a complete solution? Did it cover all aspects of a PKI? Key generation, verification, distribution, revocation? Package signing? Database signing?
      (4) Did it included any documentation for the developers and for the end user?

      If any of these four items were not completed by the people most invested in this feature, then those items would have to be completed by the (admittedly understaffed) developers, would they not?

      You see, the point of a bazaar model is that everybody who can pitch in, pitches in. If you hand over a half-baked solution, you may in fact be creating *more* work for the developers. In the past, I have received user-contributed changes that are tangled messes of code that take many hours of work to “fix” so they can be merged into the codebase. The users’ hearts are undeniably in the right place, but their execution can ofttimes be so poor that I’m forced to throw it out and re-implement their changes myself. This comes at a personal cost to me, as the developer.

      This comment went a lot longer than I had anticipated. Allow me to close with the crux of my message: Eliminate emotion from the debate, recognize that everybody wants Arch to be better tomorrow than it is today, and be very clear with your requests and abilities. And above all else, show some empathy.

      Comment by anonybot | March 25, 2011

      • I just noticed this in the Reddit comments:


        I think this validates my sneaking suspicion that the Arch devs do want what is best for the project. If they were simply bowing to PR pressure, then the they would have done this long ago. As IG himself admits, this is far from the first article about the lack of package signing in Arch.

        Comment by anonybot | March 25, 2011

      • If you want to understand what I write,
        than it is clearly about the social relations within the Arch community. I do appreciate everybody’s effort a great deal, but that doesn’t absolve devs from treating users or visitors on the forums and in other contacts with some openness and respect. IG put forward , I assume to the best of his abilities a partial solution that was ridiculed as hilarious by Dan. IG stresses the urgency of the matter and is only personally attacked, when he expresses his concerns.
        Therefore I find it unbelievable that you say I play ad hominem, which is merely reversing the complaints I made. I just have to establish the fact that criticism is something the Arch community can’t deal with in a self-conscious way but has to put in a bad light by blaming the person who criticizes. This is what is also central to your post, put into doubt my intentions or prove me ignorant of the bazaar model, that nobody questions or negates as matter of fact.

        Nobody denies this is a difficult problem that has to be tackled on a few levels, but the discussion was to my understanding about providing a temporary solution, that is better than none or/and better than paccheck.

        The inner contradiction in sending the message: ‘do it yourself if you think it is important’ and the known fact the core devs are working on it since 2006 is denied time after time.
        We both agree there should be more empathy;the emotions expressed shouldn’t hurt one another; how do you feel about these comments then on Dans post then: Griffin: LWN article is complete and utter bullshit… There are enough rumor/conspiracy websites out there….A-grade twat..you now have the choice to PUT UP, SHUT UP or SHOVE OFF…I would treat your immature childish temper-tantrums exactly the same way ..someone lets off a fart bomb and calls the whole place rotten…I love the fact that Arch has its very own troll….we’d be better off with one less lying troll…one obnoxious and ignorant amateur ..etcetera

        Comment by Pablo | March 25, 2011

        • Oops. My response appears farther down in the comment region. Mea culpa!

          Comment by anonybot | March 25, 2011

        • Even though I don’t read into IgnorantGuru’s blog any ill intentions, I don’t think you do him any favours by diminishing his less polite responses. Bias in this case would do more even more harm, since it probably only would encourage similar actions somewhere else. I’m quite sure this isn’t an Arch only issue. Sure as humans with flaws, he’s not alone guilty of unnecessary comments, thus both sides are guilty in some degree. Still you need to put it into perspective:

          – was he invited or did he invite others to a community?
          – has he contributed more to the code base of Arch than the ones he’s criticizing?

          Common sense, even though the Internet seems to erase civil etiquettes, would encourage an individual to play the social game differently than IgnorantGuru did. It doesn’t matter how much you know or what skills you have or if your IQ hits the sky, because if you’re to collaborate you need to add some EI to the mix. We can all error, and no real harm is done. Important though is to stop before it becomes an obsession, leaving no doors open for a productive reconciliation, or at least get on with what matters in life.

          Comment by KimTjik | March 25, 2011

    • It’s Friday. Relax and enjoy the company of friends and family. IgnorantGuru isn’t my enemy, neither is LWN, or you. Everyone is allowed to have an opinion, even how annoying it at times might be. It’s not that dramatic as it’s made to look like. Still it would be good if people more often remembered to show respect for traditions or etiquettes within communities. This attitude of demanding others to accept one’s ideas makes me feel old. It’s not the way I was raised.

      Signing will eventually be a part of pacman, as this has been a known missing feature for years with or without criticism by IgnorantGuru. Personally I’m not interested in a hacked solution as the one suggested by IgnorantGuru and rather wait for a final solution. How long have you been around the Arch forum, Pablo? Enough to know about previous discussions, years back? What seems to be the case in your eyes today, might not reflect the true state of affairs.

      In this context respect has to be earned, something that requires patience. In my opinion that’s where the real lesson is to be found.

      Comment by KimTjik | March 25, 2011

  4. Oh, come on. A flamefest is very satisfying for the ego, but doesn’t get anything done. And to get something done in open source, you’d better step in and do it.

    Please also consider that really handling this problem is very hard. On LWN a history on this topic by Gentoo was cited (GLEP 57), discussing developments as far back as 2001, now they have working signatures for less than half their packages (and seem to be still missing parts of the functionality in their package management system).

    Comment by vonbrand | March 25, 2011

  5. I agree with you Pablo. There have been ad-hominem attacks from both sides, and emotions are definitely on high. I offer no defence for the Arch team on this point. I think exasperation is playing a role on both parties.

    I realize that IG has offered what he deems an interim solution. I’m not trying to put words in the mouths of the developers, but perhaps they thought the interim solution was too immature or didn’t solve the problem completely enough to be released to the thousands of Arch users around the globe. Again, I am just speculating.

    As an aside, I do wonder… IG, why did you stop at an interim solution and not work towards a full solution that could be packaged up and delivered to the Arch dev team? In such a scenario, things could have concluded in an entirely different, better way. Who knows, you could be a very valued security manager on the Arch team. I suppose I’m dreaming at this point. :)

    Comment by anonybot | March 25, 2011

    • > IG, why did you stop at an interim solution and not work towards a full solution that could be packaged up and delivered to the Arch dev team?

      Good question. Probably the most central reason was that I didn’t believe any such contribution from me would be accepted – I don’t like putting considerable work into something only to have it sit unused. The Arch devs portray themselves as friendly and desperately welcoming contributions, but the reality is much more tightly controlled and political (and not just in Arch). I am not the first to try to accomplish something with package signing – they have actively resisted and rejected contributions in this area for years. Thus I felt making people more aware of their dysfunctional and careless dev team was a more valuable approach, in addition to the interim solutions. Also, the interim solutions don’t conflict with pacman’s approach to package signing. So their willingness to implement them would have signaled a genuine willingness to accept larger contributions. Everything I have put in their hands has been rejected and ignored, nothing acted upon. This doesn’t encourage further investment. Paccheck was my answer to that – it works without any help from the Arch devs (they wouldn’t even allow me to add something as trivial as SHA256 sums to their database to make it more efficient). Complete hostility to innovation – ego-invested and too comfortable with a dysfunctional status quo.

      > In such a scenario, things could have concluded in an entirely different, better way. Who knows, you could be a very valued security manager on the Arch team. I suppose I’m dreaming at this point. :)

      Yes, you are. :) In dreams we tend to focus on just what we want to see, so we don’t notice that the larger context of the dream is missing, or that the elements would not fit together in reality. You are ignoring the realities of Arch development (and lack thereof). Like many dreamers, perhaps you don’t want to wake up – understandable.

      Comment by igurublog | March 26, 2011


    GNOME and various other pieces of software dont sign the tarballs they distribute: http://lwn.net/Articles/435533/

    What good is if Arch signs their distributed packages if the place where the sources comes from have been compromised?

    Comment by chris | March 26, 2011

    • Linux has many security issues. Weeks ago Bernard Baeyens and I discussed unsigned code from Xorg and some of the security implications of that.

      Yet pointing to one security problem as justification for ignoring another (larger one) is not productive – it’s just a form of denial and resistance to innovation. The scale of these problems is also different. Mirrors that hold source code are generally smaller, and thus more easily monitored and managed, usually by known and trusted parties. This is different than end-user package mirrors, some of which are run anonymously all over the world. This is why most distributions take package signing seriously even if they are more lax with source code. That’s not making an excuse for the developers not signing their work – it is inexcusable. Yet it is some explanation for how they have managed to avoid serious intrusion thus far. It is security through obscurity, well known to be a dangerous slope.

      Packagers using such unsigned source code need to take extra measures to help ensure that the source code they’re using is authentic, lest they package compromised material. This happened to Gentoo with the Backdoor Found In UnrealIRCd Source.

      Comment by igurublog | March 26, 2011

  7. A Followup

    It’s good to see this issue being more widely discussed, even if that means the obligatory fanboy attacks, developer denials and accusations, etc. This too is part of how things are accomplished in open source development. This is exactly what the censorship on the Arch forums wasn’t allowing – the discussion and understanding to evolve. But I realize it takes patience to read it all. My quiet developer’s blog is quite a mess – but this too shall pass, and I’m glad it served a purpose. I’m a big believer in everyone having their say, uncensored and unedited.

    This blog now contains a snapshot of an evolving process. I have spoken with many people the last few weeks, inside and outside of Arch, publicly and privately, and along the way I formed some working conclusions and hypotheses, which I have shared. I have also shared the raw data in the form of discussions and links, and given readers the freedom to add whatever you felt I had missed – your comments are just as visible as mine. I’ve seen a couple of accusations elsewhere that I deleted or blocked comments on this blog. Let me assure you that this is not the case. One of Allan’s comments from last week was found in the spam folder – it has been restored, and I emailed him an apology (unanswered). I’m not aware of any other comments that didn’t make it to the blog, but if it happens, an email to me will correct that. This blog receives hundreds of spam comments, so some automatic filtering is required – your patience is appreciated. To date, only one legit comment was temporarily blocked as spam.

    It was and is my intention to give everyone as much information as possible on this, including my own analyses and viewpoints. Yet everyone is capable of drawing their own working conclusions, and assessing Arch’s security practices for their needs. You don’t need to negate or silence my opinion to form or publish your own. In general I consider the comments here productive, so my thanks to everyone who has shared their views in a productive way.

    My current view: What I’ve seen in Arch’s kitchen has changed the way I view the restaurant. The common dev attitude is careless and some of their practices incompetent (this doesn’t apply to every Arch dev – some aren’t happy with it themselves). Their leadership is largely closed to innovation and contributions, which is why dev contributions in Arch are sharply declining. And the attitude toward users is arrogant and derisive. For their part, Arch users are too accepting of censorship and ill treatment, which inhibits the evolution of Arch. Given all this, I simply don’t trust Arch, nor do I wish to be associated with it. I am a person who bends over backwards to help and accomodate inexperienced users, and frankly I am embarrassed by the attitudes in the Arch dev and user community. Attitude plays a big role in security as well, which is a difficult development area in the best of conditions. Even if they implement package signing tomorrow, I now know they don’t take their users’ security seriously, or even much consider it at all.

    So I have already begun the process of moving on from Arch. I hope Arch, or some fork of it, matures in the years to come. I think some technical aspects of Arch have much to contribute to the overall Linux community. But contributing carelessness and arrogance is no welcome gift – I hope these attitudes are not a trend. From what I’ve seen there are distros out there with devs that have a much healthier attitude and geniune respect for their users. I think they deserve our attention and support more than Arch.

    As such, I don’t consider this my problem anymore, not directly, but I was glad LWN picked up the story. They have certainly impressed me with the level of information they provide to their subscribers – the problems with the Arch dev team are complex, and their article did not shy away from addressing those complexities. As for the Arch community, I will let that response speak for itself.

    Comment by igurublog | March 26, 2011

    • “I’m a big believer in everyone having their say, uncensored and unedited.”

      Good, but that doesn’t mean you don’t need to behave well when being a guest, does it? There’s no such thing as absolute freedom, something that only would be possible as being the singularity of all.

      Don’t burn bridges for the sake of vanity. Learn from this experience and do it differently next time. Keep a distance to folks only cheering a discontent, because they usually disappear as soon as you expect real work to be done. Be more patience and you mature as well. This isn’t a magic coin with only one side.

      Comment by KimTjik | March 26, 2011

      • > that doesn’t mean you don’t need to behave well when being a guest, does it?

        Actually, that’s exactly what it means. If I say people here have to “behave well”, then depending on my definition of that, I will begin censoring and editing them. I don’t find you behaving particularly well, for example – every comment you make seems to be aimed at taking a jab at someone else, usually me. You cloak this in semi-polite language like is expected on the Arch forum, but the hostility of your message still comes through. Yet obviously you feel this is helping you in some way.

        If I disagree with someone, I can state my disagreement. I don’t need to silence them to make my own views heard. This difference of approach to dissent is often missed, especially in online forums. They try to silence (delete) what is said instead of merely addressing it.

        That said, I encourage everyone to ‘behave well’, but I know everyone’s definition of that will vary. I also know that sometimes people are upset and want to have their say, their way. So be it. I’m not afraid of what you have to say. I like to hear different perspectives, even if I don’t subscribe to them.

        Comment by igurublog | March 26, 2011

        • No need to be suspicious. I’ve no hidden agenda, and there’s no need to read between my lines. I wish you well, however hard that might be to see in text, but if you ever met me you would believe it.

          That doesn’t mean I’m telling everyone what they want to hear. No, on the contrary my whole adult life has been dedicated to voluntarily help people in need, something that demands straightforwardness both in relation to benefiter and the ones you defend him or her against, be it individuals or authorities. This isn’t the proper forum to go into detail, and it doesn’t have any direct relation to this matter, but you might get an idea of who I am.

          Arch isn’t as an important part of my life, as you might have understood it. It’s as important to me as my toolbox; yes it’s a set of tools. As I’ve written before Arch forum doesn’t fill any social function whatsoever. It seems like read too much into why I made some responses here, as if I feel an urge to defend its forum. It’s not my family, but it fills the exact function I expect: a technical troubleshooter.

          You’re free to draw your own conclusions, which by the nature of these anonymous discussions won’t affect either of us much. I’m of course not particularly pleased to read your judgement about me, but what can I do about it? I suppose this might become my last comment here, and most have been the result of small tea times while working.

          I don’t support your way of handling this matter, but it’s not a personal matter. Take care and enjoy life!

          Comment by KimTjik | March 26, 2011

  8. Thank you for your blog and warnings about the problems with Arch security.

    As an Arch user (soon to be former), I was appalled by the attitude of Arch’s developers and the trolls who support them.
    Instead of addressing the issue, they display incredible arrogance trying to hush it up.

    The most reasonable approach is to assume that Arch has been bugged for a numbers of years, and the people behind it are trying to keep it that way. Anything else would be an exercise in PC nonsense.

    People who value their work are eager to protect it from spoilage, Arch developers are eager to… keep it unprotected, enough said.

    I think Arch needs a fork… yesterday.

    Comment by Sam Band | June 5, 2011

    • >The most reasonable approach is to assume that Arch has been bugged for a numbers of years, and the people behind it are trying to keep it that way.

      I think that’s a reasonable assumption. If it’s not the case, it’s certainly the perfect platform. And beyond just package signing (which they’re allegedly getting close to implementing), I wouldn’t trust them – attitude counts for a lot in security, and theirs is just bad and careless.

      > I think Arch needs a fork… yesterday.

      I hope so – the design has some nice aspects. At any rate, Aptosid has been working well for me.

      Comment by IgnorantGuru | June 5, 2011

Sorry, the comment form is closed at this time.

%d bloggers like this: