touch-packages team mailing list archive

["Edward Z. Yang" <ezyang@xxxxxxxxxxxxxxx>]

** Attachment added: "apt-get update -o Debug::Acquire::http=true log"
   https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1413428/+attachment/4303390/+files/log2

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1413428

Title:
  apt-get's HTTP pipeline desynchronizes, hilarity ensues

Status in apt package in Ubuntu:
  New

Bug description:
  tl;dr: apt-get improperly handles servers which respond 404 with HTTP
  content to a Range query, resulting in a desychronized HTTP buffer and
  hilarious bugs.

  OK, this is going to be a long one. Where to begin? I was updating my
  Aptitude packages and noticed that my Dropbox source was not updating
  correctly:

  Err http://linux.dropbox.com utopic/main amd64 Packages
    Bad header line

  Silly Dropbox, not checking their package list! I report it to them,
  and they report back that the URL being fetched seems to be giving
  back a well formed HTTP response, and that they couldn't reproduce. I
  verify that is the case. We ponder the problem for a while, clearing
  caches and permuting the source.list line, and finally someone
  suggests running -o Debug::Acquire::http=true. I take the log and
  scroll to the error line:

  Answer for: http://linux.dropbox.com/ubuntu/dists/utopic/main/i18n/Translation-en_US.lzma
  Package: dropbox
  Priority: optional
  Section: gnome
  Installed-Size: 404
  Maintainer: Rian Hunter <rian@xxxxxxxxxxx>
  Architecture: amd64
  Version: 2.10.0
  Replaces: nautilus-dropbox
  Provides: nautilus-dropbox
  Depends: procps, python-gtk2 (>= 2.12), python (>= 2.5), libatk1.0-0 (>= 1.20.0), libc6 (>= 2.4), libcairo2 (>= 1.6.0), libglib2.0-0 (>= 2.16.0), libgtk2.0-0 (>= 2.12.0), libpango1.0-0 (>= 1.20.1)
  Suggests: nautilus (>= 2.16.0), python-gpgme (>= 0.1)
  Breaks: nautilus-dropbox
  Filename: pool/main/dropbox_2.10.0_amd64.deb
  Size: 94296
  MD5sum: 39d2f6558a35defbb4e3346c66651da9
  SHA1: f68b9e102b96a72f37e79f74ac7030cd881db284
  SHA256: 5ddf820c1f2e2b12c7824f9691d09f204c33ec7073736891544b774f7e0a0812
  Description: cloud synchronization engine - CLI and Nautilus extension
   Dropbox is a free service that lets you bring your photos, docs, and videos
   anywhere and share them easily.
   .
   This package provides a command-line tool and a Nautilus extension that
   integrates the Dropbox web service with your GNOME Desktop.
  Homepage: https://www.dropbox.com/

  Err http://linux.dropbox.com utopic/main amd64 Packages
    Bad header line

  Well. That *sort* of looks reasonable. But I looked around at some of
  the other responses in the log, and I realized, "Oh shit, these should
  be HTTP headers!"

  Answer for: http://debian.stanford.edu/ubuntu/dists/utopic/InRelease
  HTTP/1.1 404 Not Found
  Date: Wed, 21 Jan 2015 22:54:17 GMT
  Server: Apache
  Vary: Accept-Encoding
  Content-Length: 227
  Content-Type: text/html; charset=iso-8859-1

  So, why, then, does Apt think that the content is the HTTP headers? I
  was reminded of an old bug I encountered in MediaWiki:

  https://issues.apache.org/bugzilla/show_bug.cgi?id=40953
  https://bugzilla.mozilla.org/show_bug.cgi?id=363109#c12
  https://phabricator.wikimedia.org/T19537

  Checking the source, it does seem apt pipelines requests by default,
  so if it desynchronized in its processing of the HTTP stream, that
  would be bad news. Seeking back in the log, we see this:

  
  Answer for: http://linux.dropbox.com/ubuntu/dists/utopic/main/binary-amd64/Packages.bz2
  HTTP/1.1 404 Not Found
  Server: nginx
  Date: Wed, 21 Jan 2015 22:54:17 GMT
  Content-Type: text/html
  Content-Length: 162
  Connection: keep-alive
  Content-Range: bytes */1142

  GET /ubuntu/dists/utopic/main/binary-i386/Packages.bz2 HTTP/1.1
  Host: linux.dropbox.com
  Cache-Control: max-age=0
  Range: bytes=2635-
  If-Range: Mon, 29 Dec 2014 22:30:54 GMT
  User-Agent: Debian APT-HTTP/1.3 (1.0.9.2ubuntu2)

  
  Answer for: http://linux.dropbox.com/ubuntu/dists/utopic/main/binary-i386/Packages.bz2
  <html>
  <head><title>404 Not Found</title></head>
  <body bgcolor="white">
  <center><h1>404 Not Found</h1></center>
  <hr><center>nginx</center>
  </body>
  </html>

  Bingo.

  By the way, you won't be able to reproduce the error unless you can
  induce apt-get to send the If-Range/Range header to the server. apt-
  get only sends the header if it has some cached partial lists (which,
  BY THE WAY, are not cleared when you clear your apt cache, WHY?!) I'll
  attach some files which you can put in /var/lib/apt/lists/partial
  which, along with adding

  deb [arch=amd64,i386] http://linux.dropbox.com/ubuntu utopic main

  to your sources list, should cause you to be able to reproduce the
  error.

  For what it's worth, I also think the server is also partially to
  blame; I'm not sure but 404 doesn't seem like the right code to return
  here. I'll also attach full HTTP cache logs.

  Can forward to upstream on request. (In fact, I'll probably do it
  anyway.)

  ProblemType: Bug
  DistroRelease: Ubuntu 14.10
  Package: apt 1.0.9.2ubuntu2
  ProcVersionSignature: Ubuntu 3.16.0-28.38-generic 3.16.7-ckt1
  Uname: Linux 3.16.0-28-generic x86_64
  NonfreeKernelModules: openafs
  ApportVersion: 2.14.7-0ubuntu8.1
  Architecture: amd64
  Date: Wed Jan 21 15:27:02 2015
  EcryptfsInUse: Yes
  InstallationDate: Installed on 2013-11-21 (426 days ago)
  InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Release amd64 (20131016.1)
  SourcePackage: apt
  UpgradeStatus: Upgraded to utopic on 2014-12-04 (48 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1413428/+subscriptions