yahoo-eng-team team mailing list archive

[Jeremy Stanley <fungi@xxxxxxxxxxx>]

Public bug reported:

Pip 1.5 (released January 2nd, 2014) will by default refuse to download
packages which are linked from PyPI but not hosted on pypi.python.org.
The workaround is to whitelist these package names individually with
both the --allow-external and --allow-insecure options.

These options are new in pip 1.4, so encoding them will break for people
trying to use pip 1.3.x or earlier. Those earlier versions of pip are
not secure anyway since they don't connect via HTTPS with host
certificate validation, so we should be encouraging people to use 1.4
and later anyway.

The --allow-insecure option is transitioning to a clearer --allow-
unverified option name starting with 1.5, but the new form does not work
with pip before 1.5 so we should use the old version for now to allow
people to transition gracefully. The --allow-insecure form won't be
removed until at least pip 1.7 according to comments in the source code.

Virtualenv 1.11 (released the same day) bundles pip 1.5 by default, and
so requires these workarounds when using requirements external to PyPI.
Be aware that 1.11 is broken for projects using sitepackages=True in
their tox.ini. The fix is
https://github.com/pypa/virtualenv/commit/a6ca6f4 which is slated to
appear in 1.11.1 (no ETA available). We've worked around it on our test
infrastructure with https://git.openstack.org/cgit/openstack-
infra/config/commit/?id=20cd18a for now, but that is hiding the
external-packages issue since we're currently running all tests with pip
1.4.1 as a result.

This bug will also be invisible in our test infrastructure for projects
listed as having the PyPI mirror enforced in openstack/requirements
(except for jobs which bypass the mirror, such as those for requirements
changes), since our update jobs will pull in and mirror external
packages and pip sees the mirror as being PyPI itself in that situation.

We'll use this bug to track necessary whitelist updates to tox.ini and
test scripts.

** Affects: nova
     Importance: Undecided
     Assignee: Jeremy Stanley (fungi)
         Status: In Progress

** Affects: openstack-ci
     Importance: Critical
     Assignee: Jeremy Stanley (fungi)
         Status: In Progress

** Affects: swift
     Importance: Undecided
     Assignee: Jeremy Stanley (fungi)
         Status: In Progress


** Tags: elastic-recheck grizzly-backport-potential havana-backport-potential pypi-mirror reviewday

** Changed in: openstack-ci
       Status: New => In Progress

** Also affects: nova
   Importance: Undecided
       Status: New

** Also affects: swift
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => In Progress

** Changed in: swift
       Status: New => In Progress

** Changed in: nova
     Assignee: (unassigned) => Jeremy Stanley (fungi)

** Changed in: swift
     Assignee: (unassigned) => Jeremy Stanley (fungi)

** Tags added: elastic-recheck reviewday

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1266513

Title:
  Some Python requirements are not hosted on PyPI

Status in OpenStack Compute (Nova):
  In Progress
Status in OpenStack Core Infrastructure:
  In Progress
Status in OpenStack Object Storage (Swift):
  In Progress

Bug description:
  Pip 1.5 (released January 2nd, 2014) will by default refuse to
  download packages which are linked from PyPI but not hosted on
  pypi.python.org. The workaround is to whitelist these package names
  individually with both the --allow-external and --allow-insecure
  options.

  These options are new in pip 1.4, so encoding them will break for
  people trying to use pip 1.3.x or earlier. Those earlier versions of
  pip are not secure anyway since they don't connect via HTTPS with host
  certificate validation, so we should be encouraging people to use 1.4
  and later anyway.

  The --allow-insecure option is transitioning to a clearer --allow-
  unverified option name starting with 1.5, but the new form does not
  work with pip before 1.5 so we should use the old version for now to
  allow people to transition gracefully. The --allow-insecure form won't
  be removed until at least pip 1.7 according to comments in the source
  code.

  Virtualenv 1.11 (released the same day) bundles pip 1.5 by default,
  and so requires these workarounds when using requirements external to
  PyPI. Be aware that 1.11 is broken for projects using
  sitepackages=True in their tox.ini. The fix is
  https://github.com/pypa/virtualenv/commit/a6ca6f4 which is slated to
  appear in 1.11.1 (no ETA available). We've worked around it on our
  test infrastructure with https://git.openstack.org/cgit/openstack-
  infra/config/commit/?id=20cd18a for now, but that is hiding the
  external-packages issue since we're currently running all tests with
  pip 1.4.1 as a result.

  This bug will also be invisible in our test infrastructure for
  projects listed as having the PyPI mirror enforced in
  openstack/requirements (except for jobs which bypass the mirror, such
  as those for requirements changes), since our update jobs will pull in
  and mirror external packages and pip sees the mirror as being PyPI
  itself in that situation.

  We'll use this bug to track necessary whitelist updates to tox.ini and
  test scripts.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1266513/+subscriptions