openstack team mailing list archive

[corvus@xxxxxxxxxxxx (James E. Blair)]

Hi,

We encountered some problems today that delayed several patches from
going in (I believe they have all been merged now).  It was a bit of a
rough day, but I'm pleased that several people pitched in to help
identify and solve those problems, and we've made some good progress
overall.  Here's a quick update:

1) Syslogs are now available.

   If your patch runs into a problem, here's where to look: first,
   follow the link in Gerrit to Jenkins for the failed
   gate-integration-tests-devstack job.  For example:

   https://jenkins.openstack.org/job/gate-integration-tests-devstack-vm/273/

   To see the output from devstack (ie, install, setup, and test
   output), click the "[raw]" link next to "Console Log" to see the full
   console output.  eg:

   https://jenkins.openstack.org/job/gate-integration-tests-devstack-vm/273/consoleText

   To see the syslog output from the OpenStack components themselves,
   click the syslog.txt link under "Build Artifacts".  eg:
   
   https://jenkins.openstack.org/job/gate-integration-tests-devstack-vm/273/artifact/logs/syslog.txt

   That starts right before devstack is invoked.

2) Nova's install_requires in setup.py has been removed, reducing our
   dependency on external network resources.

   A number of jobs failed because the website of the developer of
   pylint was slow or unavailable for much of the day.

   This brought to our attention that nova's setup.py was including the
   contents of pip-requires as the setuptools install_requires, which
   gets installed when python setup.py develop is run.  Monty Taylor,
   Dean Troyer, and Anthony Young all pitched in to identify the problem
   and devise a solution.  Here's the commit message from Monty's patch
   to nova:

   ---------------------------------------------------------------
   Loading install_requires with the contents of pip-requires
   isn't getting us any real beneift and is causing issues.
   
   a) It can conflict with installing nova into an environment
   where deps have been installed from packages (devstack)
   b) It breaks the ability to use -e git urls in pip-requires
   which we want to start using for python-novaclient and
   python-keystoneclient
   c) It causes spurious network traffic when we're trying to
   test things.
   
   At the same time, since we are not expecting anyone to
   install nova from setup.py for production, the normal benefit
   of the feature is not needed.
   ---------------------------------------------------------------

   With that change in, we should avoid seeing a repeat of the bulk of
   the problems today.

   Our inclination for how to handle dependencies going forward is:

   * List OS package dependencies in devstack wherever possible.

   * List python dependencies that are not yet available in OS packages
     in devstack, considering each one a bug that should be resolved by
     an OS package being available.

3) Anthony identified a bug in nova that appears non-deterministically
   in the integration test job:

   https://bugs.launchpad.net/nova/+bug/912033

   Any future patches have the potential to run into it until it's
   fixed, though it seems to be rather infrequent.  The exercise.sh
   output looks like:

      SKIP swift
      SKIP volumes
      PASS euca
      FAILED floating_ips

   And the relevant syslog entry:

   Jan 4 21:01:17 devstack-1325709057 2012-01-04 21:01:17,430 INFO nova.api.openstack.wsgi [15ac1bc9-d5c7-4f89-8c65-1d8870c4d4e3 d70af9385f7b4cbb828d54d6d1045a33 cb276eb4e5fe489ea73b696c69d5e0a6] http://127.0.0.1:8774/v1.1/cb276eb4e5fe489ea73b696c69d5e0a6/servers/detail returned with HTTP 404

Thanks again to everyone who helped and those who were patient as their
patches were struggling to be merged.

-Jim