yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1592043] Re: os-brick 1.4.0 increases volume setup failure rates

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1592043@xxxxxxxxxxxxxxxxxx>
Date: Mon, 01 Aug 2016 21:13:46 -0000
Reply-to: Bug 1592043 <1592043@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

Reviewed:  https://review.openstack.org/329769
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4a8f2b0d44ee10dfac2d3d828cd9dc574d5ddbb2
Submitter: Jenkins
Branch:    master

commit 4a8f2b0d44ee10dfac2d3d828cd9dc574d5ddbb2
Author: Angus Lees <gus@xxxxxxxxxx>
Date:   Wed Jun 15 15:46:38 2016 +1000

    Initialise oslo.privsep early in main
    
    Any process using oslo.privsep should now initialise the library before
    first use with things like the rootwrap command to use.
    
    This should be done near the top of main() in any command that expects
    to make privileged calls via oslo.privsep (eg: nova-compute, and not
    nova-api).
    
    See I3ea73e16b07a870629e7d69e897f2524d7068ae8 for the corresponding
    change in oslo.privsep.
    
    Change-Id: I3a52f762deb176fe9201b2a0f0da363057f8aaec
    Depends-On: I52259e2023e277e8fd62be5df4fd7f799e9b36d7
    Closes-Bug: #1592043


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1592043

Title:
  os-brick 1.4.0 increases volume setup failure rates

Status in OpenStack Compute (nova):
  Fix Released
Status in os-brick:
  In Progress
Status in oslo.privsep:
  New

Bug description:
  Since merging upper constraints 1.4.0 into upper-constraints, the
  multinode grenade jobs are hitting a nearly 1/3 failure rate on boot
  from volume scenarios around volume setup. This would be on Newton
  code using Mitaka configs.

  Representative failures are of the following form:
  http://logs.openstack.org/71/327971/5/gate/gate-grenade-dsvm-neutron-
  multinode/f2690e3/logs/new/screen-n-cpu.txt.gz?level=WARNING#_2016-06-13_15_22_59_095

  The 1/3 failure rate is suspicious, and in the past has often hinted
  towards a race condition interacting between parallel API requests.

  The failure rate increase can be seen here -
  http://tinyurl.com/zrq35e8

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1592043/+subscriptions