yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1597712] [NEW] Unexpected API Error on large scale: MessagingTimeout

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrew Kalach <akndex@xxxxxxxxx>
Date: Thu, 30 Jun 2016 10:59:47 -0000
Reply-to: Bug 1597712 <1597712@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
During 9.0RC2 Certification Testing Sutie run, the following error log appeared:

Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500)
(Request-ID: req-4bffec57-e890-486d-b197-2edf5aa19e97)

Steps to reproduce
==================
Install Rally on masternode and run task "create-and-attach-volume.yaml":

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start /opt/stack/rally-scenarios/cinder/create-and-attach-volume.yaml

Expected result
===============
Task run without errors

Actual result
=============
The following error logs:

2016-06-27T14:15:36.672815+00:00 err:  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions [req-4bffec57-e890-486d-b197-2edf5aa19e97 eb078fc45f524ec3a33276a9ebc4a307 82306fa61eae4bb99a308d72c919afe8 - - -] Unexpected exception in API method
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/volumes.py", line 316, in create
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     volume_id, device)
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 170, in wrapped
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return func(self, context, target, *args, **kwargs)
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 159, in inner
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 140, in inner
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packag
2016-06-27T14:15:36.676579+00:00 info:  2016-06-27 14:15:36.681 21518 INFO nova.api.openstack.wsgi [req-4bffec57-e890-486d-b197-2edf5aa19e97 eb078fc45f524ec3a33276a9ebc4a307 82306fa61eae4bb99a308d72c919afe8 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  199 compute nodes
  20 ceph nodes
  DVR enabled
  XVLAN
Image RC2 build:
  fuel-9.0-mos-495-2016-06-16_18-18-00.iso

Diagnostic snapshot:
  http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "Nova API logs"
   https://bugs.launchpad.net/bugs/1597712/+attachment/4692728/+files/nova-api.log.6.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1597712

Title:
  Unexpected API Error on large scale: MessagingTimeout

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  During 9.0RC2 Certification Testing Sutie run, the following error log appeared:

  Unexpected API Error. Please report this at
  http://bugs.launchpad.net/nova/ and attach the Nova API log if
  possible. <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP
  500) (Request-ID: req-4bffec57-e890-486d-b197-2edf5aa19e97)

  Steps to reproduce
  ==================
  Install Rally on masternode and run task "create-and-attach-volume.yaml":

  1. cd /opt/stack
  2. source .venv/bin/activate
  3. rally task start /opt/stack/rally-scenarios/cinder/create-and-attach-volume.yaml

  Expected result
  ===============
  Task run without errors

  Actual result
  =============
  The following error logs:

  2016-06-27T14:15:36.672815+00:00 err:  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions [req-4bffec57-e890-486d-b197-2edf5aa19e97 eb078fc45f524ec3a33276a9ebc4a307 82306fa61eae4bb99a308d72c919afe8 - - -] Unexpected exception in API method
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/volumes.py", line 316, in create
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     volume_id, device)
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 170, in wrapped
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return func(self, context, target, *args, **kwargs)
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 159, in inner
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 140, in inner
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2016-06-27 14:15:36.673 21518 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packag
  2016-06-27T14:15:36.676579+00:00 info:  2016-06-27 14:15:36.681 21518 INFO nova.api.openstack.wsgi [req-4bffec57-e890-486d-b197-2edf5aa19e97 eb078fc45f524ec3a33276a9ebc4a307 82306fa61eae4bb99a308d72c919afe8 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>

  Environment
  ===========
  Scale 200-node lab (ENV-10):
    3 controllers
    199 compute nodes
    20 ceph nodes
    DVR enabled
    XVLAN
  Image RC2 build:
    fuel-9.0-mos-495-2016-06-16_18-18-00.iso

  Diagnostic snapshot:
    http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1597712/+subscriptions
Follow ups

[Bug 1597712] Re: Unexpected API Error on large scale: MessagingTimeout
From: Michael Semenov, 2016-06-30