yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1689278] [NEW] Compute Node goes down periodically

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andres Sanchez <1689278@xxxxxxxxxxxxxxxxxx>
Date: Mon, 08 May 2017 10:48:23 -0000
Reply-to: Bug 1689278 <1689278@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hello,

Recently a node on my Openstack Cloud is going down periodically without
nothing relevant happening, i have not found an explanation for this
yet.

I have the following environment:

1.- Version 13.1.2
2.- KVM
3.- Ceph.
4.- Neutron with VXlan tunneling.

After reviewing nova logs i have seen these Error lines repeteadly:

2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] Unexpected exception in API method
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/remote_consoles.py", line 56, in get_vnc_console
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     console_type)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 171, in wrapped
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return func(self, context, target, *args, **kwargs)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 151, in wrapped
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2933, in get_vnc_console
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     instance=instance, console_type=console_type)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 574, in get_vnc_console
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     instance=instance, console_type=console_type)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     retry=self.retry)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     retry=retry)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 501, in _send
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 379, in wait
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 277, in get
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID c670a34162914270ad9ca6b0bd40b2fb
2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions 
2017-05-05 00:33:51.119 10983 INFO nova.api.openstack.wsgi [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'

Thank you in advance for your response.

Best regards,

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "nova-api.log-20170505-1493969701"
   https://bugs.launchpad.net/bugs/1689278/+attachment/4873430/+files/nova-api.log-20170505-1493969701

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1689278

Title:
  Compute Node goes down periodically

Status in OpenStack Compute (nova):
  New

Bug description:
  Hello,

  Recently a node on my Openstack Cloud is going down periodically
  without nothing relevant happening, i have not found an explanation
  for this yet.

  I have the following environment:

  1.- Version 13.1.2
  2.- KVM
  3.- Ceph.
  4.- Neutron with VXlan tunneling.

  After reviewing nova logs i have seen these Error lines repeteadly:

  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] Unexpected exception in API method
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/remote_consoles.py", line 56, in get_vnc_console
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     console_type)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 171, in wrapped
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return func(self, context, target, *args, **kwargs)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 151, in wrapped
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2933, in get_vnc_console
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     instance=instance, console_type=console_type)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 574, in get_vnc_console
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     instance=instance, console_type=console_type)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     retry=self.retry)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     retry=retry)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 501, in _send
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 379, in wait
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 277, in get
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID c670a34162914270ad9ca6b0bd40b2fb
  2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions 
  2017-05-05 00:33:51.119 10983 INFO nova.api.openstack.wsgi [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'

  Thank you in advance for your response.

  Best regards,

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1689278/+subscriptions
Follow ups

[Bug 1689278] Re: Compute Node goes down periodically
From: Launchpad Bug Tracker, 2017-08-14