← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #84033

[Bug 1897424] [NEW] attach volume consume too long time, and if attaching another volume at the same time, create the bdm will be timeout

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] Unexpected exception in API method: MessagingTimeout: Timed out waiting for a reply to message ID 998b0b462864458f9a24dbb0896f9093
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/inspur/api/volumes.py", line 331, in create
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     supports_multiattach=supports_multiattach)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/inspur/compute/api_inspur.py", line 1327, in attach_volume
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     is_root=is_root)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/inspur/compute/api_inspur.py", line 1345, in _attach_volume
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     device_type=device_type, tag=tag)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/api.py", line 3958, in _create_volume_bdm
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     multiattach=volume['multiattach'])
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 897, in reserve_block_device_name
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     retry=self.retry)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     retry=retry)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     call_monitor_timeout, retry=retry)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     call_monitor_timeout)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     'to message ID %s' % msg_id)
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi MessagingTimeout: Timed out waiting for a reply to message ID 998b0b462864458f9a24dbb0896f9093
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi
2020-09-24 12:26:46.763 29 INFO nova.api.openstack.wsgi [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi
2020-09-24 12:26:46.763 29 INFO nova.api.openstack.wsgi [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2020-09-24 12:26:46.789 29 INFO nova.osapi_compute.wsgi.server [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 521, in handle_one_response
    write(b''.join(towrite))
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 462, in write
    wfile.flush()
  File "/usr/lib64/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
    tail = self.send(data, flags)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1897424

Title:
  attach volume consume too long time, and if attaching another volume
  at the same time, create the bdm will be timeout

Status in OpenStack Compute (nova):
  New

Bug description:
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] Unexpected exception in API method: MessagingTimeout: Timed out waiting for a reply to message ID 998b0b462864458f9a24dbb0896f9093
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/inspur/api/volumes.py", line 331, in create
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     supports_multiattach=supports_multiattach)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/inspur/compute/api_inspur.py", line 1327, in attach_volume
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     is_root=is_root)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/inspur/compute/api_inspur.py", line 1345, in _attach_volume
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     device_type=device_type, tag=tag)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/api.py", line 3958, in _create_volume_bdm
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     multiattach=volume['multiattach'])
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 897, in reserve_block_device_name
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     retry=self.retry)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     retry=retry)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     call_monitor_timeout, retry=retry)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     call_monitor_timeout)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi     'to message ID %s' % msg_id)
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi MessagingTimeout: Timed out waiting for a reply to message ID 998b0b462864458f9a24dbb0896f9093
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi
  2020-09-24 12:26:46.763 29 INFO nova.api.openstack.wsgi [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2020-09-24 12:26:46.758 29 ERROR nova.api.openstack.wsgi
  2020-09-24 12:26:46.763 29 INFO nova.api.openstack.wsgi [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2020-09-24 12:26:46.789 29 INFO nova.osapi_compute.wsgi.server [req-408c9350-fe1e-11ea-b09f-ad32aa0197ce req-1903c5bf-9263-4afe-b72f-9f3e7295c790 4e6abdba9163438dafc7219cb119d222 f34ca0cc0b664331ba7f6220a816bd81 - default default] Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 521, in handle_one_response
      write(b''.join(towrite))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py", line 462, in write
      wfile.flush()
    File "/usr/lib64/python2.7/socket.py", line 303, in flush
      self._sock.sendall(view[write_offset:write_offset+buffer_size])
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
      tail = self.send(data, flags)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
      return self._send_loop(self.fd.send, data, flags)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
      return send_method(data, *args)
  error: [Errno 104] Connection reset by peer

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1897424/+subscriptions
  Thread Previous • Date Previous • • Thread Next