yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #93872
[Bug 2062127] Re: Rebuild from BFV Instance Snapshot is failed
Adding the error trace here since the paste could expire
Apr 18 09:29:40 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:40.553 2499881 INFO nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Rebuilding instance
Apr 18 09:29:45 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:45.986 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance shutdown successfully after 5 seconds.
Apr 18 09:29:45 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:45.994 2499881 INFO nova.virt.libvirt.driver [-] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance destroyed successfully.
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.004 2499881 INFO nova.virt.libvirt.driver [-] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Instance destroyed successfully.
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.220 2499881 INFO os_vif [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:ea:b9:d7,bridge_name='qbrc6b8410e-fb',has_traffic_filtering=True,id=c6b8410e-fbdc-45b6-b3b3-6dd5515226c5,network=Network(579804e9-8b7c-4402-b23b-02a89c31284d),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapc6b8410e-fb')
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.266 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Deleting instance files /var/lib/nova/instances/43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd_del
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.268 2499881 INFO nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Deletion of /var/lib/nova/instances/43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd_del complete
Apr 18 09:29:46 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:46.432 2499881 WARNING nova.virt.libvirt.driver [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd could not be found.
Apr 18 09:29:47 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:47.520 2499881 WARNING nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Timeout waiting for ['volume-reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987'] for instance with vm_state active and task_state rebuilding. Event states are: volume-reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987: timed out after 0.00 seconds: eventlet.timeout.Timeout: 0 seconds
Apr 18 09:29:47 dev-compute1 nova-compute[2499881]: /openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_serialization/jsonutils.py:180: UserWarning: Cannot convert <Timeout at 0x7c580c578b80 seconds=0> to primitive, will raise ValueError instead of warning in version 3.0
Apr 18 09:29:47 dev-compute1 nova-compute[2499881]: warnings.warn("Cannot convert %r to primitive, will raise ValueError "
Apr 18 09:29:48 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:48.523 2499881 INFO nova.compute.manager [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Successfully reverted task state from rebuilding on failure for instance.
Apr 18 09:29:48 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server [None req-3dfd2905-9dbe-4a2d-ac7c-e0a7328c39f3 4841276dcdbe4ab096ef60b1744c4fa9 f2e52a5c5d1c4ca1b51274619b517e0e - - default default] Exception during message handling: ValueError: Circular reference detected
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 203, in decorated_function
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3826, in rebuild_instance
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server self._do_rebuild_instance_with_claim(
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3912, in _do_rebuild_instance_with_claim
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server self._do_rebuild_instance(
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 4104, in _do_rebuild_instance
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server self._rebuild_default_impl(**kwargs)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3693, in _rebuild_default_impl
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server self._rebuild_volume_backed_instance(
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 3613, in _rebuild_volume_backed_instance
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server with self.virtapi.wait_for_instance_event(instance, events,
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server next(self.gen)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 559, in wait_for_instance_event
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server self._wait_for_instance_events(
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 471, in _wait_for_instance_events
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server actual_event = event.wait()
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 436, in wait
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server instance_event = self.event.wait()
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server result = hub.switch()
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return self.greenlet.switch()
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server eventlet.timeout.Timeout: 0 seconds
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 244, in inner
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server raise self.value
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 166, in decorated_function
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server raise self.value
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/manager.py", line 157, in decorated_function
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1437, in decorated_function
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server with EventReporter(context, event_name, host, instance_uuid,
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/compute/utils.py", line 1400, in __exit__
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server objects.InstanceActionEvent.event_finish_with_failure(
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/objects/base.py", line 355, in wrapper
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return fn.__get__(None, obj)(*args, **kwargs)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server result = cls.indirection_api.object_class_action_versions(
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return cctxt.call(context, 'object_class_action_versions',
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server result = self.transport._send(
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return self._driver.send(target, ctxt, message,
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server msg = rpc_common.serialize_msg(msg)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_messaging/_drivers/common.py", line 292, in serialize_msg
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-27.3.0/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 202, in dumps
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return json.dumps(obj, default=default, **kwargs)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server **kw).encode(obj)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server chunks = self.iterencode(o, _one_shot=True)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server return _iterencode(o, 0)
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server ValueError: Circular reference detected
2024-04-18 09:29:48.537 2499881 ERROR oslo_messaging.rpc.server
Apr 18 09:29:49 dev-compute1 nova-compute[2499881]: 2024-04-18 09:29:49.052 2499881 WARNING nova.compute.manager [req-792db0e8-3379-438e-9fb9-f5e1c8970907 req-c0b887a2-09fc-449a-bf52-7a4400bd9067 51c5ce5de8844debac57ac395dc0b5c1 e1204eb8867846eea31b90b95c76e8e5 - - default default] [instance: 43dc3e18-5ca7-4df7-8cac-b5d96eba6bfd] Received unexpected event network-vif-unplugged-c6b8410e-fbdc-45b6-b3b3-6dd5515226c5 for instance with vm_state active and task_state None.
** Project changed: cinder => nova
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2062127
Title:
Rebuild from BFV Instance Snapshot is failed
Status in OpenStack Compute (nova):
In Progress
Bug description:
Issue: Rebuild from Booted From Volume Instance Snapshot is failed
Timeout waiting for ['volume-
reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987'] for instance with
vm_state active and task_state rebuilding. Event states are: volume-
reimaged-d4b22ad0-dcc7-485f-bab7-b4d82eb61987: timed out after 0.00
seconds: eventlet.timeout.Timeout: 0 seconds
Expected State: Min Timeout Per Gb is 20 second.
Analysis:
I have followed below steps:
1. Create Booted From Volume Instance
https://paste.openstack.org/show/bWBSIO7Mr9OIHJna9wk7/
2. Create snapshot of Booted From Volume Instance
https://paste.openstack.org/show/bJVPXDYB45FsRbTIhel5/
3. Rebuild from Booted From Volume Instance Snapshot
https://paste.openstack.org/show/bHau3AXs789hpYSnLFvW/
Status: It is failed and throws below error logs:
https://paste.openstack.org/show/bRsJ69NmWzzDVbFiaWW5/
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2062127/+subscriptions
