yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1994083] [NEW] "FileNotFoundError: [Errno 2] No such file or directory" is raised when dmidecode is not installed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Felipe Reyes <1994083@xxxxxxxxxxxxxxxxxx>
Date: Mon, 24 Oct 2022 23:17:02 -0000
Reply-to: Bug 1994083 <1994083@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

On s390x there is no "dmidecode" package[0], with the inclusion of this
change to get system uuid in privsep[1] nova regressed since privsep
raises FileNotFoundError when there is an attempt to execute a program
that doesn't exist, while subprocess would raise a higher level
exception as subprocess.CalledProcessError.

Steps to reproduce:

- Deploy an OpenStack cloud with Zed release on s390x
- Launch an instance that boots from a volume, like tempest test tempest.api.compute.servers.test_create_server.ServersTestBootFromVolume

Expected result:

- The launched instance reaches active/running state

Actual result:

- The instance goes in error state with the folloing fault:

Details: Fault: {'code': 500, 'created': '2022-10-24T22:11:44Z',
'message': 'Build of instance 25879943-2e64-456c-b500-96a8a8735736
aborted: [Errno 2] No such file or directory'}. Server boot request ID:
req-bb83a7cd-7cb2-4627-b4c8-4b6c05ba6068.

The /var/log/nova/nova-compute.log contains the following stacktrace:

49a 443c1aae740a4332bf85249b516d149a] [instance: 25879943-2e64-456c-b500-96a8a8735736] Instance failed block device setup: FileNotFoundError: [Errno 2] No such file or directory
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] Traceback (most recent call last):
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2087, in _prep_block_device
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     driver_block_device.attach_block_devices(
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 936, in attach_block_devi
ces
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     _log_and_attach(device)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 933, in _log_and_attach
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     bdm.attach(*attach_args, **attach_kwargs)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 46, in wrapped
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     ret_val = method(obj, context, *args, **kwargs)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 734, in attach
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     self._do_attach(context, instance, volume, volume_api,
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 713, in _do_attach
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     connector = virt_driver.get_volume_connector(instance)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1783, in get_volume_con
nector
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     return connector.get_connector_properties(
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 171, in trace_logging_wrapper
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     return f(*args, **kwargs)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/os_brick/initiator/connector.py", line 232, in get_connect
or_properties
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     connector.get_connector_properties(
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/nvmeof.py", line 772, in get
_connector_properties
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     suuid = priv_nvmeof.get_system_uuid()
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     return self.channel.remote_call(name, args, kwargs,
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     raise exc_type(*result[2])
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] FileNotFoundError: [Errno 2] No such file or directory
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] 
2022-10-24 18:11:43.134 1115327 ERROR nova.compute.manager [None req-bb83a7cd-7cb2-4627-b4c8-4b6c05ba6068 c1ae2fad28a24171badea4c539bb28a7 94686174ec6441559786f037d98e7159 - - 443c1aae740a4332bf85249b516d1
49a 443c1aae740a4332bf85249b516d149a] [instance: 25879943-2e64-456c-b500-96a8a8735736] Build of instance 25879943-2e64-456c-b500-96a8a8735736 aborted: [Errno 2] No such file or directory: nova.exception.Bu
ildAbortException: Build of instance 25879943-2e64-456c-b500-96a8a8735736 aborted: [Errno 2] No such file or directory

Environment:

- OpenStak Zed on Ubuntu Jammy

# dpkg -l  | grep nova
ii  nova-common                           3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - common files
ii  nova-compute                          3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - compute node base
ii  nova-compute-kvm                      3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                  3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - compute node libvirt support
ii  python3-nova                          3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute Python 3 libraries
ii  python3-novaclient                    2:18.1.0-0ubuntu1~cloud0                             all          client library for OpenStack Compute API - 3.x

- The storage type is Ceph
- The networking is OpenVSwitch

[0] https://packages.ubuntu.com/jammy-updates/dmidecode
[1] https://review.opendev.org/c/openstack/os-brick/+/836057

** Affects: cloud-archive
     Importance: Undecided
         Status: New

** Affects: cloud-archive/zed
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: Invalid

** Affects: os-brick
     Importance: Undecided
         Status: New

** Affects: python-os-brick (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: python-os-brick (Ubuntu Kinetic)
     Importance: Undecided
         Status: New

** Also affects: nova (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: nova (Ubuntu Kinetic)
   Importance: Undecided
       Status: New

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/zed
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1994083

Title:
  "FileNotFoundError: [Errno 2] No such file or directory" is raised
  when dmidecode is not installed

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive zed series:
  New
Status in OpenStack Compute (nova):
  Invalid
Status in os-brick:
  New
Status in python-os-brick package in Ubuntu:
  New
Status in python-os-brick source package in Kinetic:
  New

Bug description:
  On s390x there is no "dmidecode" package[0], with the inclusion of
  this change to get system uuid in privsep[1] nova regressed since
  privsep raises FileNotFoundError when there is an attempt to execute a
  program that doesn't exist, while subprocess would raise a higher
  level exception as subprocess.CalledProcessError.

  Steps to reproduce:

  - Deploy an OpenStack cloud with Zed release on s390x
  - Launch an instance that boots from a volume, like tempest test tempest.api.compute.servers.test_create_server.ServersTestBootFromVolume

  Expected result:

  - The launched instance reaches active/running state

  Actual result:

  - The instance goes in error state with the folloing fault:

  Details: Fault: {'code': 500, 'created': '2022-10-24T22:11:44Z',
  'message': 'Build of instance 25879943-2e64-456c-b500-96a8a8735736
  aborted: [Errno 2] No such file or directory'}. Server boot request
  ID: req-bb83a7cd-7cb2-4627-b4c8-4b6c05ba6068.

  The /var/log/nova/nova-compute.log contains the following stacktrace:

  49a 443c1aae740a4332bf85249b516d149a] [instance: 25879943-2e64-456c-b500-96a8a8735736] Instance failed block device setup: FileNotFoundError: [Errno 2] No such file or directory
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] Traceback (most recent call last):
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2087, in _prep_block_device
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     driver_block_device.attach_block_devices(
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 936, in attach_block_devi
  ces
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     _log_and_attach(device)
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 933, in _log_and_attach
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     bdm.attach(*attach_args, **attach_kwargs)
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 46, in wrapped
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     ret_val = method(obj, context, *args, **kwargs)
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 734, in attach
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     self._do_attach(context, instance, volume, volume_api,
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 713, in _do_attach
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     connector = virt_driver.get_volume_connector(instance)
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1783, in get_volume_con
  nector
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     return connector.get_connector_properties(
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 171, in trace_logging_wrapper
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     return f(*args, **kwargs)
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/os_brick/initiator/connector.py", line 232, in get_connect
  or_properties
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     connector.get_connector_properties(
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/nvmeof.py", line 772, in get
  _connector_properties
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     suuid = priv_nvmeof.get_system_uuid()
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 271, in _wrap
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     return self.channel.remote_call(name, args, kwargs,
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]   File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 215, in remote_call
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]     raise exc_type(*result[2])
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] FileNotFoundError: [Errno 2] No such file or directory
  2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] 
  2022-10-24 18:11:43.134 1115327 ERROR nova.compute.manager [None req-bb83a7cd-7cb2-4627-b4c8-4b6c05ba6068 c1ae2fad28a24171badea4c539bb28a7 94686174ec6441559786f037d98e7159 - - 443c1aae740a4332bf85249b516d1
  49a 443c1aae740a4332bf85249b516d149a] [instance: 25879943-2e64-456c-b500-96a8a8735736] Build of instance 25879943-2e64-456c-b500-96a8a8735736 aborted: [Errno 2] No such file or directory: nova.exception.Bu
  ildAbortException: Build of instance 25879943-2e64-456c-b500-96a8a8735736 aborted: [Errno 2] No such file or directory

  Environment:

  - OpenStak Zed on Ubuntu Jammy

  # dpkg -l  | grep nova
  ii  nova-common                           3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - common files
  ii  nova-compute                          3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                      3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                  3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute - compute node libvirt support
  ii  python3-nova                          3:26.0.0-0ubuntu1~cloud0                             all          OpenStack Compute Python 3 libraries
  ii  python3-novaclient                    2:18.1.0-0ubuntu1~cloud0                             all          client library for OpenStack Compute API - 3.x

  - The storage type is Ceph
  - The networking is OpenVSwitch

  [0] https://packages.ubuntu.com/jammy-updates/dmidecode
  [1] https://review.opendev.org/c/openstack/os-brick/+/836057

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1994083/+subscriptions
Follow ups

[Bug 1994083] Re: "FileNotFoundError: [Errno 2] No such file or directory" is raised when dmidecode is not installed
From: OpenStack Infra, 2023-01-25
[Bug 1994083] Re: "FileNotFoundError: [Errno 2] No such file or directory" is raised when dmidecode is not installed
From: Felipe Reyes, 2022-10-24