yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1840068] Re: (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len()

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 13 Aug 2019 19:18:35 -0000
Reply-to: Bug 1840068 <1840068@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This filter code goes back to 2012 so we could backport the fix further
(to pike and ocata) but no one is really using the libvirt+lxc code as
far as I can tell, at least not with python3, so we can just backport to
the non-extended-maintenance branches unless someone wants to backport
them upstream to pike and ocata.

** Also affects: nova/stein
   Importance: Undecided
       Status: New

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Summary changed:

- (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len()
+ (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() - python3

** Changed in: nova/queens
   Importance: Undecided => Medium

** Changed in: nova
   Importance: High => Medium

** Changed in: nova/queens
       Status: New => Confirmed

** Changed in: nova/rocky
   Importance: Undecided => Medium

** Changed in: nova/stein
       Status: New => Confirmed

** Changed in: nova/stein
   Importance: Undecided => Medium

** Changed in: nova/rocky
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1840068

Title:
  (lxc) Instance failed to spawn: TypeError: object of type 'filter' has
  no len() - python3

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) queens series:
  Confirmed
Status in OpenStack Compute (nova) rocky series:
  Confirmed
Status in OpenStack Compute (nova) stein series:
  Confirmed

Bug description:
  Seen in the nova-lxc CI job here:

  https://logs.opendev.org/24/676024/2/experimental/nova-
  lxc/f9a892c/controller/logs/screen-n-cpu.txt.gz#_Aug_12_23_31_05_043911

  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [None req-55d6dd1b-96ca-4afe-9a0c-cec902d3bd87 tempest-ServerAddressesTestJSON-1311986476 tempest-ServerAddressesTestJSON-1311986476] [instance: 842a9704-3700-42ef-adb3-b038ca525127] Instance failed to spawn: TypeError: object of type 'filter' has no len()
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] Traceback (most recent call last):
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/compute/manager.py", line 2495, in _build_resources
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     yield resources
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/compute/manager.py", line 2256, in _build_and_run_instance
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     block_device_info=block_device_info)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3231, in spawn
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     destroy_disks_on_failure=True)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5823, in _create_domain_and_network
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     destroy_disks_on_failure)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     self.force_reraise()
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     six.reraise(self.type_, self.value, self.tb)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     raise value
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5789, in _create_domain_and_network
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     block_device_info):
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     return next(self.gen)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5701, in _lxc_disk_handler
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     block_device_info)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5651, in _create_domain_setup_lxc
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     container_dir=container_dir)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/disk/api.py", line 396, in setup_container
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     dev = img.mount()
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/disk/api.py", line 307, in mount
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     if mounter.do_mount():
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/disk/mount/api.py", line 276, in do_mount
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     status = self.get_dev() and self.map_dev() and self.mnt_dev()
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 121, in get_dev
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     return self._get_dev_retry_helper()
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/disk/mount/api.py", line 168, in _get_dev_retry_helper
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     device = self._inner_get_dev()
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     return f(*args, **kwargs)
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 74, in _inner_get_dev
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     device = self._allocate_nbd()
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 64, in _allocate_nbd
  Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     random.shuffle(devices)
  Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]   File "/usr/lib/python3.6/random.py", line 274, in shuffle
  Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]     for i in reversed(range(1, len(x))):
  Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] TypeError: object of type 'filter' has no len()
  Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1840068/+subscriptions
References

[Bug 1840068] [NEW] (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len()
From: Matt Riedemann, 2019-08-13