yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1717533] [NEW] No rootwrap filter for chmod in libvirt/utils

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Evgeny Antyshev <eantyshev@xxxxxxxxx>
Date: Fri, 15 Sep 2017 15:05:52 -0000
Reply-to: Bug 1717533 <1717533@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

After https://review.openstack.org/459166 was applied, Virtuozzo-specific code became broken,
which was noticed when we started running Tempest tests
for ephemeral disk.

n-cpu.service log:
Sep 15 10:15:09.633992 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [None req-ff184083-1ba2-44ec-a961-111adafb4cbe service nova] [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Instance failed to spawn: ProcessExecutionError: Unexpected error while running command.
Sep 15 10:15:09.634505 localhost.localdomain nova-compute[67509]: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0
Sep 15 10:15:09.634683 localhost.localdomain nova-compute[67509]: Exit code: 99
Sep 15 10:15:09.634852 localhost.localdomain nova-compute[67509]: Stdout: u''
Sep 15 10:15:09.635244 localhost.localdomain nova-compute[67509]: Stderr: u'/usr/bin/nova-rootwrap: Unauthorized command: chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0 (no filter matched)\n'
Sep 15 10:15:09.635435 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Traceback (most recent call last):
Sep 15 10:15:09.635601 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2162, in _build_resources
Sep 15 10:15:09.635772 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     yield resources
Sep 15 10:15:09.636252 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1977, in _build_and_run_instance
Sep 15 10:15:09.636523 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     block_device_info=block_device_info)
Sep 15 10:15:09.636965 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2797, in spawn
Sep 15 10:15:09.637339 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     block_device_info=block_device_info)
Sep 15 10:15:09.637582 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3273, in _create_image
Sep 15 10:15:09.637833 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     specified_fs=specified_fs)
Sep 15 10:15:09.638079 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 242, in cache
Sep 15 10:15:09.638483 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     *args, **kwargs)
Sep 15 10:15:09.638733 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 1087, in create_image
Sep 15 10:15:09.638973 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     prepare_template(target=self.path, *args, **kwargs)
Sep 15 10:15:09.639245 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
Sep 15 10:15:09.639494 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     return f(*args, **kwargs)
Sep 15 10:15:09.639732 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 238, in fetch_func_sync
Sep 15 10:15:09.640069 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     fetch_func(target=target, *args, **kwargs)
Sep 15 10:15:09.640367 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3017, in _create_ephemeral
Sep 15 10:15:09.640615 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     specified_fs)
Sep 15 10:15:09.640852 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/utils.py", line 119, in create_ploop_image
Sep 15 10:15:09.641093 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     run_as_root=True, check_exit_code=True)
Sep 15 10:15:09.641367 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/utils.py", line 223, in execute
Sep 15 10:15:09.641616 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
Sep 15 10:15:09.641862 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/utils.py", line 106, in execute
Sep 15 10:15:09.642104 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     return processutils.execute(*cmd, **kwargs)
Sep 15 10:15:09.642382 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 419, in execute
Sep 15 10:15:09.642726 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     cmd=sanitized_cmd)
Sep 15 10:15:09.642965 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] ProcessExecutionError: Unexpected error while running command.
Sep 15 10:15:09.643238 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0
Sep 15 10:15:09.643486 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Exit code: 99
Sep 15 10:15:09.643724 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Stdout: u''
Sep 15 10:15:09.643970 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Stderr: u'/usr/bin/nova-rootwrap: Unauthorized command: chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0 (no filter matched)\n'
Sep 15 10:15:09.644248 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]

** Affects: nova
     Importance: Undecided
     Assignee: Evgeny Antyshev (eantyshev)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1717533

Title:
  No rootwrap filter for chmod in libvirt/utils

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  After https://review.openstack.org/459166 was applied, Virtuozzo-specific code became broken,
  which was noticed when we started running Tempest tests
  for ephemeral disk.

  n-cpu.service log:
  Sep 15 10:15:09.633992 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [None req-ff184083-1ba2-44ec-a961-111adafb4cbe service nova] [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Instance failed to spawn: ProcessExecutionError: Unexpected error while running command.
  Sep 15 10:15:09.634505 localhost.localdomain nova-compute[67509]: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0
  Sep 15 10:15:09.634683 localhost.localdomain nova-compute[67509]: Exit code: 99
  Sep 15 10:15:09.634852 localhost.localdomain nova-compute[67509]: Stdout: u''
  Sep 15 10:15:09.635244 localhost.localdomain nova-compute[67509]: Stderr: u'/usr/bin/nova-rootwrap: Unauthorized command: chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0 (no filter matched)\n'
  Sep 15 10:15:09.635435 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Traceback (most recent call last):
  Sep 15 10:15:09.635601 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2162, in _build_resources
  Sep 15 10:15:09.635772 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     yield resources
  Sep 15 10:15:09.636252 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1977, in _build_and_run_instance
  Sep 15 10:15:09.636523 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     block_device_info=block_device_info)
  Sep 15 10:15:09.636965 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2797, in spawn
  Sep 15 10:15:09.637339 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     block_device_info=block_device_info)
  Sep 15 10:15:09.637582 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3273, in _create_image
  Sep 15 10:15:09.637833 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     specified_fs=specified_fs)
  Sep 15 10:15:09.638079 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 242, in cache
  Sep 15 10:15:09.638483 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     *args, **kwargs)
  Sep 15 10:15:09.638733 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 1087, in create_image
  Sep 15 10:15:09.638973 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     prepare_template(target=self.path, *args, **kwargs)
  Sep 15 10:15:09.639245 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  Sep 15 10:15:09.639494 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     return f(*args, **kwargs)
  Sep 15 10:15:09.639732 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 238, in fetch_func_sync
  Sep 15 10:15:09.640069 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     fetch_func(target=target, *args, **kwargs)
  Sep 15 10:15:09.640367 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3017, in _create_ephemeral
  Sep 15 10:15:09.640615 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     specified_fs)
  Sep 15 10:15:09.640852 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/virt/libvirt/utils.py", line 119, in create_ploop_image
  Sep 15 10:15:09.641093 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     run_as_root=True, check_exit_code=True)
  Sep 15 10:15:09.641367 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/utils.py", line 223, in execute
  Sep 15 10:15:09.641616 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
  Sep 15 10:15:09.641862 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/opt/stack/new/nova/nova/utils.py", line 106, in execute
  Sep 15 10:15:09.642104 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     return processutils.execute(*cmd, **kwargs)
  Sep 15 10:15:09.642382 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 419, in execute
  Sep 15 10:15:09.642726 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]     cmd=sanitized_cmd)
  Sep 15 10:15:09.642965 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] ProcessExecutionError: Unexpected error while running command.
  Sep 15 10:15:09.643238 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0
  Sep 15 10:15:09.643486 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Exit code: 99
  Sep 15 10:15:09.643724 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Stdout: u''
  Sep 15 10:15:09.643970 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc] Stderr: u'/usr/bin/nova-rootwrap: Unauthorized command: chmod -R a+r /opt/stack/data/nova/instances/c9d08a85-4a46-4b34-b919-8c2cb283ecfc/disk.eph0 (no filter matched)\n'
  Sep 15 10:15:09.644248 localhost.localdomain nova-compute[67509]: ERROR nova.compute.manager [instance: c9d08a85-4a46-4b34-b919-8c2cb283ecfc]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1717533/+subscriptions
Follow ups

[Bug 1717533] Re: No rootwrap filter for chmod in libvirt/utils
From: OpenStack Infra, 2017-10-04