yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1405044] [NEW] [GPFS] nova volume-attach a gpfs volume with an error log in nova-compute

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lan Qi song <lqslan@xxxxxxxxxx>
Date: Tue, 23 Dec 2014 02:56:39 -0000
Reply-to: Bug 1405044 <1405044@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When I attached a gpfs volume to an instance, the volume has been
successfully attached to the instance, but  there were some error logs
in nova-compute log file as below:

2014-12-22 21:52:10.863 13396 ERROR nova.openstack.common.threadgroup [-] Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /gpfs/volume-98520c4e-935d-43d8-9c8d-00fcb54bb335
Exit code: 1
Stdout: u''
Stderr: u'BLKGETSIZE64: Inappropriate ioctl for device\n'
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 125, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     x.wait()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 173, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 121, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 293, in switch
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 490, in run_service
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     service.start()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/service.py", line 181, in start
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1159, in pre_start_hook
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 6037, in update_available_resource
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     nodenames = set(self.driver.get_available_nodes())
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 1237, in get_available_nodes
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     stats = self.get_host_stats(refresh=refresh)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5794, in get_host_stats
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self.host_state.get_host_stats(refresh=refresh)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 473, in host_state
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self._host_state = HostState(self)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6360, in __init__
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self.update_status()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6411, in update_status
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     data['disk_available_least'] = _get_disk_available_least()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6384, in _get_disk_available_least
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     disk_over_committed = (self.driver.
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5744, in _get_disk_over_committed_size_total
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self._get_instance_disk_info(dom.name(), xml))
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5697, in _get_instance_disk_info
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     dk_size = lvm.get_volume_size(path)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/lvm.py", line 55, in decorated_function
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return function(path)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/lvm.py", line 186, in get_volume_size
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     run_as_root=True)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 53, in execute
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return utils.execute(*args, **kwargs)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 163, in execute
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return processutils.execute(*cmd, **kwargs)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 203, in execute
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     cmd=sanitized_cmd)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /gpfs/volume-98520c4e-935d-43d8-9c8d-00fcb54bb335
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Exit code: 1
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Stdout: u''
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Stderr: u'BLKGETSIZE64: Inappropriate ioctl for device\n'
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup


This error occurred when nova tried to get the gpfs volume size using "blockdev --getsize64" command.  
Cinder  configuration "gpfs_mount_point_base"  was set to "/gpfs"
The blockdev filter in /usr/share/nova/rootwrap/compute.filters was set to
"blockdev: RegExpFilter, blockdev, root, blockdev, (--getsize64|--flushbufs), (/dev/.*|/gpfs/.*)"

It seems we couldn't use "blockdev --getsize64" command to calculate the
gpfs volume size.  This error wouldn't affect the volume attachment, but
it affect the nova-compute service,  nova-compute service couldn't
restart successfully with this error.

** Affects: nova
     Importance: Undecided
         Status: New

** Summary changed:

- [GPFS] nova volume-attach a gpfs volume with a error log
+ [GPFS] nova volume-attach a gpfs volume with an error log in nova-compute

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1405044

Title:
  [GPFS] nova volume-attach a gpfs volume with an error log in nova-
  compute

Status in OpenStack Compute (Nova):
  New

Bug description:
  When I attached a gpfs volume to an instance, the volume has been
  successfully attached to the instance, but  there were some error logs
  in nova-compute log file as below:

  2014-12-22 21:52:10.863 13396 ERROR nova.openstack.common.threadgroup [-] Unexpected error while running command.
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /gpfs/volume-98520c4e-935d-43d8-9c8d-00fcb54bb335
  Exit code: 1
  Stdout: u''
  Stderr: u'BLKGETSIZE64: Inappropriate ioctl for device\n'
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 125, in wait
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     x.wait()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 47, in wait
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 173, in wait
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 121, in wait
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 293, in switch
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 490, in run_service
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     service.start()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/service.py", line 181, in start
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1159, in pre_start_hook
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 6037, in update_available_resource
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     nodenames = set(self.driver.get_available_nodes())
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 1237, in get_available_nodes
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     stats = self.get_host_stats(refresh=refresh)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5794, in get_host_stats
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return self.host_state.get_host_stats(refresh=refresh)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 473, in host_state
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self._host_state = HostState(self)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6360, in __init__
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self.update_status()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6411, in update_status
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     data['disk_available_least'] = _get_disk_available_least()
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6384, in _get_disk_available_least
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     disk_over_committed = (self.driver.
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5744, in _get_disk_over_committed_size_total
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     self._get_instance_disk_info(dom.name(), xml))
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5697, in _get_instance_disk_info
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     dk_size = lvm.get_volume_size(path)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/lvm.py", line 55, in decorated_function
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return function(path)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/lvm.py", line 186, in get_volume_size
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     run_as_root=True)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 53, in execute
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return utils.execute(*args, **kwargs)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 163, in execute
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     return processutils.execute(*cmd, **kwargs)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 203, in execute
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup     cmd=sanitized_cmd)
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /gpfs/volume-98520c4e-935d-43d8-9c8d-00fcb54bb335
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Exit code: 1
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Stdout: u''
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Stderr: u'BLKGETSIZE64: Inappropriate ioctl for device\n'
  2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup

  
  This error occurred when nova tried to get the gpfs volume size using "blockdev --getsize64" command.  
  Cinder  configuration "gpfs_mount_point_base"  was set to "/gpfs"
  The blockdev filter in /usr/share/nova/rootwrap/compute.filters was set to
  "blockdev: RegExpFilter, blockdev, root, blockdev, (--getsize64|--flushbufs), (/dev/.*|/gpfs/.*)"

  It seems we couldn't use "blockdev --getsize64" command to calculate
  the gpfs volume size.  This error wouldn't affect the volume
  attachment, but it affect the nova-compute service,  nova-compute
  service couldn't restart successfully with this error.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1405044/+subscriptions
Follow ups

[Bug 1405044] Re: [GPFS] nova volume-attach a gpfs volume with an error log in nova-compute
From: Thierry Carrez, 2015-02-05
[Bug 1405044] Re: [GPFS] nova volume-attach a gpfs volume with an error log in nova-compute
From: Chuck Short, 2015-02-05
[Bug 1405044] Re: [GPFS] nova volume-attach a gpfs volume with an error log in nova-compute
From: Chuck Short, 2015-01-29
[Bug 1405044] [NEW] [GPFS] nova volume-attach a gpfs volume with an error log in nova-compute
From: Lan Qi song, 2014-12-23
References

[Bug 1405044] [NEW] [GPFS] nova volume-attach a gpfs volume with an error log in nova-compute
From: Lan Qi song, 2014-12-23