yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => havana-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1177104

Title:
  PowerVM drver uses stale ssh connection if hypervisor is rebooted

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  While trying to run this (on grizzly 2013.1):

  tempest/tempest/tests/compute/images/test_images_oneserver.py:ImagesOneServerTestJSON.test_create_second_image_when_first_image_is_being_saved

  We get this:

  2013-05-06 14:53:51.344 5682 DEBUG nova.utils [-] Running cmd (SSH): lshwres -r mem --level sys -F configurable_sys_mem,curr_avail_sys_mem,sys_firmware_mem ssh_execute /usr/lib/python2.6/site-packages/nova/utils.py:288
  2013-05-06 14:53:51.344 5682 ERROR nova.manager [-] Error during ComputeManager.update_available_resource: SSH session not active
  2013-05-06 14:53:51.344 5682 TRACE nova.manager Traceback (most recent call last):
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/manager.py", line 244, in periodic_tasks
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     task(self, context)
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3898, in update_available_resource
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     nodenames = set(self.driver.get_available_nodes())
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 849, in get_available_nodes
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     stats = self.get_host_stats(refresh=True)
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/virt/powervm/driver.py", line 97, in get_host_stats
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     return self._powervm.get_host_stats(refresh=refresh)
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/virt/powervm/operator.py", line 170, in get_host_stats
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     self._update_host_stats()
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/virt/powervm/operator.py", line 174, in _update_host_stats
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     memory_info = self._operator.get_memory_info()
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/virt/powervm/operator.py", line 852, in get_memory_info
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     output = self.run_vios_command(cmd)
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/virt/powervm/operator.py", line 934, in run_vios_command
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     check_exit_code=check_exit_code)
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 297, in ssh_execute
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     stdin_stream, stdout_stream, stderr_stream = ssh.exec_command(cmd)
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/paramiko/client.py", line 364, in exec_command
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     chan = self._transport.open_session()
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/paramiko/transport.py", line 661, in open_session
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     return self.open_channel('session')
  2013-05-06 14:53:51.344 5682 TRACE nova.manager   File "/usr/lib/python2.6/site-packages/paramiko/transport.py", line 730, in open_channel
  2013-05-06 14:53:51.344 5682 TRACE nova.manager     raise SSHException('SSH session not active')
  2013-05-06 14:53:51.344 5682 TRACE nova.manager SSHException: SSH session not active

  This is because the test was ran earlier and the hypervisor was locked
  up.  The tester rebooted the hypervisor but trying to run the test
  again results in the SSHException.  This is due to the powervm
  operator.py code uses a cached paramiko ssh client connection:

  https://github.com/openstack/nova/blob/master/nova/virt/powervm/operator.py#L467

  The code should check the connection before assuming it's OK and then
  failing because it's bad.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1177104/+subscriptions