yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1237282] Re: vm fails to start when compute nodes suddenly powered off

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: shihanzhang <shihanzhang@xxxxxxxxxx>
Date: Sat, 15 Mar 2014 00:27:43 -0000
Reply-to: Bug 1237282 <1237282@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1237282

Title:
  vm fails to start when compute nodes suddenly powered off

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  My environment is built by devstack based on ubuntu 12.04

  Compute nodes suddenly powered off, the  vm on the compute nodes fail to start.
  I get this error after executing:
  root@controller:~# nova show cicd-slave-04
  +-------------------------------------+--------------------
  | Property                            | Value                                                    
  +-------------------------------------+------------------------
  | status                              | SHUTOFF                                                
  | updated                             | 2013-10-09T06:38:06Z                                 
  | OS-EXT-STS:task_state               | None                                                    
  | OS-EXT-SRV-ATTR:host                | compute2    

  root@controller:~# nova start  cicd-slave-04

  I check the nova-compute node then I have this trace in the logs:

  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     pass
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1415, in start_instance
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     self.driver.power_on(instance)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1407, in power_on
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     self._create_domain(domain=dom, instance=instance)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2405, in _create_domain
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     domain.createWithFlags(launch_flags)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     rv = execute(f,*args,**kwargs)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     rv = meth(*args,**kwargs)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2013-10-09 16:26:43.965 32682 TRACE nova.openstack.common.rpc.amqp libvirtError: Unable to add bridge qbrc3a169ff-31 port tapc3a169ff-31: Invalid argument

  I fond the port tapc3a169ff-31 is already added on  bridge qbrc3a169ff-31, I delete the port from bridge qbrc3a169ff-31,then I start 
  the vm successfully

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1237282/+subscriptions