yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1355177] Re: hyperV: compute service goes off when booting an instance fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alessandro Pilotti <1355177@xxxxxxxxxxxxxxxxxx>
Date: Mon, 11 Aug 2014 14:27:23 -0000
Reply-to: Bug 1355177 <1355177@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is the expected behaviour and it's unrelated to the Hyper-V driver.

You can tell Nova not to use Neutron by unsetting "network_api_class" in
nova.conf on your Hyper-V compute node(s) which will revert to the
default (Nova network).

Please note that Nova network is not supported on Hyper-V beyond the
very basic flat networking.


** Changed in: nova
       Status: New => Invalid

** Tags removed: neutron-agent
** Tags added: hyper-v

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1355177

Title:
  hyperV: compute service goes off when booting an instance fails

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  I have a 2012 R2 windows machine where hyperV, openstack nova compute
  service and openstack   neutron hyperV agent is installed.

  I have a devstack running where neutron is not installed.

  When I am trying to boot an instance, it fails with error:
  "ConnectionFailed: Connection to neutron failed:
  HTTPConnectionPool(host='x.x.x.x', port=9696): Max retries exceeded
  with url" and further re-scheduled to a different host.

  But in the mean time compute service becomes dead in the hyperV host.

  nova list: 
  nova-compute     WIN-4EUO2SEHJ92                      nova             enabled    XXX   2014-08-11 09:57:55

  compute.log:
  014-08-11 10:21:53.653 3924 ERROR nova.compute.manager [req-69c46c41-0aa1-47cc-8abe-7306ec5ef57d None] [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] Instance failed to spawn
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] Traceback (most recent call last):
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 2108, in _build_resources
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     yield resources
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1994, in _build_and_run_instance
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     block_device_info=block_device_info)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\driver.py", line 55, in spawn
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     admin_password, network_info, block_device_info)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 87, in wrapper
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     return function(self, *args, **kwds)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 255, in spawn
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     self.destroy(instance)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\openstack\common\excutils.py", line 82, in __exit__
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     six.reraise(self.type_, self.value, self.tb)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 246, in spawn
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     root_vhd_path, eph_vhd_path)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 289, in create_instance
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     for vif in network_info:
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\model.py", line 441, in __iter__
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     return self._sync_wrapper(fn, *args, **kwargs)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\model.py", line 432, in _sync_wrapper
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     self.wait()
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\model.py", line 464, in wait
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     self[:] = self._gt.wait()
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\eventlet\greenthread.py", line 168, in wait
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     return self._exit_event.wait()
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\eventlet\event.py", line 124, in wait
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     current.throw(*self._exc)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\eventlet\greenthread.py", line 207, in main
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     result = function(*args, **kwargs)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1553, in _allocate_network_async
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     dhcp_options=dhcp_options)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\neutronv2\api.py", line 287, in allocate_for_instance
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     net_ids)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\neutronv2\api.py", line 150, in _get_available_networks
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     nets = neutron.list_networks(**search_opts).get('networks', [])
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\v2_0\client.py", line 101, in with_params
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     ret = self.function(instance, *args, **kwargs)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\v2_0\client.py", line 341, in list_networks
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     **_params)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\v2_0\client.py", line 1344, in list
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     for r in self._pagination(collection, path, **params):
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\v2_0\client.py", line 1357, in _pagination
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     res = self.get(path, params=params)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\v2_0\client.py", line 1330, in get
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     headers=headers, params=params)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\v2_0\client.py", line 1307, in retry_request
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     headers=headers, params=params)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\v2_0\client.py", line 1240, in do_request
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     resp, replybody = self.httpclient.do_request(action, method, body=body)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\client.py", line 163, in do_request
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     **kwargs)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\neutronclient\client.py", line 113, in _cs_request
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40]     raise exceptions.ConnectionFailed(reason=e)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='10.1.5.139', port=9696): Max retries exceeded with url: /v2.0/networks.json?tenant_id=d7e8ef6c573045fcb983b77bd2ecf7e4&shared=False (Caused by <class 'socket.error'>: [Errno 10061] WSAECONNREFUSED)
  2014-08-11 10:21:53.653 3924 TRACE nova.compute.manager [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] 
  2014-08-11 10:21:53.657 3924 AUDIT nova.compute.manager [req-69c46c41-0aa1-47cc-8abe-7306ec5ef57d None] [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] Terminating instance
  2014-08-11 10:21:53.658 3924 INFO nova.virt.hyperv.vmops [req-69c46c41-0aa1-47cc-8abe-7306ec5ef57d None] [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] Got request to destroy instance
  2014-08-11 10:21:53.668 3924 DEBUG nova.virt.hyperv.vmops [req-69c46c41-0aa1-47cc-8abe-7306ec5ef57d None] [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] Instance not found destroy C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py:375
  2014-08-11 10:21:53.671 3924 DEBUG nova.compute.claims [req-69c46c41-0aa1-47cc-8abe-7306ec5ef57d None] [instance: 6d430014-faa6-4af5-9701-063f61f4eb40] Aborting claim: [Claim: 2048 MB memory, 20 GB disk, 1 VCPUS] abort C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\claims.py:113

  conductor.log:
  2014-08-11 18:46:32.261 ERROR nova.scheduler.utils [req-b9f1073d-f99c-4120-b87e-214a62148e7f admin demo] [instance: 3c010141-dd65-44c3-a776-2b1cdbbe2522] Erro
  r from last host: WIN-7VBQHQQ93FG (node WIN-7VBQHQQ93FG): [u'Traceback (most recent call last):\n', u'  File "C:\\Program Files (x86)\\Cloudbase Solutions\\Op
  enStack\\Nova\\Python27\\lib\\site-packages\\nova\\compute\\manager.py", line 1913, in do_build_and_run_instance\n    filter_properties)\n', u'  File "C:\\Pro
  gram Files (x86)\\Cloudbase Solutions\\OpenStack\\Nova\\Python27\\lib\\site-packages\\nova\\compute\\manager.py", line 2042, in _build_and_run_instance\n    i
  nstance_uuid=instance.uuid, reason=str(e))\n', u"RescheduledException: Build of instance 3c010141-dd65-44c3-a776-2b1cdbbe2522 was re-scheduled: Connection to
  neutron failed: HTTPConnectionPool(host='10.1.5.139', port=9696): Max retries exceeded with url: /v2.0/networks.json?tenant_id=d7e8ef6c573045fcb983b77bd2ecf7e
  4&shared=False (Caused by <class 'socket.error'>: [Errno 10061] WSAECONNREFUSED)\n"]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1355177/+subscriptions
References

[Bug 1355177] [NEW] hyperV: compute service goes off when booting an instance fails
From: Mayank, 2014-08-11