yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1505313] [NEW] nova kilo and contrail 2.2, instances fail to boot throwing an endpoint_override error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Max <max@xxxxxxxxxxxxxxxx>
Date: Mon, 12 Oct 2015 17:12:34 -0000
Reply-to: Bug 1505313 <1505313@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I am running:

Ubuntu 14.04

Contrail 2.20
root@cmpt02:~# dpkg -l | grep contrail
ii  contrail-lib                         2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail libraries
ii  contrail-nodemgr                     2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail nodemgr implementation
ii  contrail-nova-driver                 2.20+0~1443532552.81~1.c0c6c68        amd64        OpenStack Nova compute-node driver for OpenContrail
ii  contrail-utils                       2.20+0~1441967460.80~1.bb1145b        amd64        OpenContrail tools and utilities
ii  contrail-vrouter-agent               2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail vrouter agent
ii  contrail-vrouter-dkms                2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail VRouter - DKMS version
ii  contrail-vrouter-utils               2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail VRouter - Utilities
ii  python-backports.ssl-match-hostname  3.4.0.2-1contrail1                    all          The ssl.match_hostname() function from Python 3.4
ii  python-bitarray                      0.8.0-2contrail1                      amd64        Python module for efficient boolean array handling
ii  python-contrail                      2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail python-libs
ii  python-contrail-vrouter-api          2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail vrouter agent api
ii  python-geventhttpclient              1.1.0-1contrail1                      amd64        http client library for gevent
ii  python-opencontrail-vrouter-netns    2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail vrouter network namespace package


OpenStack Kilo packages
root@cmpt02:~# dpkg -l | grep nova
ii  contrail-nova-driver                 2.20+0~1443532552.81~1.c0c6c68        amd64        OpenStack Nova compute-node driver for OpenContrail
ii  nova-common                            1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - common files
ii  nova-compute                            1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node base
ii  nova-compute-kvm                   1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt               1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node libvirt support
ii  python-nova                               1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute Python libraries
ii  python-novaclient                    1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API


Nova will not boot an instance; debug log snippet from /var/log/nova/nova-compute.log:

ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
TRACE nova.compute.manager Traceback (most recent call last):
TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1782, in _allocate_network_async
TRACE nova.compute.manager     dhcp_options=dhcp_options)
TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 406, in allocate_for_instance
TRACE nova.compute.manager     neutron = get_client(context)
TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 221, in get_client
TRACE nova.compute.manager     region_name=CONF.neutron.region_name)
TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1200, in __init__
TRACE nova.compute.manager     self.httpclient = client.construct_http_client(**kwargs)
TRACE nova.compute.manager TypeError: construct_http_client() got an unexpected keyword argument 'endpoint_override'


2015-10-12 18:40:06.334 27007 ERROR nova.compute.manager [req-58ce7b2e-bc73-40ee-a368-cfbeaed434ca 42329176f69a4cc1b7d5e6ae805080cd 7812bd244b7f4a8eba3a5cb1213210a5 - - -] [instance: 0713d74f-fe59-4992-894c-
e3378fb1752d] Instance failed to spawn
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager Traceback (most recent call last):
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2461, in _build_resources
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     yield resources
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2333, in _build_and_run_insta
nce                                                      
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     block_device_info=block_device_info)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2382, in spawn
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     write_to_disk=True)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4169, in _get_guest_xml
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     network_info_str = str(network_info)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 480, in __str__
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     return self._sync_wrapper(fn, *args, **kwargs)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 463, in _sync_wrapper
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     self.wait()
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 495, in wait
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     self[:] = self._gt.wait()
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     return self._exit_event.wait()
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     current.throw(*self._exc)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     result = function(*args, **kwargs)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1782, in _allocate_network_as
ync                                                      
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     dhcp_options=dhcp_options)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 406, in allocate_for_in
stance                                                   
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     neutron = get_client(context)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 221, in get_client
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     region_name=CONF.neutron.region_name)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1200, in __init__
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     self.httpclient = client.construct_http_client(**kwargs)
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager TypeError: construct_http_client() got an unexpected keyword argument 'endpoint_override'
2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager 
2015-10-12 18:40:06.336 27007 INFO nova.compute.manager [req-58ce7b2e-bc73-40ee-a368-cfbeaed434ca 42329176f69a4cc1b7d5e6ae805080cd 7812bd244b7f4a8eba3a5cb1213210a5 - - -] [instance: 0713d74f-fe59-4992-894c-e
3378fb1752d] Terminating instance


There is nothing that appears to be wrong in /var/log/neutron-
server.conf or in contrail's logfiles.

Has anyone come across the log "TypeError: construct_http_client() got
an unexpected keyword argument 'endpoint_override'" before?

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "debug log for nova"
   https://bugs.launchpad.net/bugs/1505313/+attachment/4492592/+files/debug_kilo_nova_log.txt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1505313

Title:
  nova kilo and contrail 2.2, instances fail to boot throwing an
  endpoint_override error

Status in OpenStack Compute (nova):
  New

Bug description:
  I am running:

  Ubuntu 14.04

  Contrail 2.20
  root@cmpt02:~# dpkg -l | grep contrail
  ii  contrail-lib                         2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail libraries
  ii  contrail-nodemgr                     2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail nodemgr implementation
  ii  contrail-nova-driver                 2.20+0~1443532552.81~1.c0c6c68        amd64        OpenStack Nova compute-node driver for OpenContrail
  ii  contrail-utils                       2.20+0~1441967460.80~1.bb1145b        amd64        OpenContrail tools and utilities
  ii  contrail-vrouter-agent               2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail vrouter agent
  ii  contrail-vrouter-dkms                2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail VRouter - DKMS version
  ii  contrail-vrouter-utils               2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail VRouter - Utilities
  ii  python-backports.ssl-match-hostname  3.4.0.2-1contrail1                    all          The ssl.match_hostname() function from Python 3.4
  ii  python-bitarray                      0.8.0-2contrail1                      amd64        Python module for efficient boolean array handling
  ii  python-contrail                      2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail python-libs
  ii  python-contrail-vrouter-api          2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail vrouter agent api
  ii  python-geventhttpclient              1.1.0-1contrail1                      amd64        http client library for gevent
  ii  python-opencontrail-vrouter-netns    2.20+0~1443532552.81~1.c0c6c68        amd64        OpenContrail vrouter network namespace package

  
  OpenStack Kilo packages
  root@cmpt02:~# dpkg -l | grep nova
  ii  contrail-nova-driver                 2.20+0~1443532552.81~1.c0c6c68        amd64        OpenStack Nova compute-node driver for OpenContrail
  ii  nova-common                            1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - common files
  ii  nova-compute                            1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                   1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt               1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node libvirt support
  ii  python-nova                               1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute Python libraries
  ii  python-novaclient                    1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

  
  Nova will not boot an instance; debug log snippet from /var/log/nova/nova-compute.log:

  ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
  TRACE nova.compute.manager Traceback (most recent call last):
  TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1782, in _allocate_network_async
  TRACE nova.compute.manager     dhcp_options=dhcp_options)
  TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 406, in allocate_for_instance
  TRACE nova.compute.manager     neutron = get_client(context)
  TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 221, in get_client
  TRACE nova.compute.manager     region_name=CONF.neutron.region_name)
  TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1200, in __init__
  TRACE nova.compute.manager     self.httpclient = client.construct_http_client(**kwargs)
  TRACE nova.compute.manager TypeError: construct_http_client() got an unexpected keyword argument 'endpoint_override'

  
  2015-10-12 18:40:06.334 27007 ERROR nova.compute.manager [req-58ce7b2e-bc73-40ee-a368-cfbeaed434ca 42329176f69a4cc1b7d5e6ae805080cd 7812bd244b7f4a8eba3a5cb1213210a5 - - -] [instance: 0713d74f-fe59-4992-894c-
  e3378fb1752d] Instance failed to spawn
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager Traceback (most recent call last):
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2461, in _build_resources
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     yield resources
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2333, in _build_and_run_insta
  nce                                                      
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     block_device_info=block_device_info)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2382, in spawn
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     write_to_disk=True)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4169, in _get_guest_xml
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     network_info_str = str(network_info)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 480, in __str__
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     return self._sync_wrapper(fn, *args, **kwargs)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 463, in _sync_wrapper
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     self.wait()
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 495, in wait
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     self[:] = self._gt.wait()
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     return self._exit_event.wait()
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     current.throw(*self._exc)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     result = function(*args, **kwargs)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1782, in _allocate_network_as
  ync                                                      
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     dhcp_options=dhcp_options)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 406, in allocate_for_in
  stance                                                   
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     neutron = get_client(context)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 221, in get_client
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     region_name=CONF.neutron.region_name)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1200, in __init__
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager     self.httpclient = client.construct_http_client(**kwargs)
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager TypeError: construct_http_client() got an unexpected keyword argument 'endpoint_override'
  2015-10-12 18:40:06.334 27007 TRACE nova.compute.manager 
  2015-10-12 18:40:06.336 27007 INFO nova.compute.manager [req-58ce7b2e-bc73-40ee-a368-cfbeaed434ca 42329176f69a4cc1b7d5e6ae805080cd 7812bd244b7f4a8eba3a5cb1213210a5 - - -] [instance: 0713d74f-fe59-4992-894c-e
  3378fb1752d] Terminating instance


  There is nothing that appears to be wrong in /var/log/neutron-
  server.conf or in contrail's logfiles.

  Has anyone come across the log "TypeError: construct_http_client() got
  an unexpected keyword argument 'endpoint_override'" before?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1505313/+subscriptions
Follow ups

[Bug 1505313] Re: nova kilo and contrail 2.2, instances fail to boot throwing an endpoint_override error
From: Max, 2015-10-19