yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1269029] Re: vsphere networking works with br100 only

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Tracy Jones <tjones@xxxxxxxxxx>
Date: Mon, 03 Mar 2014 18:26:57 -0000
Reply-to: Bug 1269029 <1269029@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
no info given after over a month

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1269029

Title:
  vsphere networking works with br100 only

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Our configuration is:

  [DEFAULT]
  flat_network_bridge=br-int
  flat_interface=eth0

  [vmware]
  integration_bridge=br100

  
  When changing br100 to br107 in nova.conf and vsphere results in the following error:

  NetworkAdapterNotFound: Network adapter br107 could not be found.

  
  Full trace from compute.log:

  
  ==> /var/log/nova/compute.log <==
  2014-01-14 09:54:36.095 3897 ERROR nova.compute.manager [req-ba2fdf07-dee3-46c8-9dff-1627d61eeaf4 78fa1b4284df4bcab09336354855c3b7 ff01d74511684ca697a87f4d82ab54d6] [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14] Failed to deallocate network for instance.
  2014-01-14 09:54:37.352 3897 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 364610
  2014-01-14 09:54:37.352 3897 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 1024
  2014-01-14 09:54:37.352 3897 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 61
  2014-01-14 09:54:37.429 3897 INFO nova.compute.resource_tracker [-] Compute_service record updated for v-os-nova-01.synygy.net:domain-c17(OS-Cluster)
  2014-01-14 09:54:37.534 3897 ERROR nova.compute.manager [req-ba2fdf07-dee3-46c8-9dff-1627d61eeaf4 78fa1b4284df4bcab09336354855c3b7 ff01d74511684ca697a87f4d82ab54d6] [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14] Error: Network adapter br107 could not be found.
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14] Traceback (most recent call last):
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1037, in _build_instance
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     set_access_ip=set_access_ip)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1420, in _spawn
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1417, in _spawn
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     block_device_info)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 633, in spawn
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     admin_password, network_info, block_device_info)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 268, in spawn
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     vif_infos = _get_vif_infos()
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 259, in _get_vif_infos
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     self._is_neutron)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vif.py", line 154, in get_network_ref
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     create_vlan=create_vlan)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vif.py", line 73, in ensure_vlan_bridge
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     vlan_interface, cluster)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vif.py", line 48, in _get_associated_vswitch_for_interface
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]     raise exception.NetworkAdapterNotFound(adapter=interface)
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14] NetworkAdapterNotFound: Network adapter br107 could not be found.
  2014-01-14 09:54:37.534 3897 TRACE nova.compute.manager [instance: 440effe8-7ee2-4e61-a431-0646bf4b4e14]
  2014-01-14 09:54:38.931 3897 ERROR nova.openstack.common.rpc.amqp [req-ba2fdf07-dee3-46c8-9dff-1627d61eeaf4 78fa1b4284df4bcab09336354855c3b7 ff01d74511684ca697a87f4d82ab54d6] Exception during message handling
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     pass
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 294, in decorated_function
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1623, in run_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     do_run_instance()
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1622, in do_run_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     legacy_bdm_in_spec)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 962, in _run_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     notify("error", msg=unicode(e))  # notify that build failed
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 946, in _run_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     instance, image_meta, legacy_bdm_in_spec)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1085, in _build_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     filter_properties, bdms, legacy_bdm_in_spec)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1129, in _reschedule_or_error
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     self._log_original_error(exc_info, instance_uuid)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1124, in _reschedule_or_error
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     bdms, requested_networks)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1674, in _shutdown_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     self._try_deallocate_network(context, instance, requested_networks)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1634, in _try_deallocate_network
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     self._set_instance_error_state(context, instance['uuid'])
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1629, in _try_deallocate_network
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     self._deallocate_network(context, instance, requested_networks)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1473, in _deallocate_network
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     context, instance, requested_networks=requested_networks)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 94, in wrapped
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return func(self, context, *args, **kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 318, in deallocate_for_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     self.network_rpcapi.deallocate_for_instance(context, **args)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/rpcapi.py", line 193, in deallocate_for_instance
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     host=host, requested_networks=requested_networks)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 85, in call
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return self._invoke(self.proxy.call, ctxt, method, **kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 63, in _invoke
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return cast_or_call(ctxt, msg, **self.kwargs)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/proxy.py", line 126, in call
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     result = rpc.call(context, real_topic, msg, timeout)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 139, in call
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     return _get_impl().call(CONF, context, topic, msg, timeout)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 783, in call
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 574, in call
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     rv = list(rv)
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 539, in __iter__
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp     raise result
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp RemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.107.1.4 fa:16:3e:31:1a:9d
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp Exit code: 1
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp Stdout: ''
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp Stderr: 'cannot setup interface: No such device\n'
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data\n    **args)\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 547, in deallocate_for_instance\n    self.deallocate_fixed_ip(context, fixed_ip[\'address\'], host=host)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1507, in deallocate_fixed_ip\n    teardown)\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 940, in deallocate_fixed_ip\n    self.driver.release_dhcp(dev, address, vif[\'address\'])\n', u'  File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 976, in release_dhcp\n    utils.execute(\'dhcp_release\', dev, address, mac_address, run_as_root=True)\n', u'  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute\n    return processutils.execute(*cmd, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute\n    cmd=\' \'.join(cmd))\n', u"ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.107.1.4 fa:16:3e:31:1a:9d\nExit code: 1\nStdout: ''\nStderr: 'cannot setup interface: No such device\\n'\n"].
  2014-01-14 09:54:38.931 3897 TRACE nova.openstack.common.rpc.amqp

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1269029/+subscriptions
References

[Bug 1269029] [NEW] vsphere networking works with br100 only
From: cristi1979, 2014-01-14