yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1536942] Re: lb: "RTNETLINK answers: Transport endpoint is not connected" when setting vxlan device

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1536942@xxxxxxxxxxxxxxxxxx>
Date: Sun, 02 Apr 2017 04:19:46 -0000
Reply-to: Bug 1536942 <1536942@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for neutron because there has been no activity for 60 days.]

** Changed in: neutron
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1536942

Title:
  lb: "RTNETLINK answers: Transport endpoint is not connected" when
  setting vxlan device

Status in neutron:
  Expired

Bug description:
  Sometimes the following error appears in the linuxbridge q-agt log:
  [1]

  Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1066', 'up']
  Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected

  
  This happens after a vxlan device has been created and should be set up. Interesting is, that it's always the first vxlan device that has been created (Had a look at 3 different logs). All vxlan devices that have been created after that could be set up fine.

  After that a agent resync is triggered. The ensure_vxlan method
  detects, that the interface is already there and does not create it
  again. But it does not check if it is up as well - so it may still be
  in the down state (don't know, can't locally reproduce it) and never
  put up. This might be an issue.

  
  Along logstash, the issue seemed to be occured the first time at January 13 [2].
  The tests do not seem to fail due to this error, there are about as many succeeded runs as failed runs when this error shows up in the log. Probably that's because it's just a single node testing, and no package will ever leave the devstack node....

  
  A workaround could be to modify ensure_vxlan to add a check if it is up (and not just checking if it exists).

  More details from [1]:
  2016-01-20 16:44:35.226 DEBUG neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Creating vxlan interface vxlan-1066 for VNI 1066 ensure_vxlan /opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:265
  2016-01-20 16:44:35.228 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Running command (rootwrap daemon): ['ip', 'link', 'add', 'vxlan-1066', 'type', 'vxlan', 'id', '1066', 'group', '224.0.0.1', 'dev', 'eth0'] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:100
  2016-01-20 16:44:35.316 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Exit code: 0 execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:142
  2016-01-20 16:44:35.316 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1066', 'up'] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:100
  2016-01-20 16:44:35.329 ERROR neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected

  2016-01-20 16:44:35.333 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Error in agent loop. Devices info: {'current': set(['tapce39edcc-d9']), 'removed': set([]), 'added': set(['tapce39edcc-d9']), 'updated': set([])}
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1161, in daemon_loop
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     sync = self.process_network_devices(device_info)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 967, in process_network_devices
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     resync_a = self.treat_devices_added_updated(devices_added_updated)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1001, in treat_devices_added_updated
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     device_details['port_id'], device_details['device_owner'])
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 473, in add_interface
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     tap_device_name, device_owner)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 435, in add_tap_interface
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     segmentation_id)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 391, in ensure_physical_in_bridge
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     return self.ensure_vxlan_bridge(network_id, segmentation_id)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 200, in ensure_vxlan_bridge
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     interface = self.ensure_vxlan(segmentation_id)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 288, in ensure_vxlan
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     int_vxlan.link.set_up()
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 461, in set_up
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     return self._as_root([], ('set', self.name, 'up'))
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 321, in _as_root
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     use_root_namespace=use_root_namespace)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 94, in _as_root
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     log_fail_as_error=self.log_fail_as_error)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 103, in _execute
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     log_fail_as_error=log_fail_as_error)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 140, in execute
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent     raise RuntimeError(msg)
  2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected


  [1] http://logs.openstack.org/69/143169/69/check/gate-tempest-dsvm-neutron-linuxbridge/cd66b94/logs/screen-q-agt.txt.gz
  [2] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Transport%20endpoint%20is%20not%20connected%5C%22%20AND%20build_name%3A%5C%22gate-tempest-dsvm-neutron-linuxbridge%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1536942/+subscriptions
References

[Bug 1536942] [NEW] lb: "RTNETLINK answers: Transport endpoint is not connected" when setting vxlan device
From: Andreas Scheuring, 2016-01-22