yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #71694
[Bug 1748658] Re: Restarting Neutron containers which make use of network namespaces doesn't work
I believe the fix belongs to packaging / containers / tripleo and not
neutron itself.
** Changed in: neutron
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1748658
Title:
Restarting Neutron containers which make use of network namespaces
doesn't work
Status in neutron:
Invalid
Status in tripleo:
Fix Released
Bug description:
When DHCP, L3, Metadata or OVN-Metadata containers are restarted they can't
set the previous namespaces:
[heat-admin@overcloud-novacompute-0 neutron]$ sudo docker restart 8559f5a7fa45
8559f5a7fa45
[heat-admin@overcloud-novacompute-0 neutron]$ tail -f /var/log/containers/neutron/networking-ovn-metadata-agent.log
2018-02-09 08:34:41.059 5 CRITICAL neutron [-] Unhandled error: ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2018-02-09 08:34:41.059 5 ERROR neutron Traceback (most recent call last):
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/bin/networking-ovn-metadata-agent", line 10, in <module>
2018-02-09 08:34:41.059 5 ERROR neutron sys.exit(main())
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/cmd/eventlet/agents/metadata.py", line 17, in main
2018-02-09 08:34:41.059 5 ERROR neutron metadata_agent.main()
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata_agent.py", line 38, in main
2018-02-09 08:34:41.059 5 ERROR neutron agt.start()
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 147, in start
2018-02-09 08:34:41.059 5 ERROR neutron self.sync()
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 56, in wrapped
2018-02-09 08:34:41.059 5 ERROR neutron return f(*args, **kwargs)
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 169, in sync
2018-02-09 08:34:41.059 5 ERROR neutron metadata_namespaces = self.ensure_all_networks_provisioned()
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 350, in ensure_all_networks_provisioned
2018-02-09 08:34:41.059 5 ERROR neutron netns = self.provision_datapath(datapath)
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 294, in provision_datapath
2018-02-09 08:34:41.059 5 ERROR neutron veth_name[0], veth_name[1], namespace)
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 182, in add_veth
2018-02-09 08:34:41.059 5 ERROR neutron self._as_root([], 'link', tuple(args))
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
2018-02-09 08:34:41.059 5 ERROR neutron namespace=namespace)
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 102, in _execute
2018-02-09 08:34:41.059 5 ERROR neutron log_fail_as_error=self.log_fail_as_error)
2018-02-09 08:34:41.059 5 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute
2018-02-09 08:34:41.059 5 ERROR neutron raise ProcessExecutionError(msg, returncode=returncode)
2018-02-09 08:34:41.059 5 ERROR neutron ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2018-02-09 08:34:41.059 5 ERROR neutron
2018-02-09 08:34:41.059 5 ERROR neutron
2018-02-09 08:34:41.177 21 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
2018-02-09 08:34:41.178 21 INFO eventlet.wsgi.server [-] (21) wsgi exited, is_accepting=True
An easy way to reproduce the bug:
[heat-admin@overcloud-novacompute-0 ~]$ sudo docker exec -u root -it
5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d
/bin/bash
[root@overcloud-novacompute-0 /]# ip netns a my_netns
[root@overcloud-novacompute-0 /]# exit
[heat-admin@overcloud-novacompute-0 ~]$ sudo ip netns
[heat-admin@overcloud-novacompute-0 ~]$ sudo docker restart 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d
5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d
[heat-admin@overcloud-novacompute-0 ~]$ sudo docker exec -u root -it 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d /bin/bash
[root@overcloud-novacompute-0 /]# ip netns
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
my_netns
[root@overcloud-novacompute-0 /]# ip netns e my_netns ip a
RTNETLINK answers: Invalid argument
setting the network namespace "my_netns" failed: Invalid argument
Deleting everything under /run/netns/* from kolla_start but this would involve
a full sync of the agents which is not desirable:
[root@overcloud-novacompute-0 /]# rm /run/netns/my_netns
rm: remove regular empty file '/run/netns/my_netns'? y
[root@overcloud-novacompute-0 /]# ip netns
[root@overcloud-novacompute-0 /]# ip netns a my_netns
[root@overcloud-novacompute-0 /]#
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1748658/+subscriptions
References