yahoo-eng-team team mailing list archive

[OpenStack Infra <1748658@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/542858
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fb27465c6fb734d7106130765f918ed91bc329ad
Submitter: Zuul
Branch:    master

commit fb27465c6fb734d7106130765f918ed91bc329ad
Author: Brent Eagles <beagles@xxxxxxxxxx>
Date:   Fri Feb 9 11:11:00 2018 -0330

    Mount netns as shared to persist namespaces
    
    Dataplane is breaking when containers are killed. Keeping the namespaces
    around allows the network objects (ports, bridges) etc. to remain
    intact without the containers running.
    
    Closes-Bug: #1748658
    Change-Id: I092500e9ec0820347ba0f865f3c24f828980af3a


** Changed in: tripleo
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1748658

Title:
  Restarting Neutron containers which make use of network namespaces
  doesn't work

Status in neutron:
  New
Status in tripleo:
  Fix Released

Bug description:
  When DHCP, L3, Metadata or OVN-Metadata containers are restarted they can't
  set the previous namespaces:

  
  [heat-admin@overcloud-novacompute-0 neutron]$ sudo docker restart 8559f5a7fa45
  8559f5a7fa45


  [heat-admin@overcloud-novacompute-0 neutron]$ tail -f /var/log/containers/neutron/networking-ovn-metadata-agent.log 
  2018-02-09 08:34:41.059 5 CRITICAL neutron [-] Unhandled error: ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2018-02-09 08:34:41.059 5 ERROR neutron Traceback (most recent call last):
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/bin/networking-ovn-metadata-agent", line 10, in <module>
  2018-02-09 08:34:41.059 5 ERROR neutron     sys.exit(main())
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/networking_ovn/cmd/eventlet/agents/metadata.py", line 17, in main
  2018-02-09 08:34:41.059 5 ERROR neutron     metadata_agent.main()
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata_agent.py", line 38, in main
  2018-02-09 08:34:41.059 5 ERROR neutron     agt.start()
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 147, in start
  2018-02-09 08:34:41.059 5 ERROR neutron     self.sync()
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 56, in wrapped
  2018-02-09 08:34:41.059 5 ERROR neutron     return f(*args, **kwargs)
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 169, in sync
  2018-02-09 08:34:41.059 5 ERROR neutron     metadata_namespaces = self.ensure_all_networks_provisioned()
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 350, in ensure_all_networks_provisioned
  2018-02-09 08:34:41.059 5 ERROR neutron     netns = self.provision_datapath(datapath)
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/agent.py", line 294, in provision_datapath
  2018-02-09 08:34:41.059 5 ERROR neutron     veth_name[0], veth_name[1], namespace)
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 182, in add_veth
  2018-02-09 08:34:41.059 5 ERROR neutron     self._as_root([], 'link', tuple(args))
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
  2018-02-09 08:34:41.059 5 ERROR neutron     namespace=namespace)
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 102, in _execute
  2018-02-09 08:34:41.059 5 ERROR neutron     log_fail_as_error=self.log_fail_as_error)
  2018-02-09 08:34:41.059 5 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute
  2018-02-09 08:34:41.059 5 ERROR neutron     raise ProcessExecutionError(msg, returncode=returncode)
  2018-02-09 08:34:41.059 5 ERROR neutron ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2018-02-09 08:34:41.059 5 ERROR neutron 
  2018-02-09 08:34:41.059 5 ERROR neutron 
  2018-02-09 08:34:41.177 21 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
  2018-02-09 08:34:41.178 21 INFO eventlet.wsgi.server [-] (21) wsgi exited, is_accepting=True

  
  An easy way to reproduce the bug:

  [heat-admin@overcloud-novacompute-0 ~]$ sudo docker exec -u root -it
  5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d
  /bin/bash

  [root@overcloud-novacompute-0 /]# ip netns a my_netns
  [root@overcloud-novacompute-0 /]# exit

  [heat-admin@overcloud-novacompute-0 ~]$ sudo ip netns
  [heat-admin@overcloud-novacompute-0 ~]$ sudo docker restart 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d
  5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d

  [heat-admin@overcloud-novacompute-0 ~]$ sudo docker exec -u root -it 5c5f254a9321bd74b5911f46acb9513574c2cd9a3c59805a85cffd960bcc864d /bin/bash
  [root@overcloud-novacompute-0 /]# ip netns
  RTNETLINK answers: Invalid argument
  RTNETLINK answers: Invalid argument
  my_netns

  [root@overcloud-novacompute-0 /]# ip netns e my_netns ip a
  RTNETLINK answers: Invalid argument
  setting the network namespace "my_netns" failed: Invalid argument

  Deleting everything under /run/netns/* from kolla_start but this would involve
  a full sync of the agents which is not desirable:

  [root@overcloud-novacompute-0 /]# rm /run/netns/my_netns 
  rm: remove regular empty file '/run/netns/my_netns'? y
  [root@overcloud-novacompute-0 /]# ip netns                
  [root@overcloud-novacompute-0 /]# ip netns a my_netns
  [root@overcloud-novacompute-0 /]#

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1748658/+subscriptions