yahoo-eng-team team mailing list archive

[Hemanth Nakkina <1850779@xxxxxxxxxxxxxxxxxx>]

** Also affects: ubuntu
   Importance: Undecided
       Status: New

** Package changed: ubuntu => neutron (Ubuntu)

** Also affects: neutron (Ubuntu Hirsute)
   Importance: Undecided
       Status: New

** Also affects: neutron (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Also affects: neutron (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Also affects: neutron (Ubuntu Groovy)
   Importance: Undecided
       Status: New

** Changed in: neutron (Ubuntu Hirsute)
       Status: New => Fix Released

** Changed in: neutron (Ubuntu Groovy)
       Status: New => Fix Released

** Changed in: neutron (Ubuntu Focal)
       Status: New => Fix Released

** Description changed:

  If the  DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart.
  And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent.
  
  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108
+ 
+ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ SRU:
+ 
+ [Impact]
+ Longer l3-agent initialization time during restarts due to creation of snat namespace and setting corresponding sysctl twice.
+ With this fix, the initialization phase is triggered only once.
+ 
+ [Test Case]
+ * deploy Openstack on bionic queens (with neutron dvr l3 ha settings and debug mode on for neutron ) and create a router
+   (If stsstack-bundles are used, here are the commands
+    ./generate-bundle.sh -s bionic -n bionicqueens --dvr-snat-l3ha --create-model --run
+    ./configure
+    # Configure creates a router with external gateway attached
+   )
+ * Restart neutron-l3-agent on one of the node
+   systemctl restart neutron-l3-agent.service
+ 
+ * Check /var/log/neutron/neutron-l3-agent.log and wait for the logs to be settled with all initialization steps
+   During initialization steps, following sysctl's are configured [1] [2]. 
+   Verify if the debug logs show sysctl execution statements are displayed twice after restart for snat namespace.
+   (If the fix is applied they should be displayed only once)
+    
+   grep -inr snat-<router-id> /var/log/neutron/neutron-l3-agent.log | grep sysctl
+ 
+   Example log:
+   2718:2021-04-14 05:17:20.114 10868 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'snat-f64dded1-ef73-47b4-bcee-bb25840e9a02', 'sysctl', '-w', 'net.ipv4.ip_forward=1'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:87
+ 
+ [Where problems could occur]
+ This patch reduces the l3 agent initialization time during restart and is not expected to introduce new regressions.
+ 
+ [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
+ [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

** Patch added: "Debdiff for bionic"
   https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1850779/+attachment/5487649/+files/lp1850779_bionic.debdiff

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1850779

Title:
  [L3] snat-ns will be initialized twice for DVR+HA routers during agent
  restart

Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  New
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released
Status in neutron source package in Hirsute:
  Fix Released

Bug description:
  If the  DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart.
  And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent.

  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  SRU:

  [Impact]
  Longer l3-agent initialization time during restarts due to creation of snat namespace and setting corresponding sysctl twice.
  With this fix, the initialization phase is triggered only once.

  [Test Case]
  * deploy Openstack on bionic queens (with neutron dvr l3 ha settings and debug mode on for neutron ) and create a router
    (If stsstack-bundles are used, here are the commands
     ./generate-bundle.sh -s bionic -n bionicqueens --dvr-snat-l3ha --create-model --run
     ./configure
     # Configure creates a router with external gateway attached
    )
  * Restart neutron-l3-agent on one of the node
    systemctl restart neutron-l3-agent.service

  * Check /var/log/neutron/neutron-l3-agent.log and wait for the logs to be settled with all initialization steps
    During initialization steps, following sysctl's are configured [1] [2]. 
    Verify if the debug logs show sysctl execution statements are displayed twice after restart for snat namespace.
    (If the fix is applied they should be displayed only once)
     
    grep -inr snat-<router-id> /var/log/neutron/neutron-l3-agent.log | grep sysctl

    Example log:
    2718:2021-04-14 05:17:20.114 10868 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'snat-f64dded1-ef73-47b4-bcee-bb25840e9a02', 'sysctl', '-w', 'net.ipv4.ip_forward=1'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:87

  [Where problems could occur]
  This patch reduces the l3 agent initialization time during restart and is not expected to introduce new regressions.

  [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1850779/+subscriptions