yahoo-eng-team team mailing list archive

[Jakub Libosvar <libosvar@xxxxxxxxxx>]

Public bug reported:

As shown in http://logs.openstack.org/71/520371/7/check/legacy-neutron-
dsvm-fullstack/ad585a2/logs/dsvm-fullstack-
logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-
native_.txt.gz fullstack doesn't wait enough time for agents to report
to neutron server. Currently it uses default value of 60 seconds, so
once all agent processes are started they have 60 seconds to report
back.

On busy or weak hardware machines this is not enough. Here comes RCA:
Logs:
Problematic neutron-openvswitch-agent: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-openvswitch-agent--2017-11-20--21-51-57-666771.txt.gz
Neutron-server: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz
Test runner: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz

RCA:
Agent started by test runner at 21:51:57.666: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_51_57_666 
Test runner stopped polling server for agents at 21:52:51.506: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_52_51_506
Agent process started at 21:52:23.444, 26 seconds! after async_process claimed process has started
Server got report of the agent at 21:52:58.939 - 7 seconds after test runner stopped polling: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz#_2017-11-20_21_52_58_939


Summary: On environment with three nodes, running dhcp and ovs agent on each, last agent reported 7 seconds after API polling was finished by test runner.

** Affects: neutron
     Importance: Undecided
     Assignee: Jakub Libosvar (libosvar)
         Status: In Progress


** Tags: fullstack gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1734357

Title:
  fullstack: Test runner doesn't wait enough time for env to come up

Status in neutron:
  In Progress

Bug description:
  As shown in http://logs.openstack.org/71/520371/7/check/legacy-
  neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-
  logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-
  native_.txt.gz fullstack doesn't wait enough time for agents to report
  to neutron server. Currently it uses default value of 60 seconds, so
  once all agent processes are started they have 60 seconds to report
  back.

  On busy or weak hardware machines this is not enough. Here comes RCA:
  Logs:
  Problematic neutron-openvswitch-agent: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-openvswitch-agent--2017-11-20--21-51-57-666771.txt.gz
  Neutron-server: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz
  Test runner: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz

  RCA:
  Agent started by test runner at 21:51:57.666: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_51_57_666 
  Test runner stopped polling server for agents at 21:52:51.506: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_52_51_506
  Agent process started at 21:52:23.444, 26 seconds! after async_process claimed process has started
  Server got report of the agent at 21:52:58.939 - 7 seconds after test runner stopped polling: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz#_2017-11-20_21_52_58_939

  
  Summary: On environment with three nodes, running dhcp and ovs agent on each, last agent reported 7 seconds after API polling was finished by test runner.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1734357/+subscriptions