yahoo-eng-team team mailing list archive

[Ihar Hrachyshka <1734356@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1734356

Title:
  fullstack: Test runner doesn't wait enough time for env to come up

Status in neutron:
  Fix Released

Bug description:
  As shown in http://logs.openstack.org/71/520371/7/check/legacy-
  neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-
  logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-
  native_.txt.gz fullstack doesn't wait enough time for agents to report
  to neutron server. Currently it uses default value of 60 seconds, so
  once all agent processes are started they have 60 seconds to report
  back.

  On busy or weak hardware machines this is not enough. Here comes RCA:
  Logs:
  Problematic neutron-openvswitch-agent: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-openvswitch-agent--2017-11-20--21-51-57-666771.txt.gz
  Neutron-server: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz
  Test runner: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz

  RCA:
  Agent started by test runner at 21:51:57.666: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_51_57_666 
  Test runner stopped polling server for agents at 21:52:51.506: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_52_51_506
  Agent process started at 21:52:23.444, 26 seconds! after async_process claimed process has started
  Server got report of the agent at 21:52:58.939 - 7 seconds after test runner stopped polling: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz#_2017-11-20_21_52_58_939

  
  Summary: On environment with three nodes, running dhcp and ovs agent on each, last agent reported 7 seconds after API polling was finished by test runner.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1734356/+subscriptions