yahoo-eng-team team mailing list archive

[OpenStack Infra <1734357@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/522872
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=89e0ed9e3d1630197545280ef6c11828d0f2b1dc
Submitter: Zuul
Branch:    master

commit 89e0ed9e3d1630197545280ef6c11828d0f2b1dc
Author: Jakub Libosvar <libosvar@xxxxxxxxxx>
Date:   Fri Nov 24 16:53:11 2017 +0000

    fullstack: Wait at least 3 minute for agents to report
    
    Test runner polls neutron-server for agents for default value of
    wait_until_true() which is 60 seconds. It has been proven to be
    insufficient time on busy environments. This patch bumps the timeout to
    3 minutes and also highers polling interval from 1 seconds to 10
    seconds, to offload neutron-server.
    
    Change-Id: I108176ee5668c2bc456dda1810aa509212e7cb38
    Closes-bug: #1734357


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1734357

Title:
  fullstack: Test runner doesn't wait enough time for env to come up

Status in neutron:
  Fix Released

Bug description:
  As shown in http://logs.openstack.org/71/520371/7/check/legacy-
  neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-
  logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-
  native_.txt.gz fullstack doesn't wait enough time for agents to report
  to neutron server. Currently it uses default value of 60 seconds, so
  once all agent processes are started they have 60 seconds to report
  back.

  On busy or weak hardware machines this is not enough. Here comes RCA:
  Logs:
  Problematic neutron-openvswitch-agent: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-openvswitch-agent--2017-11-20--21-51-57-666771.txt.gz
  Neutron-server: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz
  Test runner: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz

  RCA:
  Agent started by test runner at 21:51:57.666: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_51_57_666 
  Test runner stopped polling server for agents at 21:52:51.506: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_52_51_506
  Agent process started at 21:52:23.444, 26 seconds! after async_process claimed process has started
  Server got report of the agent at 21:52:58.939 - 7 seconds after test runner stopped polling: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz#_2017-11-20_21_52_58_939

  
  Summary: On environment with three nodes, running dhcp and ovs agent on each, last agent reported 7 seconds after API polling was finished by test runner.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1734357/+subscriptions