yahoo-eng-team team mailing list archive

[Ihar Hrachyshka <1603268@xxxxxxxxxxxxxxxxxx>]

Seems like grafana dashboard shows that the issue went away, and the job
is now on par with other multinode jobs in terms of rate of failures in
check queue. Closing the bug.

** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1603268

Title:
  Unstable grenade multinode job

Status in neutron:
  Fix Released

Bug description:
  Grafana is showing the gate-grenade-dsvm-neutron-multinode being
  unstable since July 12th [1,2]. The issue seems related to Grenade
  verify phase getting stalled [3]. Probably MTU related as the SSH
  times out.

  2016-07-09 10:16:00.716 | ++ /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:177 :   resource_get cinder cinder_server_ip
  2016-07-09 10:16:00.717 | ++ /opt/stack/new/grenade/inc/plugin:resource_get:137 :   local project=cinder
  2016-07-09 10:16:00.718 | ++ /opt/stack/new/grenade/inc/plugin:resource_get:138 :   local key=cinder_server_ip
  2016-07-09 10:16:00.720 | +++ /opt/stack/new/grenade/inc/plugin:resource_get:139 :   iniget /opt/stack/save/grenade_db.ini cinder cinder_server_ip
  2016-07-09 10:16:00.733 | ++ /opt/stack/new/grenade/inc/plugin:resource_get:139 :   local value=172.24.5.51
  2016-07-09 10:16:00.734 | ++ /opt/stack/new/grenade/inc/plugin:resource_get:140 :   echo 172.24.5.51
  2016-07-09 10:16:00.737 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:177 :   local server_ip=172.24.5.51
  2016-07-09 10:16:00.739 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:178 :   ping_check_public 172.24.5.51 30
  2016-07-09 10:16:00.740 | + /opt/stack/new/grenade/functions:ping_check_public:56 :   local ip=172.24.5.51
  2016-07-09 10:16:00.741 | + /opt/stack/new/grenade/functions:ping_check_public:57 :   local boot_timeout=30
  2016-07-09 10:16:00.743 | + /opt/stack/new/grenade/functions:ping_check_public:58 :   local expected=True
  2016-07-09 10:16:00.745 | + /opt/stack/new/grenade/functions:ping_check_public:59 :   local check_command=
  2016-07-09 10:16:00.746 | + /opt/stack/new/grenade/functions:ping_check_public:60 :   [[ True = \T\r\u\e ]]
  2016-07-09 10:16:00.747 | + /opt/stack/new/grenade/functions:ping_check_public:61 :   check_command='while ! ping -c1 -w1 172.24.5.51; do sleep 1; done'
  2016-07-09 10:16:00.749 | + /opt/stack/new/grenade/functions:ping_check_public:65 :   timeout 30 sh -c 'while ! ping -c1 -w1 172.24.5.51; do sleep 1; done'
  2016-07-09 10:16:00.755 | PING 172.24.5.51 (172.24.5.51) 56(84) bytes of data.
  2016-07-09 10:16:00.755 | 64 bytes from 172.24.5.51: icmp_seq=1 ttl=63 time=2.81 ms
  2016-07-09 10:16:00.755 |
  2016-07-09 10:16:00.755 | --- 172.24.5.51 ping statistics ---
  2016-07-09 10:16:00.755 | 1 packets transmitted, 1 received, 0% packet loss, time 0ms
  2016-07-09 10:16:00.755 | rtt min/avg/max/mdev = 2.811/2.811/2.811/0.000 ms
  2016-07-09 10:16:00.757 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:181 :   timeout 30 ssh -v -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /opt/stack/save/cinder_key.pem cirros@172.24.5.51 sync
  2016-07-09 10:16:00.761 | OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
  2016-07-09 10:16:00.762 | debug1: Reading configuration data /etc/ssh/ssh_config
  2016-07-09 10:16:00.762 | debug1: /etc/ssh/ssh_config line 19: Applying options for *
  2016-07-09 10:16:00.762 | debug1: Connecting to 172.24.5.51 [172.24.5.51] port 22.
  2016-07-09 10:16:00.763 | debug1: fd 3 clearing O_NONBLOCK
  2016-07-09 10:16:00.763 | debug1: Connection established.
  2016-07-09 10:16:00.764 | debug1: identity file /opt/stack/save/cinder_key.pem type -1
  2016-07-09 10:16:00.764 | debug1: identity file /opt/stack/save/cinder_key.pem-cert type -1
  2016-07-09 10:16:00.764 | debug1: Enabling compatibility mode for protocol 2.0
  2016-07-09 10:16:00.764 | debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
  2016-07-09 10:16:00.770 | debug1: Remote protocol version 2.0, remote software version dropbear_2012.55
  2016-07-09 10:16:00.770 | debug1: no match: dropbear_2012.55
  2016-07-09 10:16:00.770 | debug1: SSH2_MSG_KEXINIT sent
  2016-07-09 10:16:00.773 | debug1: SSH2_MSG_KEXINIT received
  2016-07-09 10:16:00.773 | debug1: kex: server->client aes128-ctr hmac-md5 none
  2016-07-09 10:16:00.773 | debug1: kex: client->server aes128-ctr hmac-md5 none
  2016-07-09 10:16:00.774 | debug1: sending SSH2_MSG_KEXDH_INIT

  ==============================
  2016-07-09 10:16:00.774 | debug1: expecting SSH2_MSG_KEXDH_REPLY
  2016-07-09 10:16:30.760 | + 
  /opt/stack/new/grenade/projects/70_cinder/resources.sh:verify_noapi:1 :   exit_trap
  ==============================

  
  2016-07-09 10:16:30.762 | + /opt/stack/new/grenade/functions:exit_trap:103 :   local r=124
  2016-07-09 10:16:30.763 | 1 verify_noapi /opt/stack/new/grenade/projects/70_cinder/resources.sh
  2016-07-09 10:16:30.763 | 173 verify /opt/stack/new/grenade/projects/70_cinder/resources.sh
  2016-07-09 10:16:30.763 | 224 main /opt/stack/new/grenade/projects/70_cinder/resources.sh
  2016-07-09 10:16:30.763 | Exit code: 124

  [1] http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=5&fullscreen
  [2] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%2270_cinder%2Fresources.sh%3Averify_noapi%3A1%20%3A%20%20%20exit_trap%5C%22
  [3] http://logs.openstack.org/46/321146/2/gate/gate-grenade-dsvm-neutron-multinode/988f952/logs/grenade.sh.txt.gz#_2016-07-14_18_03_14_092

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1603268/+subscriptions