yahoo-eng-team team mailing list archive

[Slawek Kaplonski <1836642@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

It happens from time to time in CI that VM spawned in test don't get
public-keys from metadata service. Due to that SSH to instance using
ssh-key is not possible thus test fails.

Example of such failures:

http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/testr_results.html.gz
http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/testr_results.html.gz
http://logs.openstack.org/45/645645/8/gate/tempest-full/4d7874e/testr_results.html.gz

In each of those cases it looks that neutron metadata agent was waiting
more than 10 seconds for response from n-api-meta service:

http://logs.openstack.org/09/666409/7/check/tempest-
full/08f4c53/controller/logs/screen-n-api-
meta.txt.gz#_Jul_11_23_43_16_704979 ~ 16 seconds,

http://logs.openstack.org/35/521035/8/check/tempest-
full/031b0b9/controller/logs/screen-n-api-
meta.txt.gz#_Jul_09_17_23_47_871054 ~ 13 seconds,

http://logs.openstack.org/45/645645/8/gate/tempest-
full/4d7874e/controller/logs/screen-n-api-
meta.txt.gz#_Jul_09_01_43_56_549916 ~ 17 seconds.

I have no idea why nova is responding so slow but it would be worth if
someone from Nova team would take a look into that.

Logstash query which can help to find another examples:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20get%20http%3A%2F%2F169.254.169.254%2F2009-04-04
%2Fmeta-data%2Fpublic-keys%5C%22

It is possible that in some cases of failed tests, reason of failure may
be different but problem described above is quite common in those failed
tests IMO.

** Affects: neutron
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1836642

Title:
  Metadata responses are very slow sometimes

Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  It happens from time to time in CI that VM spawned in test don't get
  public-keys from metadata service. Due to that SSH to instance using
  ssh-key is not possible thus test fails.

  Example of such failures:

  http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/testr_results.html.gz
  http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/testr_results.html.gz
  http://logs.openstack.org/45/645645/8/gate/tempest-full/4d7874e/testr_results.html.gz

  In each of those cases it looks that neutron metadata agent was
  waiting more than 10 seconds for response from n-api-meta service:

  http://logs.openstack.org/09/666409/7/check/tempest-
  full/08f4c53/controller/logs/screen-n-api-
  meta.txt.gz#_Jul_11_23_43_16_704979 ~ 16 seconds,

  http://logs.openstack.org/35/521035/8/check/tempest-
  full/031b0b9/controller/logs/screen-n-api-
  meta.txt.gz#_Jul_09_17_23_47_871054 ~ 13 seconds,

  http://logs.openstack.org/45/645645/8/gate/tempest-
  full/4d7874e/controller/logs/screen-n-api-
  meta.txt.gz#_Jul_09_01_43_56_549916 ~ 17 seconds.

  I have no idea why nova is responding so slow but it would be worth if
  someone from Nova team would take a look into that.

  Logstash query which can help to find another examples:
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20get%20http%3A%2F%2F169.254.169.254%2F2009-04-04
  %2Fmeta-data%2Fpublic-keys%5C%22

  It is possible that in some cases of failed tests, reason of failure
  may be different but problem described above is quite common in those
  failed tests IMO.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1836642/+subscriptions