yahoo-eng-team team mailing list archive

[Ed Bak <ed.bak2@xxxxxx>]

Public bug reported:

In large scale environments, instances can fail to get their metadata.
Tests were performed in a 100 compute node environment creating 4000
vms.  15-20 vms will fail all 20 metadata request attempts.   This has
been reproduced multiple times with similar results.  All of the vms
successfully obtain a private ip and are pingable but a small number of
vms fail to be reachable via ssh.   Increasing the number of metadata
request attempts in a Cirros test image shows that the metadata requests
eventually will succeed.  It just takes a long time in some cases.
Below is a list of instance uuids I collected from one test and the
number of metadata request attempts it took to be successful.  These
tests were performed using stable/liberty with a small number of patches
cherry-picked from Mitaka which were intended to improve performance.


705c3582-f39b-482d-9a6e-d78bc033d3e7   5
27f93574-19fe-4b88-ad6e-c518022ef66a     2
ff668db8-196e-4ec3-82d9-f7ab5a302305    57
b3f97acb-6374-4406-9474-7bacfc3486cd    42
80c19187-7c19-4adc-ad3a-51342f00d799   51
071f60d5-2a9a-4448-b14b-9016c9eee4eb  47
d39f336e-0fb4-4934-b835-e791661d60f1  36
a5627d9f-fd2d-48b0-ada2-f519a97849ee    5
3c24145e-8e11-4e79-8618-fca0416ea030   41
a36ab8fd-4e53-4265-a2bf-6945ac5d8811   46
a9400361-8941-4f03-b11d-0940b5499b4b  37
7449efbd-1df6-4fcc-88d5-e4e355ae7963    24
45c6a108-c18b-4284-9ede-3e5f8d7851be  30
fbe7c6fc-6aec-464c-87b7-0800836f7754       7
cb5a3a49-45b9-40de-8c62-903bee1925f4   37
0c7151ce-79dc-4d55-a617-7f4182cb2194   14
0f1c24a0-3b97-4d56-8feb-b30d67cf6852   44
8c359465-198f-4654-84bb-f334f0400d58   10
b3a5a3df-28c4-40c3-adba-856a0fcbd29e   55
38ee6525-441e-4640-a998-ad89b8d3f8be   2
07ecde16-c274-481e-8169-4febb15c7273   48
f77cd7aa-89e2-4d2c-a89f-e19ff430e5a4     31
b9acdba1-1794-4fa8-bbe3-ffb94f86d19b     3
30824aa6-3df5-4a43-a701-dd33da7f704f   13
5216ffc0-4a8d-4a3e-a4e3-5473b96ca47b   40
999512ff-70e3-4cfd-9cb4-c5788a02fee6     4

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1526855

Title:
  VMs fail to get metadata in large scale environments

Status in neutron:
  New

Bug description:
  In large scale environments, instances can fail to get their metadata.
  Tests were performed in a 100 compute node environment creating 4000
  vms.  15-20 vms will fail all 20 metadata request attempts.   This has
  been reproduced multiple times with similar results.  All of the vms
  successfully obtain a private ip and are pingable but a small number
  of vms fail to be reachable via ssh.   Increasing the number of
  metadata request attempts in a Cirros test image shows that the
  metadata requests eventually will succeed.  It just takes a long time
  in some cases.   Below is a list of instance uuids I collected from
  one test and the number of metadata request attempts it took to be
  successful.  These tests were performed using stable/liberty with a
  small number of patches cherry-picked from Mitaka which were intended
  to improve performance.

  
  705c3582-f39b-482d-9a6e-d78bc033d3e7   5
  27f93574-19fe-4b88-ad6e-c518022ef66a     2
  ff668db8-196e-4ec3-82d9-f7ab5a302305    57
  b3f97acb-6374-4406-9474-7bacfc3486cd    42
  80c19187-7c19-4adc-ad3a-51342f00d799   51
  071f60d5-2a9a-4448-b14b-9016c9eee4eb  47
  d39f336e-0fb4-4934-b835-e791661d60f1  36
  a5627d9f-fd2d-48b0-ada2-f519a97849ee    5
  3c24145e-8e11-4e79-8618-fca0416ea030   41
  a36ab8fd-4e53-4265-a2bf-6945ac5d8811   46
  a9400361-8941-4f03-b11d-0940b5499b4b  37
  7449efbd-1df6-4fcc-88d5-e4e355ae7963    24
  45c6a108-c18b-4284-9ede-3e5f8d7851be  30
  fbe7c6fc-6aec-464c-87b7-0800836f7754       7
  cb5a3a49-45b9-40de-8c62-903bee1925f4   37
  0c7151ce-79dc-4d55-a617-7f4182cb2194   14
  0f1c24a0-3b97-4d56-8feb-b30d67cf6852   44
  8c359465-198f-4654-84bb-f334f0400d58   10
  b3a5a3df-28c4-40c3-adba-856a0fcbd29e   55
  38ee6525-441e-4640-a998-ad89b8d3f8be   2
  07ecde16-c274-481e-8169-4febb15c7273   48
  f77cd7aa-89e2-4d2c-a89f-e19ff430e5a4     31
  b9acdba1-1794-4fa8-bbe3-ffb94f86d19b     3
  30824aa6-3df5-4a43-a701-dd33da7f704f   13
  5216ffc0-4a8d-4a3e-a4e3-5473b96ca47b   40
  999512ff-70e3-4cfd-9cb4-c5788a02fee6     4

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1526855/+subscriptions