yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1481715] [NEW] test_list_servers_filtered_by_ip_regex is racey in the gate

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 05 Aug 2015 11:16:12 -0000
Reply-to: Bug 1481715 <1481715@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When running in the upstream gate,
test_list_servers_filtered_by_ip_regex is failing from time to time, and
there isn't enough info to determine why.

Example fail: http://logs.openstack.org/10/185910/13/gate/gate-tempest-
dsvm-nova-v21-full/eaf043c//console.html#_2015-08-05_10_18_54_522

2015-08-05 10:18:54.523 | Captured traceback:
2015-08-05 10:18:54.523 | ~~~~~~~~~~~~~~~~~~~
2015-08-05 10:18:54.523 |     Traceback (most recent call last):
2015-08-05 10:18:54.523 |       File "tempest/api/compute/servers/test_list_server_filters.py", line 313, in test_list_servers_filtered_by_ip_regex
2015-08-05 10:18:54.523 |         self.assertIn(self.s3_name, map(lambda x: x['name'], servers))
2015-08-05 10:18:54.523 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/testtools/testcase.py", line 356, in assertIn
2015-08-05 10:18:54.524 |         self.assertThat(haystack, Contains(needle), message)
2015-08-05 10:18:54.524 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-08-05 10:18:54.524 |         raise mismatch_error
2015-08-05 10:18:54.524 |     testtools.matchers._impl.MismatchError: 'tempest-ListServerFiltersTestJSON-instance-605437920' not in [u'tempest-ListServerFiltersTestJSON-instance-1212139707', u'tempest-ListServerFiltersTestJSON-instance-237240309']
2015-08-05 10:18:54.524 |     
2015-08-05 10:18:54.524 | 
2015-08-05 10:18:54.524 | Captured pythonlogging:
2015-08-05 10:18:54.525 | ~~~~~~~~~~~~~~~~~~~~~~~
2015-08-05 10:18:54.525 |     2015-08-05 09:59:53,317 7643 INFO     [tempest_lib.common.rest_client] Request (ListServerFiltersTestJSON:test_list_servers_filtered_by_ip_regex): 200 GET http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c 0.189s
2015-08-05 10:18:54.525 |     2015-08-05 09:59:53,317 7643 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
2015-08-05 10:18:54.525 |             Body: None
2015-08-05 10:18:54.525 |         Response - Headers: {'connection': 'close', 'content-length': '1572', 'vary': 'X-OpenStack-Nova-API-Version', 'content-location': 'http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c', 'x-compute-request-id': 'req-e3126afe-da89-47d9-82e5-5f4d31604ae3', 'date': 'Wed, 05 Aug 2015 09:59:53 GMT', 'content-type': 'application/json', 'status': '200', 'x-openstack-nova-api-version': '2.1'}
2015-08-05 10:18:54.525 |             Body: {"server": {"status": "ACTIVE", "updated": "2015-08-05T09:59:51Z", "hostId": "f037f08820372dae376b110fd010a1ce5fe9fbf7cb31383f4ab9cbb5", "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:78:87:13", "version": 4, "addr": "10.1.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "self"}, {"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "bookmark"}], "key_name": null, "image": {"id": "8e22ebff-2714-4d46-ae34-9a7e00c6cb15", "links": [{"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/images/8e22ebff-2714-4d46-ae34-9a7e00c6cb15";, "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2015-08-05T09:59:14.000000", "flavor": {"id": "42", "links": [{"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/flavors/42";, "rel": "bookmark"}]}, "id": "bd6dde46-63cb-47f6-83b5-91ea659fc87c", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "d23d48cc19f94db19e602df2fe5bda5b", "name": "tempest-ListServerFiltersTestJSON-instance-237240309", "created": "2015-08-05T09:59:10Z", "tenant_id": "83088e4e5eff4fd9b6fce26b64b57a91", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "True", "metadata": {}}}
2015-08-05 10:18:54.526 |     2015-08-05 09:59:53,423 7643 INFO     [tempest_lib.common.rest_client] Request (ListServerFiltersTestJSON:test_list_servers_filtered_by_ip_regex): 200 GET http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers?ip=10.1. 0.099s
2015-08-05 10:18:54.536 |     2015-08-05 09:59:53,423 7643 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
2015-08-05 10:18:54.536 |             Body: None
2015-08-05 10:18:54.536 |         Response - Headers: {'connection': 'close', 'content-length': '786', 'vary': 'X-OpenStack-Nova-API-Version', 'content-location': 'http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers?ip=10.1.', 'x-compute-request-id': 'req-554bc5c6-7495-4d91-8257-df83f21601b9', 'date': 'Wed, 05 Aug 2015 09:59:53 GMT', 'content-type': 'application/json', 'status': '200', 'x-openstack-nova-api-version': '2.1'}
2015-08-05 10:18:54.536 |             Body: {"servers": [{"id": "50e86b98-f853-4d97-92c5-7fcaf4a066a7", "links": [{"href": "http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/50e86b98-f853-4d97-92c5-7fcaf4a066a7";, "rel": "self"}, {"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/servers/50e86b98-f853-4d97-92c5-7fcaf4a066a7";, "rel": "bookmark"}], "name": "tempest-ListServerFiltersTestJSON-instance-1212139707"}, {"id": "bd6dde46-63cb-47f6-83b5-91ea659fc87c", "links": [{"href": "http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "self"}, {"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "bookmark"}], "name": "tempest-ListServerFiltersTestJSON-instance-237240309"}]}
2015-08-05 10:18:54.536 |     

Clearly, things aren't being returned, but it's not really clear what
the state of the world is here. Extra debugging needs to be added to
dump all the servers on a failure.

** Affects: nova
     Importance: High
         Status: Confirmed

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1481715

Title:
  test_list_servers_filtered_by_ip_regex is racey in the gate

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  When running in the upstream gate,
  test_list_servers_filtered_by_ip_regex is failing from time to time,
  and there isn't enough info to determine why.

  Example fail: http://logs.openstack.org/10/185910/13/gate/gate-
  tempest-dsvm-
  nova-v21-full/eaf043c//console.html#_2015-08-05_10_18_54_522

  2015-08-05 10:18:54.523 | Captured traceback:
  2015-08-05 10:18:54.523 | ~~~~~~~~~~~~~~~~~~~
  2015-08-05 10:18:54.523 |     Traceback (most recent call last):
  2015-08-05 10:18:54.523 |       File "tempest/api/compute/servers/test_list_server_filters.py", line 313, in test_list_servers_filtered_by_ip_regex
  2015-08-05 10:18:54.523 |         self.assertIn(self.s3_name, map(lambda x: x['name'], servers))
  2015-08-05 10:18:54.523 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/testtools/testcase.py", line 356, in assertIn
  2015-08-05 10:18:54.524 |         self.assertThat(haystack, Contains(needle), message)
  2015-08-05 10:18:54.524 |       File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
  2015-08-05 10:18:54.524 |         raise mismatch_error
  2015-08-05 10:18:54.524 |     testtools.matchers._impl.MismatchError: 'tempest-ListServerFiltersTestJSON-instance-605437920' not in [u'tempest-ListServerFiltersTestJSON-instance-1212139707', u'tempest-ListServerFiltersTestJSON-instance-237240309']
  2015-08-05 10:18:54.524 |     
  2015-08-05 10:18:54.524 | 
  2015-08-05 10:18:54.524 | Captured pythonlogging:
  2015-08-05 10:18:54.525 | ~~~~~~~~~~~~~~~~~~~~~~~
  2015-08-05 10:18:54.525 |     2015-08-05 09:59:53,317 7643 INFO     [tempest_lib.common.rest_client] Request (ListServerFiltersTestJSON:test_list_servers_filtered_by_ip_regex): 200 GET http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c 0.189s
  2015-08-05 10:18:54.525 |     2015-08-05 09:59:53,317 7643 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
  2015-08-05 10:18:54.525 |             Body: None
  2015-08-05 10:18:54.525 |         Response - Headers: {'connection': 'close', 'content-length': '1572', 'vary': 'X-OpenStack-Nova-API-Version', 'content-location': 'http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c', 'x-compute-request-id': 'req-e3126afe-da89-47d9-82e5-5f4d31604ae3', 'date': 'Wed, 05 Aug 2015 09:59:53 GMT', 'content-type': 'application/json', 'status': '200', 'x-openstack-nova-api-version': '2.1'}
  2015-08-05 10:18:54.525 |             Body: {"server": {"status": "ACTIVE", "updated": "2015-08-05T09:59:51Z", "hostId": "f037f08820372dae376b110fd010a1ce5fe9fbf7cb31383f4ab9cbb5", "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:78:87:13", "version": 4, "addr": "10.1.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "self"}, {"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "bookmark"}], "key_name": null, "image": {"id": "8e22ebff-2714-4d46-ae34-9a7e00c6cb15", "links": [{"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/images/8e22ebff-2714-4d46-ae34-9a7e00c6cb15";, "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2015-08-05T09:59:14.000000", "flavor": {"id": "42", "links": [{"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/flavors/42";, "rel": "bookmark"}]}, "id": "bd6dde46-63cb-47f6-83b5-91ea659fc87c", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "d23d48cc19f94db19e602df2fe5bda5b", "name": "tempest-ListServerFiltersTestJSON-instance-237240309", "created": "2015-08-05T09:59:10Z", "tenant_id": "83088e4e5eff4fd9b6fce26b64b57a91", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "True", "metadata": {}}}
  2015-08-05 10:18:54.526 |     2015-08-05 09:59:53,423 7643 INFO     [tempest_lib.common.rest_client] Request (ListServerFiltersTestJSON:test_list_servers_filtered_by_ip_regex): 200 GET http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers?ip=10.1. 0.099s
  2015-08-05 10:18:54.536 |     2015-08-05 09:59:53,423 7643 DEBUG    [tempest_lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
  2015-08-05 10:18:54.536 |             Body: None
  2015-08-05 10:18:54.536 |         Response - Headers: {'connection': 'close', 'content-length': '786', 'vary': 'X-OpenStack-Nova-API-Version', 'content-location': 'http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers?ip=10.1.', 'x-compute-request-id': 'req-554bc5c6-7495-4d91-8257-df83f21601b9', 'date': 'Wed, 05 Aug 2015 09:59:53 GMT', 'content-type': 'application/json', 'status': '200', 'x-openstack-nova-api-version': '2.1'}
  2015-08-05 10:18:54.536 |             Body: {"servers": [{"id": "50e86b98-f853-4d97-92c5-7fcaf4a066a7", "links": [{"href": "http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/50e86b98-f853-4d97-92c5-7fcaf4a066a7";, "rel": "self"}, {"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/servers/50e86b98-f853-4d97-92c5-7fcaf4a066a7";, "rel": "bookmark"}], "name": "tempest-ListServerFiltersTestJSON-instance-1212139707"}, {"id": "bd6dde46-63cb-47f6-83b5-91ea659fc87c", "links": [{"href": "http://127.0.0.1:8774/v2/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "self"}, {"href": "http://127.0.0.1:8774/83088e4e5eff4fd9b6fce26b64b57a91/servers/bd6dde46-63cb-47f6-83b5-91ea659fc87c";, "rel": "bookmark"}], "name": "tempest-ListServerFiltersTestJSON-instance-237240309"}]}
  2015-08-05 10:18:54.536 |     

  Clearly, things aren't being returned, but it's not really clear what
  the state of the world is here. Extra debugging needs to be added to
  dump all the servers on a failure.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1481715/+subscriptions
Follow ups

[Bug 1481715] Re: test_list_servers_filtered_by_ip_regex is racey in the gate
From: Jay Pipes, 2017-05-17