← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #82271

[Bug 1872693] [NEW] Timeout exception in scenario tests with advanced image

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

It seems that sometimes on slow nodes, tests which requires advanced image, like e.g. neutron_tempest_plugin.scenario.test_multicast.MulticastTestIPv4.test_multicast_between_vms_on_same_network or neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity may fail due to global timeout reach.
So we should give some more time for such tests, just to make sure that cleanup after test can perform fine and test will not fail in that last phase.

Examples of failure:

        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:39 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-f6803ed4-cd44-4aaf-852b-9642166f66fd', 'x-compute-request-id': 'req-f6803ed4-cd44-4aaf-852b-9642166f66fd', 'connection': 'close', 'status': '204', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9'}
        Body: b''
2020-03-24 19:34:41,164 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9 0.664s
2020-03-24 19:34:41,164 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:40 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-9e05c04f-d73f-410e-a593-9732cd0d368b', 'x-compute-request-id': 'req-9e05c04f-d73f-410e-a593-9732cd0d368b', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9'}
        Body: b'{"server": {"id": "6d8abb96-e8de-4833-86f0-2307493e42f9", "name": "tempest-multicast-server-968193459", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:24:33Z", "updated": "2020-03-24T19:34:40Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.12", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:e2:f4:69"}, {"version": 4, "addr": "172.24.5.147", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:e2:f4:69"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/6d8abb96-e8de-4833-86f0-2307493e42f9"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:24:40.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2020-03-24 19:34:41,762 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 204 DELETE https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.322s
2020-03-24 19:34:41,762 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:41 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-06f636f8-ebd2-4974-979b-cf7a81e1f29f', 'x-compute-request-id': 'req-06f636f8-ebd2-4974-979b-cf7a81e1f29f', 'connection': 'close', 'status': '204', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
        Body: b''
2020-03-24 19:34:42,583 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.820s
2020-03-24 19:34:42,583 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:41 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-6b4b40e9-7181-497d-b8dd-f5e32189b897', 'x-compute-request-id': 'req-6b4b40e9-7181-497d-b8dd-f5e32189b897', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
        Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:42Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}, {"version": 4, "addr": "172.24.5.125", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2020-03-24 19:34:44,202 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.612s
2020-03-24 19:34:44,203 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:43 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-54679f2b-ea1a-43c2-920c-757819b2845f', 'x-compute-request-id': 'req-54679f2b-ea1a-43c2-920c-757819b2845f', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
        Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:43Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}, {"version": 4, "addr": "172.24.5.125", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2020-03-24 19:34:45,520 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.307s
2020-03-24 19:34:45,521 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:45 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-7affb2b1-108d-4de5-b6f4-a42602c7c6e7', 'x-compute-request-id': 'req-7affb2b1-108d-4de5-b6f4-a42602c7c6e7', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
        Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:43Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}, {"version": 4, "addr": "172.24.5.125", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2020-03-24 19:34:46,735 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.206s
2020-03-24 19:34:46,735 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:46 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1322', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-d0381bc4-6667-48c3-8c04-0faa3bb95f00', 'x-compute-request-id': 'req-d0381bc4-6667-48c3-8c04-0faa3bb95f00', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
        Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:43Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2020-03-24 19:34:47,847 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 404 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.100s
2020-03-24 19:34:47,848 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:47 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'content-type': 'application/json; charset=UTF-8', 'content-length': '111', 'x-openstack-request-id': 'req-3af792f4-16b1-4320-8e66-d6a192ad7a5f', 'x-compute-request-id': 'req-3af792f4-16b1-4320-8e66-d6a192ad7a5f', 'connection': 'close', 'status': '404', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
        Body: b'{"itemNotFound": {"code": 404, "message": "Instance fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f could not be found."}}'
2020-03-24 19:34:48,060 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 204 DELETE https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 0.211s
2020-03-24 19:34:48,060 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:47 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-60a7c83f-4b76-4c94-9cf6-e57fbed2b982', 'x-compute-request-id': 'req-60a7c83f-4b76-4c94-9cf6-e57fbed2b982', 'connection': 'close', 'status': '204', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
        Body: b''
2020-03-24 19:34:48,438 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 0.377s
2020-03-24 19:34:48,438 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:48 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1637', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-980b088a-799e-432c-ad02-bf6204a115f6', 'x-compute-request-id': 'req-980b088a-799e-432c-ad02-bf6204a115f6', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
        Body: b'{"server": {"id": "dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6", "name": "tempest-multicast-server-1305662029", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:14:43Z", "updated": "2020-03-24T19:34:48Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.11", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}, {"version": 4, "addr": "172.24.5.249", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:15:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2020-03-24 19:34:50,655 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 1.210s
2020-03-24 19:34:50,656 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:49 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1637', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-846415cc-4906-4239-b483-2725e4cb4800', 'x-compute-request-id': 'req-846415cc-4906-4239-b483-2725e4cb4800', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
        Body: b'{"server": {"id": "dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6", "name": "tempest-multicast-server-1305662029", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:14:43Z", "updated": "2020-03-24T19:34:49Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.11", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}, {"version": 4, "addr": "172.24.5.249", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:15:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2020-03-24 19:34:51,764 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 404 GET https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 0.098s
2020-03-24 19:34:51,765 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:51 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'content-type': 'application/json; charset=UTF-8', 'content-length': '111', 'x-openstack-request-id': 'req-12f4f0a5-7fbc-4f3c-8000-1d3e5c8e3fa1', 'x-compute-request-id': 'req-12f4f0a5-7fbc-4f3c-8000-1d3e5c8e3fa1', 'connection': 'close', 'status': '404', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
        Body: b'{"itemNotFound": {"code": 404, "message": "Instance dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 could not be found."}}'
}}}

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/tempest/common/waiters.py", line 111, in wait_for_server_termination
    time.sleep(client.build_interval)
  File "/usr/local/lib/python3.6/dist-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException


links to examples of failures:
https://103900b4a03cdd60217d-625a0eb0440aa527fbdb216e8991f5a6.ssl.cf5.rackcdn.com/714726/1/check/neutron-ovn-tempest-ovs-release/2b52c20/testr_results.html
https://5481f30c7f7864f4d07b-85bb929ac58f1a419407783b8f581cf5.ssl.cf5.rackcdn.com/710755/11/check/neutron-tempest-plugin-scenario-openvswitch/2d4ae79/testr_results.html

logstash query:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20111%2C%20in%20wait_for_server_termination%5C%22

** Affects: neutron
     Importance: Critical
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: gate-failure tempest

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1872693

Title:
  Timeout exception in scenario tests with advanced image

Status in neutron:
  Confirmed

Bug description:
  It seems that sometimes on slow nodes, tests which requires advanced image, like e.g. neutron_tempest_plugin.scenario.test_multicast.MulticastTestIPv4.test_multicast_between_vms_on_same_network or neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity may fail due to global timeout reach.
  So we should give some more time for such tests, just to make sure that cleanup after test can perform fine and test will not fail in that last phase.

  Examples of failure:

          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:39 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-f6803ed4-cd44-4aaf-852b-9642166f66fd', 'x-compute-request-id': 'req-f6803ed4-cd44-4aaf-852b-9642166f66fd', 'connection': 'close', 'status': '204', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9'}
          Body: b''
  2020-03-24 19:34:41,164 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9 0.664s
  2020-03-24 19:34:41,164 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:40 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-9e05c04f-d73f-410e-a593-9732cd0d368b', 'x-compute-request-id': 'req-9e05c04f-d73f-410e-a593-9732cd0d368b', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9'}
          Body: b'{"server": {"id": "6d8abb96-e8de-4833-86f0-2307493e42f9", "name": "tempest-multicast-server-968193459", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:24:33Z", "updated": "2020-03-24T19:34:40Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.12", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:e2:f4:69"}, {"version": 4, "addr": "172.24.5.147", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:e2:f4:69"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/6d8abb96-e8de-4833-86f0-2307493e42f9"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/6d8abb96-e8de-4833-86f0-2307493e42f9"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:24:40.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
  2020-03-24 19:34:41,762 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 204 DELETE https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.322s
  2020-03-24 19:34:41,762 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:41 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-06f636f8-ebd2-4974-979b-cf7a81e1f29f', 'x-compute-request-id': 'req-06f636f8-ebd2-4974-979b-cf7a81e1f29f', 'connection': 'close', 'status': '204', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
          Body: b''
  2020-03-24 19:34:42,583 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.820s
  2020-03-24 19:34:42,583 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:41 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-6b4b40e9-7181-497d-b8dd-f5e32189b897', 'x-compute-request-id': 'req-6b4b40e9-7181-497d-b8dd-f5e32189b897', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
          Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:42Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}, {"version": 4, "addr": "172.24.5.125", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
  2020-03-24 19:34:44,202 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.612s
  2020-03-24 19:34:44,203 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:43 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-54679f2b-ea1a-43c2-920c-757819b2845f', 'x-compute-request-id': 'req-54679f2b-ea1a-43c2-920c-757819b2845f', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
          Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:43Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}, {"version": 4, "addr": "172.24.5.125", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
  2020-03-24 19:34:45,520 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.307s
  2020-03-24 19:34:45,521 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:45 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1636', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-7affb2b1-108d-4de5-b6f4-a42602c7c6e7', 'x-compute-request-id': 'req-7affb2b1-108d-4de5-b6f4-a42602c7c6e7', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
          Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:43Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.8", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}, {"version": 4, "addr": "172.24.5.125", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:30:7e:e5"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
  2020-03-24 19:34:46,735 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.206s
  2020-03-24 19:34:46,735 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:46 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1322', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-d0381bc4-6667-48c3-8c04-0faa3bb95f00', 'x-compute-request-id': 'req-d0381bc4-6667-48c3-8c04-0faa3bb95f00', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
          Body: b'{"server": {"id": "fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f", "name": "tempest-multicast-server-1504820408", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:19:26Z", "updated": "2020-03-24T19:34:43Z", "addresses": {}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:19:36.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
  2020-03-24 19:34:47,847 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 404 GET https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f 0.100s
  2020-03-24 19:34:47,848 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:47 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'content-type': 'application/json; charset=UTF-8', 'content-length': '111', 'x-openstack-request-id': 'req-3af792f4-16b1-4320-8e66-d6a192ad7a5f', 'x-compute-request-id': 'req-3af792f4-16b1-4320-8e66-d6a192ad7a5f', 'connection': 'close', 'status': '404', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f'}
          Body: b'{"itemNotFound": {"code": 404, "message": "Instance fde7c8ae-7fea-4aa9-9f3b-e2d8ac864d4f could not be found."}}'
  2020-03-24 19:34:48,060 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 204 DELETE https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 0.211s
  2020-03-24 19:34:48,060 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:47 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-60a7c83f-4b76-4c94-9cf6-e57fbed2b982', 'x-compute-request-id': 'req-60a7c83f-4b76-4c94-9cf6-e57fbed2b982', 'connection': 'close', 'status': '204', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
          Body: b''
  2020-03-24 19:34:48,438 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 0.377s
  2020-03-24 19:34:48,438 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:48 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1637', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-980b088a-799e-432c-ad02-bf6204a115f6', 'x-compute-request-id': 'req-980b088a-799e-432c-ad02-bf6204a115f6', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
          Body: b'{"server": {"id": "dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6", "name": "tempest-multicast-server-1305662029", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:14:43Z", "updated": "2020-03-24T19:34:48Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.11", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}, {"version": 4, "addr": "172.24.5.249", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:15:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
  2020-03-24 19:34:50,655 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 200 GET https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 1.210s
  2020-03-24 19:34:50,656 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:49 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'content-length': '1637', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-846415cc-4906-4239-b483-2725e4cb4800', 'x-compute-request-id': 'req-846415cc-4906-4239-b483-2725e4cb4800', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
          Body: b'{"server": {"id": "dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6", "name": "tempest-multicast-server-1305662029", "status": "ACTIVE", "tenant_id": "1247c419c0bd41bb948eacb87e8bf8e3", "user_id": "3b79839ed47d4c95b8f363497a767ff2", "metadata": {}, "hostId": "9ee32c342fa222fa161f31fa44b4ad381d7a977bf32d6c6db0070294", "image": {"id": "3a21d3c5-44f2-4543-97cd-aee290e1a6a8", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/images/3a21d3c5-44f2-4543-97cd-aee290e1a6a8"}]}, "flavor": {"id": "d1", "links": [{"rel": "bookmark", "href": "https://158.69.70.109/compute/flavors/d1"}]}, "created": "2020-03-24T19:14:43Z", "updated": "2020-03-24T19:34:49Z", "addresses": {"tempest-test-network--2075552321": [{"version": 4, "addr": "10.1.0.11", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}, {"version": 4, "addr": "172.24.5.249", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:46:f7"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}, {"rel": "bookmark", "href": "https://158.69.70.109/compute/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "tempest-keypair-test-913711661", "OS-SRV-USG:launched_at": "2020-03-24T19:15:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "secgroup_mtu"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
  2020-03-24 19:34:51,764 19073 INFO     [tempest.lib.common.rest_client] Request (MulticastTestIPv4:_run_cleanups): 404 GET https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 0.098s
  2020-03-24 19:34:51,765 19073 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: None
      Response - Headers: {'date': 'Tue, 24 Mar 2020 19:34:51 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'content-type': 'application/json; charset=UTF-8', 'content-length': '111', 'x-openstack-request-id': 'req-12f4f0a5-7fbc-4f3c-8000-1d3e5c8e3fa1', 'x-compute-request-id': 'req-12f4f0a5-7fbc-4f3c-8000-1d3e5c8e3fa1', 'connection': 'close', 'status': '404', 'content-location': 'https://158.69.70.109/compute/v2.1/servers/dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6'}
          Body: b'{"itemNotFound": {"code": 404, "message": "Instance dcc1b3f3-d7a5-4b63-b72b-0e250e2386e6 could not be found."}}'
  }}}

  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
      return func(*args, **kwargs)
    File "/opt/stack/tempest/tempest/common/waiters.py", line 111, in wait_for_server_termination
      time.sleep(client.build_interval)
    File "/usr/local/lib/python3.6/dist-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
      raise TimeoutException()
  fixtures._fixtures.timeout.TimeoutException


  links to examples of failures:
  https://103900b4a03cdd60217d-625a0eb0440aa527fbdb216e8991f5a6.ssl.cf5.rackcdn.com/714726/1/check/neutron-ovn-tempest-ovs-release/2b52c20/testr_results.html
  https://5481f30c7f7864f4d07b-85bb929ac58f1a419407783b8f581cf5.ssl.cf5.rackcdn.com/710755/11/check/neutron-tempest-plugin-scenario-openvswitch/2d4ae79/testr_results.html

  logstash query:
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20111%2C%20in%20wait_for_server_termination%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1872693/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next