yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #66827
[Bug 1707160] Re: test_create_port_in_allowed_allocation_pools test fails on ironic grenade
Reviewed: https://review.openstack.org/486706
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=f059bba6ae9642675bc1c9004a4a4adc5d1b19a6
Submitter: Jenkins
Branch: master
commit f059bba6ae9642675bc1c9004a4a4adc5d1b19a6
Author: Kenneth Giusti <kgiusti@xxxxxxxxx>
Date: Mon Jul 24 13:36:17 2017 -0400
Prevent rabbit from raising unexpected exceptions
Publishing a message using the kombu connection autoretry method may
allow exceptions from the py-amqp library to be raised up to the
application. This does not conform to the documented oslo.messaging
API.
Enhance the try except block to capture any exception and translate it
into a MessageDeliveryFailure.
There are a few cases where exceptions will be raised during autoretry
publishing: recoverable connection or channel errors, and
non-recoverable connection or channel errors.
autoretry will only retry if the error is recoverable. Non recoverable
errors are re-raised immediately regardless of the retry count.
In the case of a recoverable error it seems unlikely that retrying
either the connection or the channel yet again is going to get us
anywhere, so in this case we simply clean up the channel state, log an
error and fail the operation.
In the case of non-recoverable error we are out of luck (think
authentication failure) - further retrying will not help. Best we can
do is clean up state and log the heck out of it.
Change-Id: I2f65d2ee19a8c3e9a323b30404abbf0cbb45a216
Closes-Bug: #1705351
Closes-Bug: #1707160
** Changed in: oslo.messaging
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1707160
Title:
test_create_port_in_allowed_allocation_pools test fails on ironic
grenade
Status in neutron:
Fix Released
Status in OpenStack Compute (nova):
Incomplete
Status in oslo.messaging:
Fix Released
Bug description:
Here is an example of a job at
http://logs.openstack.org/58/487458/6/check/gate-grenade-dsvm-ironic-
ubuntu-xenial/d8f187e/console.html#_2017-07-28_09_33_52_031224
2017-07-28 09:33:52.027473 | Captured pythonlogging:
2017-07-28 09:33:52.027484 | ~~~~~~~~~~~~~~~~~~~~~~~
2017-07-28 09:33:52.027539 | 2017-07-28 09:15:48,746 9778 INFO [tempest.lib.common.rest_client] Request (PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST http://149.202.183.40:9696/v2.0/networks 0.342s
2017-07-28 09:33:52.027604 | 2017-07-28 09:15:48,746 9778 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.027633 | Body: {"network": {"name": "tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027728 | Response - Headers: {u'date': 'Fri, 28 Jul 2017 09:15:48 GMT', u'x-openstack-request-id': 'req-0502025a-db49-4f1f-b30d-c38b8098b79e', u'content-type': 'application/json', u'content-length': '582', 'content-location': 'http://149.202.183.40:9696/v2.0/networks', 'status': '201', u'connection': 'close'}
2017-07-28 09:33:52.027880 | Body: {"network":{"status":"ACTIVE","router:external":false,"availability_zone_hints":[],"availability_zones":[],"description":"","subnets":[],"shared":false,"tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:48Z","tags":[],"ipv6_address_scope":null,"mtu":1450,"updated_at":"2017-07-28T09:15:48Z","admin_state_up":true,"revision_number":2,"ipv4_address_scope":null,"is_default":false,"port_security_enabled":true,"project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","name":"tempest-PortsTestJSON-test-network-1596805013"}}
2017-07-28 09:33:52.027936 | 2017-07-28 09:15:49,430 9778 INFO [tempest.lib.common.rest_client] Request (PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST http://149.202.183.40:9696/v2.0/subnets 0.682s
2017-07-28 09:33:52.027998 | 2017-07-28 09:15:49,431 9778 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.028054 | Body: {"subnet": {"ip_version": 4, "allocation_pools": [{"end": "10.1.0.14", "start": "10.1.0.2"}], "network_id": "b8a3fb1c-86a4-4518-8c3a-dd12db585659", "gateway_ip": "10.1.0.1", "cidr": "10.1.0.0/28"}}
2017-07-28 09:33:52.028135 | Response - Headers: {u'date': 'Fri, 28 Jul 2017 09:15:49 GMT', u'x-openstack-request-id': 'req-1a50b739-8683-4aaa-ba4a-6e9daf73f1c8', u'content-type': 'application/json', u'content-length': '594', 'content-location': 'http://149.202.183.40:9696/v2.0/subnets', 'status': '201', u'connection': 'close'}
2017-07-28 09:33:52.030085 | Body: {"subnet":{"service_types":[],"description":"","enable_dhcp":true,"tags":[],"network_id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","tenant_id":"5c851bb85bef4b008714ef04d1fe3671","created_at":"2017-07-28T09:15:49Z","dns_nameservers":[],"updated_at":"2017-07-28T09:15:49Z","gateway_ip":"10.1.0.1","ipv6_ra_mode":null,"allocation_pools":[{"start":"10.1.0.2","end":"10.1.0.14"}],"host_routes":[],"revision_number":0,"ip_version":4,"ipv6_address_mode":null,"cidr":"10.1.0.0/28","project_id":"5c851bb85bef4b008714ef04d1fe3671","id":"be974b50-e56b-44a8-86a9-6bcc345f9d55","subnetpool_id":null,"name":""}}
2017-07-28 09:33:52.030176 | 2017-07-28 09:15:50,616 9778 INFO [tempest.lib.common.rest_client] Request (PortsTestJSON:test_create_port_in_allowed_allocation_pools): 201 POST http://149.202.183.40:9696/v2.0/ports 1.185s
2017-07-28 09:33:52.030232 | 2017-07-28 09:15:50,617 9778 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.030259 | Body: {"port": {"network_id": "b8a3fb1c-86a4-4518-8c3a-dd12db585659"}}
2017-07-28 09:33:52.030369 | Response - Headers: {u'date': 'Fri, 28 Jul 2017 09:15:50 GMT', u'x-openstack-request-id': 'req-6b57ff81-c874-4e97-8183-bd57c7e8de81', u'content-type': 'application/json', u'content-length': '691', 'content-location': 'http://149.202.183.40:9696/v2.0/ports', 'status': '201', u'connection': 'close'}
2017-07-28 09:33:52.030596 | Body: {"port":{"status":"DOWN","created_at":"2017-07-28T09:15:49Z","description":"","allowed_address_pairs":[],"tags":[],"network_id":"b8a3fb1c-86a4-4518-8c3a-dd12db585659","tenant_id":"5c851bb85bef4b008714ef04d1fe3671","extra_dhcp_opts":[],"admin_state_up":true,"updated_at":"2017-07-28T09:15:50Z","name":"","device_owner":"","revision_number":3,"mac_address":"fa:16:3e:a4:39:53","binding:vnic_type":"normal","port_security_enabled":true,"project_id":"5c851bb85bef4b008714ef04d1fe3671","fixed_ips":[{"subnet_id":"be974b50-e56b-44a8-86a9-6bcc345f9d55","ip_address":"10.1.0.4"}],"id":"a873e67a-a171-4860-9f47-be50dc061b10","security_groups":["107c3ae4-4c9f-4851-935f-5602c35270dc"],"device_id":""}}
2017-07-28 09:33:52.030660 | 2017-07-28 09:15:51,354 9778 INFO [tempest.lib.common.rest_client] Request (PortsTestJSON:_run_cleanups): 204 DELETE http://149.202.183.40:9696/v2.0/ports/a873e67a-a171-4860-9f47-be50dc061b10 0.735s
2017-07-28 09:33:52.030736 | 2017-07-28 09:15:51,354 9778 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.030747 | Body: None
2017-07-28 09:33:52.030824 | Response - Headers: {u'date': 'Fri, 28 Jul 2017 09:15:51 GMT', u'x-openstack-request-id': 'req-018a5b4a-cb5e-4dc1-b807-e3223b1fc83a', u'content-length': '0', 'content-location': 'http://149.202.183.40:9696/v2.0/ports/a873e67a-a171-4860-9f47-be50dc061b10', 'status': '204', u'connection': 'close'}
2017-07-28 09:33:52.030835 | Body:
2017-07-28 09:33:52.030918 | 2017-07-28 09:16:51,390 9778 WARNING [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='149.202.183.40', port=9696): Read timed out. (read timeout=60)",)': /v2.0/subnets/be974b50-e56b-44a8-86a9-6bcc345f9d55
2017-07-28 09:33:52.030974 | 2017-07-28 09:16:51,508 9778 INFO [tempest.lib.common.rest_client] Request (PortsTestJSON:_run_cleanups): 404 DELETE http://149.202.183.40:9696/v2.0/subnets/be974b50-e56b-44a8-86a9-6bcc345f9d55 60.153s
2017-07-28 09:33:52.031023 | 2017-07-28 09:16:51,508 9778 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-07-28 09:33:52.031047 | Body: None
2017-07-28 09:33:52.031161 | Response - Headers: {u'date': 'Fri, 28 Jul 2017 09:16:51 GMT', u'x-openstack-request-id': 'req-86d01b29-c1b1-4396-a0a8-c360c78dcbd5', u'content-type': 'application/json', u'content-length': '136', 'content-location': 'http://149.202.183.40:9696/v2.0/subnets/be974b50-e56b-44a8-86a9-6bcc345f9d55', 'status': '404', u'connection': 'close'}
2017-07-28 09:33:52.031224 | Body: {"NeutronError": {"message": "Subnet be974b50-e56b-44a8-86a9-6bcc345f9d55 could not be found.", "type": "SubnetNotFound", "detail": ""}}
Seems like there is some kind of race here, this failure should not be related to ironic.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1707160/+subscriptions
References
