yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1767058] Re: Rally test boot_and_delete_server_with_secgroups fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1767058@xxxxxxxxxxxxxxxxxx>
Date: Tue, 08 May 2018 23:09:36 -0000
Reply-to: Bug 1767058 <1767058@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/565183
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a0179f7ab9b9c7f07ce9c4dd681486de5a4443a
Submitter: Zuul
Branch:    master

commit 1a0179f7ab9b9c7f07ce9c4dd681486de5a4443a
Author: Radoslav Gerganov <rgerganov@xxxxxxxxxx>
Date:   Mon Apr 30 13:18:26 2018 +0300

    Handle PortNotFoundClient exception when getting ports
    
    There could be a race between deleting an instance and retrieving its
    port groups from Neutron. In this case PortNotFoundClient is raised and
    it can be safely ignored.
    
    Change-Id: I31c9ea8628c6f3985f8e9118d9687bbfb8789b68
    Closes-Bug: #1767058


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1767058

Title:
  Rally test boot_and_delete_server_with_secgroups fails

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  When running rally scenarios the test fails with PortNotFoundClient exception.
  From the logs we see the following:

  Failure:
  -------

  ▼	11	GetResourceFailure	Failed to get the resource <Server: s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-a9471015-e72e-42cf-acc3-8b361415d91e)
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line 71, in _run_scenario_once
      getattr(scenario_inst, method_name)(**scenario_kwargs)
    File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/security_group.py", line 167, in run
      self._delete_server(server)
    File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 451, in _delete_server
      check_interval=CONF.openstack.nova_server_delete_poll_interval
    File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 214, in wait_for_status
      resource = update_resource(resource)
    File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 80, in _get_from_manager
      raise exceptions.GetResourceFailure(resource=resource, err=e)
  GetResourceFailure: Failed to get the resource <Server: s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-a9471015-e72e-42cf-acc3-8b361415d91e)


  Neutron:
  -------
  port-creation:
  2018-04-22 09:18:31.984 32139 DEBUG vmware_nsxlib.v3.client [req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] REST call: POST api/v1/logical-ports. Headers: {'X-NSX-EUSER': u'f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default', 'Content-Type': 'application/json', 'X-NSX-EREQID': 'req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b', 'Accept': 'application/json'}. Body: {"address_bindings": [{"ip_address": "100.1.124.13", "mac_address": "fa:16:3e:d4:3a:8f"}], "admin_state": "UP", "attachment": null, "description": "", "display_name": "", "logical_switch_id": "dfc6fa69-6df7-44db-ba43-68152104103f", "switching_profile_ids": [{"key": "SpoofGuardSwitchingProfile", "value": "0bb5dd95-1e3d-464e-a9d5-bd040bd91d9d"}], "tags": [{"scope": "os-neutron-port-id", "tag": "1954b052-bc2c-48ba-aa24-e8c96fac1723"}, {"scope": "os-project-id", "tag": "c87b5a2bc4e649069e9cfa4bbc874663"}, {"scope": "os-project-name", "tag": "c_rally_41596539_tkcSTCuv"}, {"scope": "os-api-version", "tag": "12.0.0.dev8294691"}, {"scope": "os-security-group", "tag": "a4d155d5-285a-4e9a-8982-04f6a242d3ed"}, {"scope": "os-security-group", "tag": "OS-Default-Section"}]} _rest_call /usr/lib/python2.7/dist-packages/vmware_nsxlib/v3/client.py:201
  .
  .
  2018-04-22 09:18:32.278 32139 INFO neutron.wsgi [req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] 10.127.108.239,10.127.108.237 "POST /v2.0/ports HTTP/1.1" status: 201  len: 983 time: 2.8035491

  port-deletion:

  2018-04-22 09:18:55.552 32135 INFO neutron.wsgi [req-774309da-
  6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b
  c87b5a2bc4e649069e9cfa4bbc874663 - default default]
  10.127.108.239,10.127.108.237 "DELETE /v2.0/ports/1954b052-bc2c-48ba-
  aa24-e8c96fac1723 HTTP/1.1" status: 204  len: 168 time: 2.4448841

  Nova:
  ----
  2018-04-22 09:18:55.539 19221 DEBUG oslo_concurrency.lockutils [req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f 69fad6d3240645b5854b52bca7e9b85e - default default] Lock "9ab78fdd-5798-4426-baf2-2706d0d69345" acqu
  ired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:276
  2018-04-22 09:18:55.541 19221 DEBUG oslo_concurrency.lockutils [req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f 69fad6d3240645b5854b52bca7e9b85e - default default] Lock "9ab78fdd-5798-4426-baf2-2706d0d69345" rele
  ased by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.002s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:288
  2018-04-22 09:18:55.662 19221 DEBUG nova.policy [req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f 69fad6d3240645b5854b52bca7e9b85e - default default] Policy check for os_compute_api:os-extended-server-attributes f
  ailed with credentials {'service_roles': [], 'user_id': u'8d478ca1fa274601a50f8d2a38c5e70f', 'roles': [u'_member_'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'servi
  ce_project_domain_id': None, 'is_admin_project': False, 'is_admin': False, 'project_id': u'69fad6d3240645b5854b52bca7e9b85e', 'project_domain_id': u'default'} authorize /usr/lib/python2.7/dist-packages/nova/policy.py:168
  2018-04-22 09:18:55.774 19225 DEBUG neutronclient.v2_0.client [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] Error message: {"NeutronError": {"message": "Port
   1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.", "type": "PortNotFound", "detail": ""}} _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] Caught error: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723
  could not be found.
  Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']: PortNotFoundClient: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
  Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Traceback (most recent call last):
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 82, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return req.get_response(self.application)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 92, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     self._log_req(req, res, start)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     self.force_reraise()
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     six.reraise(self.type_, self.value, self.tb)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 87, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     res = req.get_response(self.application)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return resp(environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 112, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return request.get_response(self.application)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 337, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     response = req.get_response(self._app)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     application, catch_exc_info=False)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return resp(environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return resp(environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     response = self.app(environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return resp(environ, start_response)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 580, in __call__
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     content_type, body, accept)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 654, in _process_stack
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     request, action_args)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 526, in process_extensions
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     **action_args)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py", line 534, in show
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     return self._show(req, resp_obj)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py", line 530, in _show
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     self._extend_servers(req, [resp_obj.obj['server']])
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py", line 512, in _extend_servers
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     servers))
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 376, in get_instances_security_groups_bindings
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 335, in _get_ports_from_server_list
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in list_ports
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     **_params)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 369, in list
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     for r in self._pagination(collection, path, **params):
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 384, in _pagination
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     res = self.get(path, params=params)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in get
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     headers=headers, params=params)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     headers=headers, params=params)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 294, in do_request
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     self._handle_fault_response(status_code, replybody, resp)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     ret = obj(*args, **kwargs)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     exception_handler_v20(status_code, error_body)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack     request_ids=request_ids)
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
  2018-04-22 09:18:55.819 19225 INFO nova.api.openstack [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50 returned with HTTP 500
  2018-04-22 09:18:55.822 19225 DEBUG nova.api.openstack.wsgi [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1064

  
  At a deeper looks we see:
  After looking more into the logs and the flow, here are the relevant log lines that explain the issue, and indicate this is an exception not being handled in Nova (see here https://github.com/openstack/nova/blob/master/nova/network/security_group/neutron_driver.py#L378), note the timestamps:

  Port ID: 1954b052-bc2c-48ba-aa24-e8c96fac1723
  Nova instance UUID: 3e714b48-5193-4b13-b6c9-3374488b2e50

  Neutron logs
  =============
  Port deletion:
  2018-04-22 09:18:55.552 32135 INFO neutron.wsgi [req-774309da-6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] 10.127.108.239,10.127.108.237 "DELETE /v2.0/ports/1954b052-bc2c-48ba-aa24-e8c96fac1723 HTTP/1.1" status: 204  len: 168 time: 2.4448841

  
  GET ports failure:
  2018-04-22 09:18:55.770 31303 INFO neutron.wsgi [req-35436010-4faa-42de-ab84-a783f48b4f92 f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] 10.127.108.236,10.127.108.237 "GET /v2.0/ports?device_id=3e714b48-5193-4b13-b6c9-3374488b2e50 HTTP/1.1" status: 404  len: 333 time: 0.5505300

  Nova log
  ========
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
  2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
  2018-04-22 09:18:55.819 19225 INFO nova.api.openstack [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50 returned with HTTP 500

  
  If there is a exception in self._get_secgroups_from_port_list should the security groups returned be []? Or a better exception raised. This is due to a race in the request in the retrieing the security groups and the deletion of the instance. The instance does not exist at this stage...

  The problem is discovered in stable queens

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1767058/+subscriptions
References

[Bug 1767058] [NEW] Rally test boot_and_delete_server_with_secgroups fails
From: Gary Kotton, 2018-04-26