← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #24552

[Bug 1393399] [NEW] Cisco N1KV plugin doesn't process correctly SQLAlchemy exceptions

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Test tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_port failed when used N1KV plugin. As I see from neutron log, Cisco plugin incorrectly process exceptions in method _get_policy_profile_by_name (file 
"/opt/stack/neutron/neutron/plugins/cisco/db/n1kv_db_v2.py") it's leads to return code 500 by Neutron service.

Problem reproduced when creating ports and n1kv profile_id not found in database. According Neutron API, for this request allowed only next error codes:
- badRequest (400)
- unauthorized (401)
- forbidden (403)
- itemNotFound (404)
- macGenerationFailure (503)
- serviceUnavailable (503)


Test log:
2014-11-17 03:24:07,926 18176 DEBUG    [tempest.common.rest_client] Request (BulkNetworkOpsTestJSON:test_bulk_create_delete_port): 201 POST http://10.20.21.2:9696/v2.0/networks 0.406s
    Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"network": {"name": "test-network--197486"}}
    Response - Headers: {'status': '201', 'content-length': '301', 'connection': 'close', 'date': 'Mon, 17 Nov 2014 03:24:07 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-eca84ad6-9ba1-4f22-b668-018201e65660'}
        Body: {"network": {"status": "ACTIVE", "subnets": [], "name": "test-network--197486", "router:external": false, "tenant_id": "443d9ae88e094a5c9fa9daa36ac4d00f", "admin_state_up": true, "n1kv:profile_id": "2b8937d0-069c-461d-aedb-020ad0e2ed99", "shared": false, "id": "dc7b7044-d3ac-450d-9f73-d87b931170a1"}}
2014-11-17 03:24:08,335 18176 DEBUG    [tempest.common.rest_client] Request (BulkNetworkOpsTestJSON:test_bulk_create_delete_port): 201 POST http://10.20.21.2:9696/v2.0/networks 0.408s
    Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"network": {"name": "test-network--380129587"}}
    Response - Headers: {'status': '201', 'content-length': '304', 'connection': 'close', 'date': 'Mon, 17 Nov 2014 03:24:08 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-0f0ca9f6-82c0-4437-906e-005ae6b31377'}
        Body: {"network": {"status": "ACTIVE", "subnets": [], "name": "test-network--380129587", "router:external": false, "tenant_id": "443d9ae88e094a5c9fa9daa36ac4d00f", "admin_state_up": true, "n1kv:profile_id": "2b8937d0-069c-461d-aedb-020ad0e2ed99", "shared": false, "id": "381d122a-f55c-4c39-9eb9-17d5cd3ef818"}}
2014-11-17 03:24:08,390 18176 DEBUG    [tempest.common.rest_client] Request (BulkNetworkOpsTestJSON:test_bulk_create_delete_port): 500 POST http://10.20.21.2:9696/v2.0/ports 0.053s
    Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"ports": [{"network_id": "dc7b7044-d3ac-450d-9f73-d87b931170a1", "name": "port--769863538", "admin_state_up": true}, {"network_id": "381d122a-f55c-4c39-9eb9-17d5cd3ef818", "admin_state_up": false}]}
    Response - Headers: {'status': '500', 'content-length': '150', 'connection': 'close', 'date': 'Mon, 17 Nov 2014 03:24:08 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-7bfbe0d3-de6b-4450-b4ee-7ead1af3fa7f'}
        Body: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
}}}

Traceback (most recent call last):
  File "tempest/api/network/test_networks.py", line 463, in test_bulk_create_delete_port
    _, body = self.client.create_bulk_port(port_list)
  File "tempest/services/network/network_client_base.py", line 207, in create_bulk_port
    resp, body = self.post(uri, body)
  File "tempest/services/network/network_client_base.py", line 74, in post
    return self.rest_client.post(uri, body, headers)
  File "tempest/common/rest_client.py", line 234, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "tempest/common/rest_client.py", line 454, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 550, in _error_checker
    raise exceptions.ServerFault(message)
ServerFault: Got server fault
Details: Request Failed: internal server error while processing your request.
Traceback (most recent call last):
_StringException: Empty attachments:
  stderr
  stdout


Neutron log:
2014-11-17 10:15:03.365 ERROR neutron.db.db_base_plugin_v2 [req-c8f914ec-00a8-4734-9557-dd14bc00b1d2 BulkNetworkOpsTestJSON-1775368720 77347f40bf50412eb636412a20c64aaf] An exception occurred while creating the port:{'port': {'binding:host_id': <object object at 0x7f0be269c110>, u'name': u'port--1581734127', u'admin_state_up': True, u'network_id': u'fee9d7af-edec-4d31-aa27-b94f5216ac7e', 'tenant_id': u'77347f40bf50412eb636412a20c64aaf', 'binding:vnic_type': 'normal', 'device_owner': '', 'n1kv:profile_id': <object object at 0x7f0be269c110>, 'mac_address': <object object at 0x7f0be269c110>, 'binding:profile': <object object at 0x7f0be269c110>, 'fixed_ips': <object object at 0x7f0be269c110>, 'device_id': ''}}
2014-11-17 10:15:03.366 ERROR neutron.api.v2.resource [req-c8f914ec-00a8-4734-9557-dd14bc00b1d2 BulkNetworkOpsTestJSON-1775368720 77347f40bf50412eb636412a20c64aaf] create failed
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 81, in resource
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     result = method(request=request, **args)
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 431, in create
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     objs = obj_creator(request.context, body, **kwargs)
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1310, in create_port_bulk
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     return self._create_bulk('port', context, ports)
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 896, in _create_bulk
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     {'resource': resource, 'item': item})
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/openstack/common/excutils.py", line 82, in __exit__
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 889, in _create_bulk
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     objects.append(obj_creator(context, item))
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/models/virt_phy_sw_v2.py", line 235, in create_port
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     args)
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/models/virt_phy_sw_v2.py", line 124, in _invoke_plugin_per_device
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     return func(*args, **kwargs)
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/n1kv/n1kv_neutron_plugin.py", line 1122, in create_port
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     p_profile = self._get_policy_profile_by_name(p_profile_name)
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/db/n1kv_db_v2.py", line 1603, in _get_policy_profile_by_name
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     filter_by(name=name).one())
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2316, in one
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     raise orm_exc.NoResultFound("No row was found for one()")
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource NoResultFound: No row was found for one()
2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource

Devstack local.conf:

[[local|localrc]]
MYSQL_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,rabbit
disable_service horizon
disable_service n-net, q-agt, q-l3
enable_service mysql, q-svc, q-dhcp, q-meta, q-lbaas, neutron, tempest
VOLUME_BACKING_FILE_SIZE=2052M
Q_PLUGIN=cisco
declare -a Q_CISCO_PLUGIN_SUBPLUGINS=(n1kv)
Q_CISCO_PLUGIN_DEVSTACK_VSM=False
Q_CISCO_PLUGIN_VSM_IP=192.168.1.10
Q_CISCO_PLUGIN_VSM_USERNAME=admin
Q_CISCO_PLUGIN_VSM_PASSWORD=Admin12345
Q_CISCO_PLUGIN_UVEM_DEB_IMAGE=nexus_1000v_vem-12.04-5.2.1.SK1.2.1.29.S0-1.deb
Q_CISCO_PLUGIN_INTEGRATION_BRIDGE=br-int
Q_CISCO_PLUGIN_HOST_MGMT_INTF=eth0
PHYSICAL_NETWORK=test-physnet1
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
API_RATE_LIMIT=False
VERBOSE=True
DEBUG=True

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: cisco n1kv

** Tags added: cisco

** Tags added: n1kv

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1393399

Title:
  Cisco N1KV plugin doesn't process correctly SQLAlchemy exceptions

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Test tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_port failed when used N1KV plugin. As I see from neutron log, Cisco plugin incorrectly process exceptions in method _get_policy_profile_by_name (file 
  "/opt/stack/neutron/neutron/plugins/cisco/db/n1kv_db_v2.py") it's leads to return code 500 by Neutron service.

  Problem reproduced when creating ports and n1kv profile_id not found in database. According Neutron API, for this request allowed only next error codes:
  - badRequest (400)
  - unauthorized (401)
  - forbidden (403)
  - itemNotFound (404)
  - macGenerationFailure (503)
  - serviceUnavailable (503)

  
  Test log:
  2014-11-17 03:24:07,926 18176 DEBUG    [tempest.common.rest_client] Request (BulkNetworkOpsTestJSON:test_bulk_create_delete_port): 201 POST http://10.20.21.2:9696/v2.0/networks 0.406s
      Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: {"network": {"name": "test-network--197486"}}
      Response - Headers: {'status': '201', 'content-length': '301', 'connection': 'close', 'date': 'Mon, 17 Nov 2014 03:24:07 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-eca84ad6-9ba1-4f22-b668-018201e65660'}
          Body: {"network": {"status": "ACTIVE", "subnets": [], "name": "test-network--197486", "router:external": false, "tenant_id": "443d9ae88e094a5c9fa9daa36ac4d00f", "admin_state_up": true, "n1kv:profile_id": "2b8937d0-069c-461d-aedb-020ad0e2ed99", "shared": false, "id": "dc7b7044-d3ac-450d-9f73-d87b931170a1"}}
  2014-11-17 03:24:08,335 18176 DEBUG    [tempest.common.rest_client] Request (BulkNetworkOpsTestJSON:test_bulk_create_delete_port): 201 POST http://10.20.21.2:9696/v2.0/networks 0.408s
      Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: {"network": {"name": "test-network--380129587"}}
      Response - Headers: {'status': '201', 'content-length': '304', 'connection': 'close', 'date': 'Mon, 17 Nov 2014 03:24:08 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-0f0ca9f6-82c0-4437-906e-005ae6b31377'}
          Body: {"network": {"status": "ACTIVE", "subnets": [], "name": "test-network--380129587", "router:external": false, "tenant_id": "443d9ae88e094a5c9fa9daa36ac4d00f", "admin_state_up": true, "n1kv:profile_id": "2b8937d0-069c-461d-aedb-020ad0e2ed99", "shared": false, "id": "381d122a-f55c-4c39-9eb9-17d5cd3ef818"}}
  2014-11-17 03:24:08,390 18176 DEBUG    [tempest.common.rest_client] Request (BulkNetworkOpsTestJSON:test_bulk_create_delete_port): 500 POST http://10.20.21.2:9696/v2.0/ports 0.053s
      Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
          Body: {"ports": [{"network_id": "dc7b7044-d3ac-450d-9f73-d87b931170a1", "name": "port--769863538", "admin_state_up": true}, {"network_id": "381d122a-f55c-4c39-9eb9-17d5cd3ef818", "admin_state_up": false}]}
      Response - Headers: {'status': '500', 'content-length': '150', 'connection': 'close', 'date': 'Mon, 17 Nov 2014 03:24:08 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-7bfbe0d3-de6b-4450-b4ee-7ead1af3fa7f'}
          Body: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
  }}}

  Traceback (most recent call last):
    File "tempest/api/network/test_networks.py", line 463, in test_bulk_create_delete_port
      _, body = self.client.create_bulk_port(port_list)
    File "tempest/services/network/network_client_base.py", line 207, in create_bulk_port
      resp, body = self.post(uri, body)
    File "tempest/services/network/network_client_base.py", line 74, in post
      return self.rest_client.post(uri, body, headers)
    File "tempest/common/rest_client.py", line 234, in post
      return self.request('POST', url, extra_headers, headers, body)
    File "tempest/common/rest_client.py", line 454, in request
      resp, resp_body)
    File "tempest/common/rest_client.py", line 550, in _error_checker
      raise exceptions.ServerFault(message)
  ServerFault: Got server fault
  Details: Request Failed: internal server error while processing your request.
  Traceback (most recent call last):
  _StringException: Empty attachments:
    stderr
    stdout


  Neutron log:
  2014-11-17 10:15:03.365 ERROR neutron.db.db_base_plugin_v2 [req-c8f914ec-00a8-4734-9557-dd14bc00b1d2 BulkNetworkOpsTestJSON-1775368720 77347f40bf50412eb636412a20c64aaf] An exception occurred while creating the port:{'port': {'binding:host_id': <object object at 0x7f0be269c110>, u'name': u'port--1581734127', u'admin_state_up': True, u'network_id': u'fee9d7af-edec-4d31-aa27-b94f5216ac7e', 'tenant_id': u'77347f40bf50412eb636412a20c64aaf', 'binding:vnic_type': 'normal', 'device_owner': '', 'n1kv:profile_id': <object object at 0x7f0be269c110>, 'mac_address': <object object at 0x7f0be269c110>, 'binding:profile': <object object at 0x7f0be269c110>, 'fixed_ips': <object object at 0x7f0be269c110>, 'device_id': ''}}
  2014-11-17 10:15:03.366 ERROR neutron.api.v2.resource [req-c8f914ec-00a8-4734-9557-dd14bc00b1d2 BulkNetworkOpsTestJSON-1775368720 77347f40bf50412eb636412a20c64aaf] create failed
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource Traceback (most recent call last):
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 81, in resource
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     result = method(request=request, **args)
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 431, in create
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     objs = obj_creator(request.context, body, **kwargs)
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1310, in create_port_bulk
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     return self._create_bulk('port', context, ports)
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 896, in _create_bulk
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     {'resource': resource, 'item': item})
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/openstack/common/excutils.py", line 82, in __exit__
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 889, in _create_bulk
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     objects.append(obj_creator(context, item))
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/models/virt_phy_sw_v2.py", line 235, in create_port
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     args)
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/models/virt_phy_sw_v2.py", line 124, in _invoke_plugin_per_device
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     return func(*args, **kwargs)
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/n1kv/n1kv_neutron_plugin.py", line 1122, in create_port
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     p_profile = self._get_policy_profile_by_name(p_profile_name)
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/cisco/db/n1kv_db_v2.py", line 1603, in _get_policy_profile_by_name
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     filter_by(name=name).one())
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2316, in one
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource     raise orm_exc.NoResultFound("No row was found for one()")
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource NoResultFound: No row was found for one()
  2014-11-17 10:15:03.366 TRACE neutron.api.v2.resource

  Devstack local.conf:

  [[local|localrc]]
  MYSQL_PASSWORD=nova
  RABBIT_PASSWORD=nova
  SERVICE_TOKEN=nova
  SERVICE_PASSWORD=nova
  ADMIN_PASSWORD=nova
  ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,rabbit
  disable_service horizon
  disable_service n-net, q-agt, q-l3
  enable_service mysql, q-svc, q-dhcp, q-meta, q-lbaas, neutron, tempest
  VOLUME_BACKING_FILE_SIZE=2052M
  Q_PLUGIN=cisco
  declare -a Q_CISCO_PLUGIN_SUBPLUGINS=(n1kv)
  Q_CISCO_PLUGIN_DEVSTACK_VSM=False
  Q_CISCO_PLUGIN_VSM_IP=192.168.1.10
  Q_CISCO_PLUGIN_VSM_USERNAME=admin
  Q_CISCO_PLUGIN_VSM_PASSWORD=Admin12345
  Q_CISCO_PLUGIN_UVEM_DEB_IMAGE=nexus_1000v_vem-12.04-5.2.1.SK1.2.1.29.S0-1.deb
  Q_CISCO_PLUGIN_INTEGRATION_BRIDGE=br-int
  Q_CISCO_PLUGIN_HOST_MGMT_INTF=eth0
  PHYSICAL_NETWORK=test-physnet1
  LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
  API_RATE_LIMIT=False
  VERBOSE=True
  DEBUG=True

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1393399/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next