yahoo-eng-team team mailing list archive

[Matthew Treinish <mtreinish@xxxxxxxxxx>]

I actually think the analysis in this bug is a bit off, looking at the
tests in question there is a bug in
tempest.api.compute.test_tenant_networks in that we should tell it to
create a network with set_network_resources() in the setupClass.

The fact its passing is actually a bug somewhere in neutron. It's a
break in the isolation between tenants that the network list from that
test, which is guaranteed to be using a different tenant, is showing a
network from a different test class. The run order and concurrency is
only what's bringing this to light. (also to complete your list we also
pre-seed the scheduler with a sampling of previous runs which effects
the grouping and therefore the run order too) However, this is not the
actual cause of the bug, it's what's letting this pass at all
incorrectly.

But, there is a real bug, and arguably a serious one with potential sec
implications (depending on how severe it actually is) I don't think a
skip is warranted at this point until we get to the bottom of why a a
user in a random project can see the unshared networks from a different
tenant.

For a little evidence, lets look at the api logs from tempest:

1. The network is created by the neutron external networks test:

http://logs.openstack.org/98/219498/4/gate/gate-tempest-dsvm-neutron-
full/5250d42/logs/tempest.txt.gz#_2015-09-03_13_38_22_285

Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: {"network": {"router:external": true}}

Response - Headers: {'date': 'Thu, 03 Sep 2015 13:38:22 GMT', 'status': '201', 'content-length': '365', 'content-type': 'application/json; charset=UTF-8', 'connection': 'close', 'x-openstack-request-id': 'req-0beeacd8-0f6d-45e8-8120-8a3409d009c7'}
        Body: {"network": {"status": "ACTIVE", "subnets": [], "name": "", "provider:physical_network": null, "admin_state_up": true, "tenant_id": "90ed0196a23c43e9a1ced3c8c9672443", "mtu": 0, "router:external": true, "shared": false, "port_security_enabled": true, "provider:network_type": "vxlan", "id": "e00bdc30-3104-46bb-b9ce-5df452a3577f", "provider:segmentation_id": 1050}}


it's an unshared external network. I'm pretty sure if shared == false
this should not be visible even through nova's network list.



2. A different user in a different tenant makes the network list call through nova:

http://logs.openstack.org/98/219498/4/gate/gate-tempest-dsvm-neutron-
full/5250d42/logs/tempest.txt.gz#_2015-09-03_13_38_23_908

Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: None
 
Response - Headers: {'content-location': 'http://127.0.0.1:8774/v2.1/7dd5fcaa4fae499dbb045dd39e657be2/os-tenant-networks', 'date': 'Thu, 03 Sep 2015 13:38:23 GMT', 'status': '200', 'vary': 'X-OpenStack-Nova-API-Version', 'content-length': '174', 'content-type': 'application/json', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'x-compute-request-id': 'req-70b7fff1-b888-4e23-814e-7d36bda5e60b'}
        Body: {"networks": [{"cidr": "None", "id": "e00bdc30-3104-46bb-b9ce-5df452a3577f", "label": ""}, {"cidr": "None", "id": "11b0fe6d-424f-46b8-825b-19633ff09f62", "label": "public"}]}

which is the same unshared network from the other test. This races
against the delete from neutron which is causing the failure. But this
network should not be visible to the user.

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1468523

Title:
  tempest.api.compute.test_tenant_networks should be skipped when
  neutron is enabled

Status in neutron:
  New
Status in tempest:
  In Progress

Bug description:
  This test is a recipe for Neutron failures, since:

  1) it assumes there is always some tenant network
  2) it assumed it will be able do a GET on all the networks returned by the list operation

  Both #1 and #2 depend on the concurrent execution of other tests which created and destroy tenant networks.
  The test can fail because at that point in time there is no tenant network.
  Also, the test can fail because a tenant network gets deleted between the list and the show operations.

  The latter condition has begun to manifest in the gate [1]. Why only
  recently? that depends on the pattern of parallel test execution which
  depend on the number of tests and on the number of CPUs.

  Even if I'm not sure how representative is a test that lists objects from a REST endpoint and then fetches them one by one using the same endpoint, I think if it's there it must make some sense. However, when neutron is enabled it might be harmful.
  Also, neutron has a test which does pretty much the same thing in tempest.api.network. So perhaps it's not a bad idea to skip it.

  [1]
  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRGV0YWlsczoge3UnbWVzc2FnZSc6IHUnTmV0d29yayBub3QgZm91bmQnLCB1J2NvZGUnOiA0MDR9XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzUxODE1MDA3MjQsIm1vZGUiOiJzY29yZSIsImFuYWx5emVfZmllbGQiOiJidWlsZF9jaGFuZ2UifQ==

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1468523/+subscriptions