yahoo-eng-team team mailing list archive

[Jakub Libosvar <libosvar@xxxxxxxxxx>]

Public bug reported:

We have a downstream failure of
neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest.test_duplicate_policy_error
probably because http timeout is different on RHEL based boxes, set 120
seconds. The reason why it started failing in Stein is the recent bump
in the time requiring to complete the retry DB mechanism:
https://review.opendev.org/#/c/583527/5 The patch increases the required
time for a reply about resource already existing to more than 160
seconds.

That is because Neutron server retries for every exception about existing entry from the DB layer:
https://opendev.org/openstack/neutron-lib/src/branch/master/neutron_lib/db/api.py#L119

Steps to reproduce:
net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }')
openstack network rbac create --type network --action access_as_shared --target-project admin $net_id
openstack network rbac create --type network --action access_as_shared --target-project admin $net_id

( yes, it's the same command twice)

I don't understand which race scenario the retry mechanism for resource
create it tries to solve. However, I can think of a race scenario it
introduces:

$ openstack network rbac delete 8a00a24e-182a-4e5e-8694-35e66635b581
$ openstack network delete $net_id
$ net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }')
$ rbac_id=$(openstack network rbac create --type network --action access_as_shared --target-project admin $net_id | awk '/ id /{ print $4 }')
$ openstack network rbac create --type network --action access_as_shared --target-project admin $net_id &
[1] 31383
$ sleep 10
$ openstack network rbac delete $rbac_id
$ fg
openstack network rbac create --type network --action access_as_shared --target-project admin $net_id
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                                                      |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action            | access_as_shared                                                                                                                                                                           |
| id                | 13c5b655-e1b5-4c72-a8af-1ed2f9ddcf89                                                                                                                                                       |
| location          | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': 'cdf84b19b71249ffaffde6627d06da12', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) |
| name              | None                                                                                                                                                                                       |
| object_id         | 618108b7-d191-4459-b18a-30b7a65be005                                                                                                                                                       |
| object_type       | network                                                                                                                                                                                    |
| project_id        | cdf84b19b71249ffaffde6627d06da12                                                                                                                                                           |
| target_project_id | cdf84b19b71249ffaffde6627d06da12                                                                                                                                                           |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The result should be that second creation of existing resource should
fail and there should not exist any rbac policy. However, the second
creation succeeded and there does exist the policy, that should have
been deleted.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1831647

Title:
  Creation of existing resource takes too much time or fails

Status in neutron:
  New

Bug description:
  We have a downstream failure of
  neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest.test_duplicate_policy_error
  probably because http timeout is different on RHEL based boxes, set
  120 seconds. The reason why it started failing in Stein is the recent
  bump in the time requiring to complete the retry DB mechanism:
  https://review.opendev.org/#/c/583527/5 The patch increases the
  required time for a reply about resource already existing to more than
  160 seconds.

  That is because Neutron server retries for every exception about existing entry from the DB layer:
  https://opendev.org/openstack/neutron-lib/src/branch/master/neutron_lib/db/api.py#L119

  Steps to reproduce:
  net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }')
  openstack network rbac create --type network --action access_as_shared --target-project admin $net_id
  openstack network rbac create --type network --action access_as_shared --target-project admin $net_id

  ( yes, it's the same command twice)

  I don't understand which race scenario the retry mechanism for
  resource create it tries to solve. However, I can think of a race
  scenario it introduces:

  $ openstack network rbac delete 8a00a24e-182a-4e5e-8694-35e66635b581
  $ openstack network delete $net_id
  $ net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }')
  $ rbac_id=$(openstack network rbac create --type network --action access_as_shared --target-project admin $net_id | awk '/ id /{ print $4 }')
  $ openstack network rbac create --type network --action access_as_shared --target-project admin $net_id &
  [1] 31383
  $ sleep 10
  $ openstack network rbac delete $rbac_id
  $ fg
  openstack network rbac create --type network --action access_as_shared --target-project admin $net_id
  +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field             | Value                                                                                                                                                                                      |
  +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | action            | access_as_shared                                                                                                                                                                           |
  | id                | 13c5b655-e1b5-4c72-a8af-1ed2f9ddcf89                                                                                                                                                       |
  | location          | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': 'cdf84b19b71249ffaffde6627d06da12', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) |
  | name              | None                                                                                                                                                                                       |
  | object_id         | 618108b7-d191-4459-b18a-30b7a65be005                                                                                                                                                       |
  | object_type       | network                                                                                                                                                                                    |
  | project_id        | cdf84b19b71249ffaffde6627d06da12                                                                                                                                                           |
  | target_project_id | cdf84b19b71249ffaffde6627d06da12                                                                                                                                                           |
  +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

  The result should be that second creation of existing resource should
  fail and there should not exist any rbac policy. However, the second
  creation succeeded and there does exist the policy, that should have
  been deleted.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1831647/+subscriptions