yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1329482] [NEW] test_quota.py QuotaReserveSqlAlchemyTestCase fails to clean up changes to QUOTA_SYNC_FUNCTIONS

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mike Bayer <mike_mp@xxxxxxxxxxxxxxxx>
Date: Thu, 12 Jun 2014 20:12:24 -0000
Reply-to: Bug 1329482 <1329482@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In nova/tests/test_quota.py -> QuotaReserveSqlAlchemyTestCase.setUp(),
alternate function definitions are swapped into
nova.db.sqlalchemy.api.QUOTA_SYNC_FUNCTIONS, however they are not
reverted in any corresponding tearDown() method.   I'm guessing this
isn't typically noticed as when using either nose or the "testr"-style
tools, test_quota.py is run well after other tests which rely on these
functions, such as those in nova/tests/api/ec2/test_cinder_cloud.py.
However, I've been using py.test which has a different natural test
ordering.  The issue can be seen using Nose by running test_cinder_cloud
after test_quota:

$ ../.venv/bin/nosetests -v nova/tests/test_quota.py nova/tests/api/ec2/test_cinder_cloud.py -x
nova.tests.test_quota.BaseResourceTestCase.test_no_flag ... ok
nova.tests.test_quota.BaseResourceTestCase.test_quota_no_project ... ok
nova.tests.test_quota.BaseResourceTestCase.test_quota_with_project ... ok
nova.tests.test_quota.BaseResourceTestCase.test_with_flag ... ok

    [ ... tests continue to run ... ]

nova.tests.test_quota.QuotaReserveSqlAlchemyTestCase.test_quota_reserve_until_refresh ... ok
nova.tests.api.ec2.test_cinder_cloud.CinderCloudTestCase.test_create_image ... /Users/classic/dev/redhat/.venv/lib/python2.7/site-packages/sqlalchemy/sql/default_comparator.py:33: SAWarning: The IN-predicate on "instances.uuid" was invoked with an empty sequence. This results in a contradiction, which nonetheless can be expensive to evaluate.  Consider alternative strategies for improved performance.
  return o[0](self, self.expr, op, *(other + o[1:]), **kwargs)
ERROR

======================================================================
ERROR: nova.tests.api.ec2.test_cinder_cloud.CinderCloudTestCase.test_create_image
----------------------------------------------------------------------
_StringException: pythonlogging:'': {{{
AUDIT [nova.service] Starting conductor node (version 2014.2)
INFO [nova.virt.driver] Loading compute driver 'nova.virt.fake.FakeDriver'
AUDIT [nova.service] Starting compute node (version 2014.2)
AUDIT [nova.compute.resource_tracker] Auditing locally available compute resources
AUDIT [nova.compute.resource_tracker] Free ram (MB): 7680
AUDIT [nova.compute.resource_tracker] Free disk (GB): 1028
AUDIT [nova.compute.resource_tracker] Free VCPUS: 1
AUDIT [nova.compute.resource_tracker] PCI stats: []
INFO [nova.compute.resource_tracker] Compute_service record created for 93851743013149aabf5b0a5492cef513:fake-mini
AUDIT [nova.service] Starting scheduler node (version 2014.2)
INFO [nova.network.driver] Loading network driver 'nova.network.linux_net'
AUDIT [nova.service] Starting network node (version 2014.2)
AUDIT [nova.service] Starting consoleauth node (version 2014.2)
INFO [nova.virt.driver] Loading compute driver 'nova.virt.fake.FakeDriver'
AUDIT [nova.service] Starting compute node (version 2014.2)
AUDIT [nova.compute.resource_tracker] Auditing locally available compute resources
AUDIT [nova.compute.resource_tracker] Free ram (MB): 7680
AUDIT [nova.compute.resource_tracker] Free disk (GB): 1028
AUDIT [nova.compute.resource_tracker] Free VCPUS: 1
AUDIT [nova.compute.resource_tracker] PCI stats: []
INFO [nova.compute.resource_tracker] Compute_service record created for d207a83c4c3f4b0ab622668b19210a10:fake-mini
WARNING [nova.service] Service killed that has no database entry
}}}

Traceback (most recent call last):
  File "/Users/classic/dev/redhat/nova/nova/tests/api/ec2/test_cinder_cloud.py", line 1019, in test_create_image
    ec2_instance_id = self._run_instance(**kwargs)
  File "/Users/classic/dev/redhat/nova/nova/tests/api/ec2/test_cinder_cloud.py", line 750, in _run_instance
    rv = self.cloud.run_instances(self.context, **kwargs)
  File "/Users/classic/dev/redhat/nova/nova/api/ec2/cloud.py", line 1352, in run_instances
    block_device_mapping=kwargs.get('block_device_mapping', {}))
  File "/Users/classic/dev/redhat/nova/nova/hooks.py", line 103, in inner
    rv = f(*args, **kwargs)
  File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 1338, in create
    legacy_bdm=legacy_bdm)
  File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 997, in _create_instance
    block_device_mapping)
  File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 818, in _provision_instances
    context, instance_type, min_count, max_count)
  File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 343, in _check_num_instances_quota
    cores=req_cores, ram=req_ram)
  File "/Users/classic/dev/redhat/nova/nova/quota.py", line 1301, in reserve
    user_id=user_id)
  File "/Users/classic/dev/redhat/nova/nova/quota.py", line 544, in reserve
    project_id=project_id, user_id=user_id)
  File "/Users/classic/dev/redhat/nova/nova/db/api.py", line 1143, in quota_reserve
    project_id=project_id, user_id=user_id)
  File "/Users/classic/dev/redhat/nova/nova/db/sqlalchemy/api.py", line 164, in wrapper
    return f(*args, **kwargs)
  File "/Users/classic/dev/redhat/nova/nova/db/sqlalchemy/api.py", line 202, in wrapped
    return f(*args, **kwargs)
  File "/Users/classic/dev/redhat/nova/nova/db/sqlalchemy/api.py", line 3130, in quota_reserve
    updates = sync(elevated, project_id, user_id, session)
  File "/Users/classic/dev/redhat/nova/nova/tests/test_quota.py", line 2101, in sync
    self.sync_called.add(res_name)
AttributeError: 'QuotaReserveSqlAlchemyTestCase' object has no attribute 'sync_called'

The symptom is cryptic here, but essentially the callables swapped in by
this test refer to "self" as a QuotaReserveSqlAlchemyTestCase object,
which has long since been torn down and no longer has the "sync_called"
set associated with it.

I'd love to use mock.patch() for this kind of thing, but for the moment
I'm going to submit a straightforward patch that restores
QUOTA_SYNC_FUNCTIONS.

** Affects: nova
     Importance: Undecided
     Assignee: Mike Bayer (zzzeek)
         Status: In Progress


** Tags: low-hanging-fruit

** Changed in: nova
     Assignee: (unassigned) => Mike Bayer (zzzeek)

** Changed in: nova
       Status: New => In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1329482

Title:
  test_quota.py QuotaReserveSqlAlchemyTestCase fails to clean up changes
  to QUOTA_SYNC_FUNCTIONS

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  In nova/tests/test_quota.py -> QuotaReserveSqlAlchemyTestCase.setUp(),
  alternate function definitions are swapped into
  nova.db.sqlalchemy.api.QUOTA_SYNC_FUNCTIONS, however they are not
  reverted in any corresponding tearDown() method.   I'm guessing this
  isn't typically noticed as when using either nose or the "testr"-style
  tools, test_quota.py is run well after other tests which rely on these
  functions, such as those in nova/tests/api/ec2/test_cinder_cloud.py.
  However, I've been using py.test which has a different natural test
  ordering.  The issue can be seen using Nose by running
  test_cinder_cloud after test_quota:

  $ ../.venv/bin/nosetests -v nova/tests/test_quota.py nova/tests/api/ec2/test_cinder_cloud.py -x
  nova.tests.test_quota.BaseResourceTestCase.test_no_flag ... ok
  nova.tests.test_quota.BaseResourceTestCase.test_quota_no_project ... ok
  nova.tests.test_quota.BaseResourceTestCase.test_quota_with_project ... ok
  nova.tests.test_quota.BaseResourceTestCase.test_with_flag ... ok

      [ ... tests continue to run ... ]

  nova.tests.test_quota.QuotaReserveSqlAlchemyTestCase.test_quota_reserve_until_refresh ... ok
  nova.tests.api.ec2.test_cinder_cloud.CinderCloudTestCase.test_create_image ... /Users/classic/dev/redhat/.venv/lib/python2.7/site-packages/sqlalchemy/sql/default_comparator.py:33: SAWarning: The IN-predicate on "instances.uuid" was invoked with an empty sequence. This results in a contradiction, which nonetheless can be expensive to evaluate.  Consider alternative strategies for improved performance.
    return o[0](self, self.expr, op, *(other + o[1:]), **kwargs)
  ERROR

  ======================================================================
  ERROR: nova.tests.api.ec2.test_cinder_cloud.CinderCloudTestCase.test_create_image
  ----------------------------------------------------------------------
  _StringException: pythonlogging:'': {{{
  AUDIT [nova.service] Starting conductor node (version 2014.2)
  INFO [nova.virt.driver] Loading compute driver 'nova.virt.fake.FakeDriver'
  AUDIT [nova.service] Starting compute node (version 2014.2)
  AUDIT [nova.compute.resource_tracker] Auditing locally available compute resources
  AUDIT [nova.compute.resource_tracker] Free ram (MB): 7680
  AUDIT [nova.compute.resource_tracker] Free disk (GB): 1028
  AUDIT [nova.compute.resource_tracker] Free VCPUS: 1
  AUDIT [nova.compute.resource_tracker] PCI stats: []
  INFO [nova.compute.resource_tracker] Compute_service record created for 93851743013149aabf5b0a5492cef513:fake-mini
  AUDIT [nova.service] Starting scheduler node (version 2014.2)
  INFO [nova.network.driver] Loading network driver 'nova.network.linux_net'
  AUDIT [nova.service] Starting network node (version 2014.2)
  AUDIT [nova.service] Starting consoleauth node (version 2014.2)
  INFO [nova.virt.driver] Loading compute driver 'nova.virt.fake.FakeDriver'
  AUDIT [nova.service] Starting compute node (version 2014.2)
  AUDIT [nova.compute.resource_tracker] Auditing locally available compute resources
  AUDIT [nova.compute.resource_tracker] Free ram (MB): 7680
  AUDIT [nova.compute.resource_tracker] Free disk (GB): 1028
  AUDIT [nova.compute.resource_tracker] Free VCPUS: 1
  AUDIT [nova.compute.resource_tracker] PCI stats: []
  INFO [nova.compute.resource_tracker] Compute_service record created for d207a83c4c3f4b0ab622668b19210a10:fake-mini
  WARNING [nova.service] Service killed that has no database entry
  }}}

  Traceback (most recent call last):
    File "/Users/classic/dev/redhat/nova/nova/tests/api/ec2/test_cinder_cloud.py", line 1019, in test_create_image
      ec2_instance_id = self._run_instance(**kwargs)
    File "/Users/classic/dev/redhat/nova/nova/tests/api/ec2/test_cinder_cloud.py", line 750, in _run_instance
      rv = self.cloud.run_instances(self.context, **kwargs)
    File "/Users/classic/dev/redhat/nova/nova/api/ec2/cloud.py", line 1352, in run_instances
      block_device_mapping=kwargs.get('block_device_mapping', {}))
    File "/Users/classic/dev/redhat/nova/nova/hooks.py", line 103, in inner
      rv = f(*args, **kwargs)
    File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 1338, in create
      legacy_bdm=legacy_bdm)
    File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 997, in _create_instance
      block_device_mapping)
    File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 818, in _provision_instances
      context, instance_type, min_count, max_count)
    File "/Users/classic/dev/redhat/nova/nova/compute/api.py", line 343, in _check_num_instances_quota
      cores=req_cores, ram=req_ram)
    File "/Users/classic/dev/redhat/nova/nova/quota.py", line 1301, in reserve
      user_id=user_id)
    File "/Users/classic/dev/redhat/nova/nova/quota.py", line 544, in reserve
      project_id=project_id, user_id=user_id)
    File "/Users/classic/dev/redhat/nova/nova/db/api.py", line 1143, in quota_reserve
      project_id=project_id, user_id=user_id)
    File "/Users/classic/dev/redhat/nova/nova/db/sqlalchemy/api.py", line 164, in wrapper
      return f(*args, **kwargs)
    File "/Users/classic/dev/redhat/nova/nova/db/sqlalchemy/api.py", line 202, in wrapped
      return f(*args, **kwargs)
    File "/Users/classic/dev/redhat/nova/nova/db/sqlalchemy/api.py", line 3130, in quota_reserve
      updates = sync(elevated, project_id, user_id, session)
    File "/Users/classic/dev/redhat/nova/nova/tests/test_quota.py", line 2101, in sync
      self.sync_called.add(res_name)
  AttributeError: 'QuotaReserveSqlAlchemyTestCase' object has no attribute 'sync_called'

  The symptom is cryptic here, but essentially the callables swapped in
  by this test refer to "self" as a QuotaReserveSqlAlchemyTestCase
  object, which has long since been torn down and no longer has the
  "sync_called" set associated with it.

  I'd love to use mock.patch() for this kind of thing, but for the
  moment I'm going to submit a straightforward patch that restores
  QUOTA_SYNC_FUNCTIONS.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1329482/+subscriptions
Follow ups

[Bug 1329482] Re: test_quota.py QuotaReserveSqlAlchemyTestCase fails to clean up changes to QUOTA_SYNC_FUNCTIONS
From: Russell Bryant, 2014-07-23
[Bug 1329482] [NEW] test_quota.py QuotaReserveSqlAlchemyTestCase fails to clean up changes to QUOTA_SYNC_FUNCTIONS
From: Mike Bayer, 2014-06-12
References

[Bug 1329482] [NEW] test_quota.py QuotaReserveSqlAlchemyTestCase fails to clean up changes to QUOTA_SYNC_FUNCTIONS
From: Mike Bayer, 2014-06-12