yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1491222] Re: Booting instance cause infinite recursion in the case of enough user quota and unlimited project quota

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Fri, 16 Jun 2017 12:18:07 -0000
Reply-to: Bug 1491222 <1491222@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova/liberty
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1491222

Title:
  Booting instance cause infinite recursion in the case of enough user
  quota and unlimited project quota

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) liberty series:
  Fix Released

Bug description:
  1. code base

  $ git log -1
  commit e041f3ee625b2737a1788304916589e3cc74db3d
  Merge: c310255 bef481b
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Tue Sep 1 21:53:31 2015 +0000

      Merge "Fix three typos on nova/pci directory"

  2. nova-api.log file:

  2015-09-01 18:03:32.490 DEBUG nova.db.sqlalchemy.api [req-b506dbaa-7157-4001-a79d-af1c9d642aa4 admin admin] Request is over project quota for resource "instances". Project limit: -1, delta: 1, current total project usage: 0 _calculate_overquota /opt/stack/nova/nova/db/sqlalchemy/api.py:3497
  ........
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     min_count, allowed)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     min_count, allowed)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     min_count, allowed)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     min_count, allowed)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 427, in _check_num_instances_quota
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     min_count, allowed)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 401, in _check_num_instances_quota
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     cores=req_cores, ram=req_ram)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return fn(self, *args, **kwargs)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/objects/quotas.py", line 92, in reserve
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     **deltas)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/quota.py", line 1343, in reserve
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     user_id=user_id)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/quota.py", line 569, in reserve
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     project_quotas=project_quotas)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/quota.py", line 406, in _get_quotas
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     project_quotas=project_quotas)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/quota.py", line 246, in get_user_quotas
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     user_id)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/db/api.py", line 1032, in quota_get_all_by_project_and_user
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return IMPL.quota_get_all_by_project_and_user(context, project_id, user_id)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 216, in wrapper
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return f(*args, **kwargs)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 3128, in quota_get_all_by_project_and_user
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     filter_by(user_id=user_id).\
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2320, in all
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return list(self)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return self._execute_and_instances(context)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     result = conn.execute(querycontext.statement, self._params)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return meth(self, multiparams, params)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return connection._execute_clauseelement(self, multiparams, params)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 819, in _execute_clauseelement
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     inline=len(distilled_params) > 1)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "<string>", line 1, in <lambda>
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 493, in compile
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return self._compiler(dialect, bind=bind, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 499, in _compiler
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return dialect.statement_compiler(dialect, self, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 395, in __init__
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     Compiled.__init__(self, dialect, statement, **kwargs)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 199, in __init__
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     self.string = self.process(self.statement, **compile_kwargs)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 222, in process
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return obj._compiler_dispatch(self, **kwargs)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 80, in _compiler_dispatch
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return meth(self, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1544, in visit_select
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     t = select._whereclause._compiler_dispatch(self, **kwargs)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return meth(self, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in visit_clauselist
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     for c in clauselist.clauses)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 659, in <genexpr>
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     s for s in
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in <genexpr>
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     for c in clauselist.clauses)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return meth(self, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in visit_clauselist
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     for c in clauselist.clauses)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 659, in <genexpr>
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     s for s in
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 662, in <genexpr>
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     for c in clauselist.clauses)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return meth(self, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 847, in visit_binary
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return self._generate_generic_binary(binary, opstring, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 864, in _generate_generic_binary
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     binary.right._compiler_dispatch(self, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 92, in _compiler_dispatch
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return meth(self, **kw)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 993, in visit_bindparam
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     name = self._truncate_bindparam(bindparam)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1049, in _truncate_bindparam
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     bind_name = self._truncated_identifier("bindparam", bind_name)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1060, in _truncated_identifier
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     anonname = name.apply_map(self.anon_map)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 3331, in apply_map
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return self % map_
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/_collections.py", line 680, in __missing__
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     self[key] = val = self.creator(key)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py", line 1079, in _process_anon
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack     return derived + "_" + str(anonymous_counter)
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack RuntimeError: maximum recursion depth exceeded while getting the str of an object
  2015-09-01 18:05:34.260 5619 ERROR nova.api.openstack
  2015-09-01 18:05:34.310 INFO nova.api.openstack [req-82c1beaa-d618-4ceb-8941-105df8720ad1 admin admin] http://localhost:8774/v2/4ffccaf0c84a44a29799b7bacab52334/servers returned with HTTP 500

  3. Reproduce steps:

  #Step 1: set project quota to unlimited
  $ nova quota-update --instances -1 $tenant

  #Step 2: set user quota to 1 (current usage 0)
  $ nova quota-update --user $tenantUser --instances 1 $tenant

  #Step 3: boot instance
  $ nova boot --image cirros-0.3.2-x86_64-disk --nic net-id=84140e3f-f20d-4ec5-9eac-d455dba7c569 --flavor m1.tiny chenrui_vm
  ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-82c1beaa-d618-4ceb-8941-105df8720ad1)

  Expected result:
  * booting instance success

  Actual result:
  * booting instance failed

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1491222/+subscriptions
References

[Bug 1491222] [NEW] Booting instance cause infinite recursion in the case of enough user quota and unlimited project quota
From: Rui Chen, 2015-09-02