yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: neutron
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1374044

Title:
  _make_subnet_dict lazy loads not required attibutes causing high
  number of not required sql query

Status in OpenStack Neutron (virtual network service):
  Fix Released

Bug description:
  The get_active_networks_info rpc call causes high number of sql query.
  For example the following query

  SELECT subnetroutes.destination AS subnetroutes_destination, subnetroutes.nexthop AS subnetroutes_nexthop, subnetroutes.subnet_id AS subnetroutes_subnet_id 
  FROM subnetroutes 
  WHERE %s = subnetroutes.subnet_id

  was used on this trace:

    File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
      func(*args, **kwargs)
    File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 462, in _process_data
      **args)
    File "/opt/stack/new/neutron/neutron/common/rpc.py", line 45, in dispatch
      neutron_ctxt, version, method, namespace, **kwargs)
    File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
      result = getattr(proxyobj, method)(ctxt, **kwargs)
    File "/opt/stack/new/neutron/neutron/db/dhcp_rpc_base.py", line 92, in get_active_networks_info
      networks = self._get_active_networks(context, **kwargs)
    File "/opt/stack/new/neutron/neutron/db/dhcp_rpc_base.py", line 42, in _get_active_networks
      plugin.auto_schedule_networks(context, host)
    File "/opt/stack/new/neutron/neutron/db/agentschedulers_db.py", line 211, in auto_schedule_networks
      self.network_scheduler.auto_schedule_networks(self, context, host)
    File "/opt/stack/new/neutron/neutron/scheduler/dhcp_agent_scheduler.py", line 114, in auto_schedule_networks
      subnets = plugin.get_subnets(context, fields=fields)
    File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1333, in get_subnets
      page_reverse=page_reverse)
    File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 209, in _get_collection
      items = [dict_func(c, fields) for c in query]
    File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 931, in _make_subnet_dict
      for route in subnet['routes']],
    File "/opt/stack/new/neutron/neutron/openstack/common/db/sqlalchemy/models.py", line 57, in __getitem__
      return getattr(self, key)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/attributes.py", line 237, in __get__
      return self.impl.get(instance_state(instance), dict_)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/attributes.py", line 590, in get
      value = self.callable_(state, passive)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/strategies.py", line 529, in _load_for_state
      return self._emit_lazyload(session, state, ident_key, passive)
    File "<string>", line 1, in <lambda>
    File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/strategies.py", line 598, in _emit_lazyload
      result = q.all()
    File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2363, in all
      return list(self)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2480, in __iter__
      return self._execute_and_instances(context)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2495, in _execute_and_instances
      result = conn.execute(querycontext.statement, self._params)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 730, in execute
      return meth(self, multiparams, params)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
      return connection._execute_clauseelement(self, multiparams, params)
    File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 827, in _execute_clauseelement
      compiled_sql, distilled_params
    File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 913, in _execute_context
      tb = str(traceback.format_stack())

  1. dhcp_agent_scheduler.py explicitly specifies he is interested only in two fields ['network_id', 'enable_dhcp']
  https://github.com/openstack/neutron/blob/ee4f94e32c8422eb5785f0bb1eb39b94b4e9b064/neutron/scheduler/dhcp_agent_scheduler.py#L103
  2. _get_collection makes lazy query (not fetching every related data)
  3.  make_subnet_dict  forces the ORM to post load not required filed
  https://github.com/openstack/neutron/blob/ee4f94e32c8422eb5785f0bb1eb39b94b4e9b064/neutron/db/db_base_plugin_v2.py#L830
  4. The self._fields  drops the lazy fetched fields.

  So the Db api provides the interface for efficient selective DB load,
  but at the and it does one of the most inefficient thing is doable on
  listing. Issues new SQL SELECt statements  per row/object.

  Looks like all function does similar thing which uses the self._field method.
  The make dict methods MUST NOT try to fetch the not requested fields from the data object at all.
  In this case the post _filter method also a not needed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1374044/+subscriptions