yahoo-eng-team team mailing list archive

[Nate Johnston <1843924@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When performing bulk actions, one area that is still very much forced
into the singleton method of processing is the resource_extend
framework, where extensions can register to extend object data.  This
RFE is to propose the changes needed to make resource_extend work in a
bulk context.

The details of the proposal are as follows:

1.) Add an optional additional argument "bulk" to
@resource_extend.extends that would default to False.  If it is True
this signals that the function being decorated can support bulk
extensions.  Normally the arguments to such a function are in the form
(resource, db_model); if it is operating in bulk mode then the first
argument would be treated as [resource, model] in an array, and the
second argument would be ignored.  This can be tested for inside the
function by checking if the first argument is an array.

2.) Create a new resource_extend.apply_funcs_bulk function that would
act like the current resource_extend.apply_funcs but would take an array
of object data instead of a single object resource and db model object.
This new function would consult the "bulk" attribute on registered
extender functions and would pass in the array of object data to a bulk
function, but if the function was bulk=False then it would loop through
as it does now.

3.) This would require a revamp of the _resource_extend_functions data
structure in neutron-lib/db/resource_extends.py.  Currently it is a dict
with the resource type as a string key, and the value is an array of
functions to be applied.  This would need to be changed to take
advantage of the bulk attribute.  I am not sure how best to do that at
this point.

One scenario where this would help is with bulk port creation, because
resource extender functions could implement more efficient commits of
their various data.  Another example is a bug from when I was last bug
deputy [1], where a user is complaining of slow responses due to large
quantities of SQL queries when using large numbers of trunk ports.  I
think that with a bulk resource extend the trunk port extender could be
implemented as qa single SQL query that fetches information for all
affected ports, thus significantly increasing performance.

Please let me know what you think of this idea.  I raised it as an RFE
because it requires significant enough modification to an element of
Neutron that is both foundational and that not many people may have
dealt with directly that an open discussion of the merits and
implementation would be a positive contribution.

[1] https://bugs.launchpad.net/neutron/+bug/1842150

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1843924

Title:
  [RFE] Create optional bulk resource_extend

Status in neutron:
  New

Bug description:
  When performing bulk actions, one area that is still very much forced
  into the singleton method of processing is the resource_extend
  framework, where extensions can register to extend object data.  This
  RFE is to propose the changes needed to make resource_extend work in a
  bulk context.

  The details of the proposal are as follows:

  1.) Add an optional additional argument "bulk" to
  @resource_extend.extends that would default to False.  If it is True
  this signals that the function being decorated can support bulk
  extensions.  Normally the arguments to such a function are in the form
  (resource, db_model); if it is operating in bulk mode then the first
  argument would be treated as [resource, model] in an array, and the
  second argument would be ignored.  This can be tested for inside the
  function by checking if the first argument is an array.

  2.) Create a new resource_extend.apply_funcs_bulk function that would
  act like the current resource_extend.apply_funcs but would take an
  array of object data instead of a single object resource and db model
  object.  This new function would consult the "bulk" attribute on
  registered extender functions and would pass in the array of object
  data to a bulk function, but if the function was bulk=False then it
  would loop through as it does now.

  3.) This would require a revamp of the _resource_extend_functions data
  structure in neutron-lib/db/resource_extends.py.  Currently it is a
  dict with the resource type as a string key, and the value is an array
  of functions to be applied.  This would need to be changed to take
  advantage of the bulk attribute.  I am not sure how best to do that at
  this point.

  One scenario where this would help is with bulk port creation, because
  resource extender functions could implement more efficient commits of
  their various data.  Another example is a bug from when I was last bug
  deputy [1], where a user is complaining of slow responses due to large
  quantities of SQL queries when using large numbers of trunk ports.  I
  think that with a bulk resource extend the trunk port extender could
  be implemented as qa single SQL query that fetches information for all
  affected ports, thus significantly increasing performance.

  Please let me know what you think of this idea.  I raised it as an RFE
  because it requires significant enough modification to an element of
  Neutron that is both foundational and that not many people may have
  dealt with directly that an open discussion of the merits and
  implementation would be a positive contribution.

  [1] https://bugs.launchpad.net/neutron/+bug/1842150

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1843924/+subscriptions