yahoo-eng-team team mailing list archive

[Arnaud Morin <1992950@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Context
=======
OpenStack Stein (but master seems affected by this as well).
OVS based deployment.
L3 routers in DVR and HA mode.
One big public "external/public" network (with subnets like /21 or /22) used by instances and router external gateways.

Problem description
===================
When adding a gateway on a router in HA+DVR, neutron api may send a lot of RPC messages toward L3 agents, depending on the size of the subnet used for the gateway.

How to reproduce
================
Add a gateway on a router:

$ openstack router set --external-gateway Ext-Net router-arnaud

On neutron server, in logs (in DEBUG):

Notify agent at l3_agent.hostxyz

We see this line for all l3 agents having a port in Ext-Net subnet
(which can be huge, like 1k).

Then, all agents are doing another RPC call (sync_routers) which is ending on neutron-rpc with this log line:
Sync routers for ids [abc]

Behing the Sync router, a big SQL request is done [1]

When 1k requests like this are done, on each router update, the database
is killed by too much SQL requests to do.


[1] https://github.com/openstack/neutron/blob/stable/stein/neutron/db/l3_dvrscheduler_db.py#L363

** Affects: neutron
     Importance: Undecided
     Assignee: Arnaud Morin (arnaud-morin)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Arnaud Morin (arnaud-morin)

** Summary changed:

- [scale] Router update is killing database
+ [scale] Setting a gateway on router is killing database

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1992950

Title:
  [scale] Setting a gateway on router is killing database

Status in neutron:
  In Progress

Bug description:
  Context
  =======
  OpenStack Stein (but master seems affected by this as well).
  OVS based deployment.
  L3 routers in DVR and HA mode.
  One big public "external/public" network (with subnets like /21 or /22) used by instances and router external gateways.

  Problem description
  ===================
  When adding a gateway on a router in HA+DVR, neutron api may send a lot of RPC messages toward L3 agents, depending on the size of the subnet used for the gateway.

  How to reproduce
  ================
  Add a gateway on a router:

  $ openstack router set --external-gateway Ext-Net router-arnaud

  On neutron server, in logs (in DEBUG):

  Notify agent at l3_agent.hostxyz

  We see this line for all l3 agents having a port in Ext-Net subnet
  (which can be huge, like 1k).

  Then, all agents are doing another RPC call (sync_routers) which is ending on neutron-rpc with this log line:
  Sync routers for ids [abc]

  Behing the Sync router, a big SQL request is done [1]

  When 1k requests like this are done, on each router update, the
  database is killed by too much SQL requests to do.

  
  [1] https://github.com/openstack/neutron/blob/stable/stein/neutron/db/l3_dvrscheduler_db.py#L363

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1992950/+subscriptions