yahoo-eng-team team mailing list archive

[Slawek Kaplonski <1954763@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When ports are created in bulk, ML2 plugin tries to do everything for all ports in one single DB transaction. That means, that if IP allocation for one of the IPs will fail, whole create_port_bulk method will be retried so it will retry to do everything for all ports.
That may be very inefficient in some use cases, where many ports in bulk are created in same network.

I did simple test:
- create 2 networks with /24 subnet in each network,
- run 24 API requests to create 10 ports in bulk in each request - so in total 240 IPs from subnet will be allocated. All those requests were done in parallel.
- wait how long it will take to have all ports created.

I run my reproducer script 10 times as ipam module is working in some kind of random way so results may be different in various runs.
Results of that simple test are below:

Run	Execution time
1	00:01:37
2	00:02:06
3	00:02:08
4	00:02:14
5	00:01:58
6	00:02:37
7	00:01:59
8	00:02:01
9	00:02:39
10	00:01:59
AVG	00:02:07

The execution time here is in fact time of the execution of the longest
API request (all of them started together). So there was many retries
done internally in Neutron to allocate IP addresses for those ports and
client had to wait sometimes more than 150 seconds to get reply for
request.

** Affects: neutron
     Importance: Medium
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: api l3-ipam-dhcp loadimpact ml2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1954763

Title:
  Creating ports in bulks can be very slow due to many IPAM module
  conflicts

Status in neutron:
  Confirmed

Bug description:
  When ports are created in bulk, ML2 plugin tries to do everything for all ports in one single DB transaction. That means, that if IP allocation for one of the IPs will fail, whole create_port_bulk method will be retried so it will retry to do everything for all ports.
  That may be very inefficient in some use cases, where many ports in bulk are created in same network.

  I did simple test:
  - create 2 networks with /24 subnet in each network,
  - run 24 API requests to create 10 ports in bulk in each request - so in total 240 IPs from subnet will be allocated. All those requests were done in parallel.
  - wait how long it will take to have all ports created.

  I run my reproducer script 10 times as ipam module is working in some kind of random way so results may be different in various runs.
  Results of that simple test are below:

  Run	Execution time
  1	00:01:37
  2	00:02:06
  3	00:02:08
  4	00:02:14
  5	00:01:58
  6	00:02:37
  7	00:01:59
  8	00:02:01
  9	00:02:39
  10	00:01:59
  AVG	00:02:07

  The execution time here is in fact time of the execution of the
  longest API request (all of them started together). So there was many
  retries done internally in Neutron to allocate IP addresses for those
  ports and client had to wait sometimes more than 150 seconds to get
  reply for request.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1954763/+subscriptions