yahoo-eng-team team mailing list archive

[OpenStack Infra <1614452@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/357052
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4bdab5cf1da333cf4e7aaf893e14b094fc5fad61
Submitter: Jenkins
Branch:    master

commit 4bdab5cf1da333cf4e7aaf893e14b094fc5fad61
Author: Oleg Bondarev <obondarev@xxxxxxxxxxxx>
Date:   Thu Aug 18 11:55:33 2016 +0300

    L3 DVR: use fanout when sending dvr arp table update
    
    Sending arp update to each l3 dvr agent one by one on every port
    creation is not scalable and causes serious performance degradation
    if router is hosted on lots of l3 dvr agents on compute nodes (see
    bug report). This increases port creation time and eventually leads
    to timeouts in Nova and VMs going to ERROR state.
    
    This patch changes notification to be fanout.
    The downside is that with fanout the arp notification will be sent to
    each l3 agent, even those not hosting the router. However such agents
    will just skip the notification if not hosting the router - this should
    be quite cheap.
    
    Closes-Bug: #1614452
    Change-Id: I1fb533d7804b131f709b790fc730ed7b97cb5499


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1614452

Title:
  Port create time grows at scale due to dvr arp update

Status in neutron:
  Fix Released

Bug description:
  Scale tests show that sometimes VMs are not able to spawn because of timeouts on port creation.
  Neutron server logs show that port creation time grows due to dvr arp table updates being sent to each l3 dvr agent hosting the router one by one - this takes > 90% of time: http://paste.openstack.org/show/560761/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1614452/+subscriptions