yahoo-eng-team team mailing list archive

[OpenStack Infra <1557640@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/293144
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ed7411fa1c4d6a26cea3a8737b3c29ce6ff64363
Submitter: Jenkins
Branch:    master

commit ed7411fa1c4d6a26cea3a8737b3c29ce6ff64363
Author: Stephen Eilert <stephen.eilert@xxxxxxx>
Date:   Tue Mar 15 13:54:23 2016 -0700

    Removes host file contents from DHCP agent logs
    
    The change I3ad7864eeb2f959549ed356a1e34fa18804395cc addressed a
    performance impact by removing logging calls from inside the loop that
    creates the hosts file in memory. However, it also introduced the
    dumping of said hosts file to the logs.
    
    On deployments with a very high number of ports(>5K), this increases the
    load of the node substantially. The problem is worse when a high number
    of port updates is received in a short amount of time.
    
    There is also the administrative burden caused by the sheer amount of
    data being logged.
    
    If the content of the host file is required for debugging purposes, it
    can be found at /opt/stack/data/neutron/dhcp/<net-id>/host
    
    This patch is trivial and only removes the logging of the file contents.
    The 'building'/'done building' logging calls may still provide useful
    information.
    
    Change-Id: Ie176ef123c194c22dca0967a6acfb9a48c959e6c
    Closes-Bug: 1557640


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1557640

Title:
  DHCP agent logging is causing performance issues

Status in neutron:
  Fix Released

Bug description:
  When a new port gets added or removed, the DHCP agent is currently
  dumping the entirety of the hosts file in the logs. This is a problem
  in some deployments we have encountered, some of which with upwards of
  10000 ports.

  Change-Id: I3ad7864eeb2f959549ed356a1e34fa18804395cc removed logging
  calls from inside the loop. It also added two new log calls: one at
  the beginning of the loop that builds the hosts file in memory and one
  at the end, that also includes the full content. Those could be useful
  pieces of debugging information. However, adding the host file
  contents can get expensive when there are large numbers of ports.

  For large numbers of ports, it can take more than one hour for the
  agent to sync up. The log output contributes a significant amount of
  that time.

  This was introduced in Liberty. One reviewer at the time had already
  expressed concerns about the impact of the log call.

  Will propose a patch to keep the log message, but remove the file
  contents.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1557640/+subscriptions