yahoo-eng-team team mailing list archive

[OpenStack Infra <1663465@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/431876
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ee5677bd61de3a044502013bf097bb3622d7e934
Submitter: Jenkins
Branch:    master

commit ee5677bd61de3a044502013bf097bb3622d7e934
Author: jufeng <jesse@xxxxxxxxxxxx>
Date:   Fri Feb 10 12:19:56 2017 +0800

    Optimize pid property in AsyncProcess class
    
    The pid property in the the AsyncProcess process class executes
    command 'ps --ppid <pid> -o pid=' every time the associated
    asynchronous process's pid is needed. This can be very heavy
    in high load network and compute nodes. Please see related bug.
    
    This change optimizes the pid property to execute 'ps --ppid <pid> -o
    pid=' only the first time it is called, memoizing the retrieved pid for
    future calls.
    
    Change-Id: Idd347b0b22320cbfdb5d6a738e218e5788dd7d6b
    Closes-Bug:1663465


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1663465

Title:
  [performance improvement] update neutron-openvswitch-agent's
  AsyncProcess

Status in neutron:
  Fix Released

Bug description:
  neutron-openvswitch-agent' rpc_loop loops every 2 seconds by default.
  In every loop, the function invocation will do every time: _agent_has_updates -> polling_manager.is_polling_required -> self._monitor.has_updates -> self.is_active() -> self.pid -> utils.get_root_helper_child_pid -> find_child_pids -> execute 'ps --ppid <pid> -o pid=' command.
  The command ('ps --ppid <pid> -o pid=') execution is heavy especially for high load server, I have 800 HA vRouters in my network node. Every time I use top command to see server load, I will always find ps process with high CPU usage, and the every rpc_loop will takes 8+ seconds according to the neutron-openvswitch-agent log.
  So we need to find a way to avoid this invocation.

  In class AsyncProcess, we can get the child's pid when the child
  process is okey to avoid getting pid every time using ps command
  in is_active function.
  When the child's pid is gotten, we just check /proc/<pid> to
  know whether the child process is active or not.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1663465/+subscriptions