yahoo-eng-team team mailing list archive

[OpenStack Infra <1810563@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/628691
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2eb31f84c9a6c9fc6340819f756a7a82cbf395f3
Submitter: Zuul
Branch:    master

commit 2eb31f84c9a6c9fc6340819f756a7a82cbf395f3
Author: Doug Wiegley <dwiegley@xxxxxxxxxxxxxx>
Date:   Fri Jan 4 14:55:29 2019 -0700

    Fix performance regression adding rules to security groups
    
    Sometime between liberty and pike, adding rules to SG's got
    slow, and slower with every rule. Streamline the rule create path,
    and get close to the old performance back.
    
    Two performance fixes:
    1. Get rid of an n^2 duplicate check, using a hash table instead,
    on bulk creates. This is more memory intensive than the previous loop,
    but usable far past where the other becomes too slow to be useful.
    2. Use an object existence check in a few places where we do not
    want to load all of the child rules.
    
    Co-Authored-By: William Hager <whager@xxxxxxxxxxxxxx>
    Change-Id: I34e41a128f28211f2e7ab814a2611ce22620fcf3
    Closes-bug: 1810563


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1810563

Title:
  adding rules to security groups is slow

Status in neutron:
  Fix Released

Bug description:
  Sometime between liberty and pike, adding rules to SG's got slow, and
  slower with every rule added.

  Gerrit review with fixes is incoming.

  You can repro with a vanilla devstack install on master, and this
  script:

  #!/bin/bash

  OPENSTACK_TOKEN=$(openstack token issue | grep '| id' | awk '{print $4}')
  export OPENSTACK_TOKEN

  CCN1=10.210.162.2
  CCN3=10.210.162.10
  export ENDPOINT=localhost

  make_rules() {
      iter=$1
      prefix=$2
      file="$3"

      echo "generating rules"

      cat >$file <<EOF
  {"security_group_rules":[
  EOF

      comma=","
      i=0
      while [ $i -lt $iter ]; do
  	j=0
  	while [ $j -lt 10 ]; do
  	    if [ $i -eq $(($iter-1)) -a $j -eq 9 ]; then
  		comma=""
  	    fi
  	    cat >>$file <<EOF
  {"direction":"ingress","ethertype":"IPv4","port_range_max":10000,"port_range_min":8000,"protocol":"tcp"
  ,"remote_ip_prefix":"$prefix.$i.$j.0/24","security_group_id":"$SG_UUID"}${comma}
  EOF
  	    j=$((j+1))
  	done
  	i=$((i+1))
      done

      cat >>$file <<EOF
  ]}
  EOF
  }

  hit_api() {
      json="$1"

      echo "hitting api"

      start=$(perl -e "print time();")
      time curl --silent -g -i -X POST http://$ENDPOINT:9696/v2.0/security-group-rules.json -H "User-Agen
  t: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-T
  oken: $OPENSTACK_TOKEN" -d @${json} >/dev/null
      end=$(perl -e "print time();")
      echo $((end-start))
  }

  tmp=/tmp/sg-test.$$.tmp

  echo "Doing test with 1000 rules in bulk"
  openstack security group delete dw-test-1
  uuid=$(openstack security group create dw-test-1 | grep '| id' | awk '{print $4}')
  export SG_UUID="$uuid"
  make_rules 100 4 $tmp
  hit_api $tmp

  echo "Doing loop test"
  openstack security group delete dw-test-2
  uuid=$(openstack security group create dw-test-2 | grep '| id' | awk '{print $4}')
  export SG_UUID="$uuid"
  elapsed=0
  mm=0
  while [ $mm -lt 20 ]; do
      make_rules 5 $(($mm+1)) $tmp
      n=$(hit_api $tmp | tail -1)
      elapsed=$((elapsed+n))
      mm=$((mm+1))
  done
  echo "Loop test took $elapsed seconds"

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1810563/+subscriptions