yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

** No longer affects: nova/diablo

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/969545

Title:
  missing quotas on security group rules

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) essex series:
  Fix Released
Status in “nova” package in Ubuntu:
  Fix Released
Status in “nova” source package in Precise:
  Fix Released

Bug description:
  Nova does not currently enforce quotas on the number of security group
  rules an *authorized* user can create.

  Using the EC2 API an authorized user could do the following to create
  2000+ rules in the default security group:

  for X in {1..1024}; do
  euca-authorize -P tcp -p $X default
  euca-authorize -P udp -p $X default
  done

  ** I choose 1024 as the upper limit of the privileged ports... you
  could go higher...

  Each of these commands would translate into 2000+ iptables rules
  getting created per/instance.  Furthermore... if you do this after you
  have instances created it causes quite a bit of thrashing of iptables
  rules on the Nova compute nodes ... to the extent that instances seem
  to get stuck in BUILD state with only a few nodes... and the nova
  compute.log file size grows quite large as well (Gigs of data) due to
  the fact that the iptables rules are getting logged (via log channel
  'info'):

  After a couple of instances and runs of this script I'm already at
  well over 10 gigs of data in Nova's compute.log file.

  ------

  The ability to create this many iptables rules is a performance
  concern from both a networking and compute service prospective. I'm
  not sure what the limit on iptables rules is but 2000+ rules per
  instance on each nova compute host is certainly going to be a drain.
  Additionally the extra overhead of maintain this many rules can
  significantly slow down Nova compute's performance to manage
  instances.

  Couple of fixes we should look at doing:

  -Add a quota to limit the number of groups and rules per group to
  something like 20 groups/20.

  -Tone down some of the log level info logging in the firewall drivers.

  Optionally there are some optimizations that we could probably look at
  to combine some of the individual iptables rules into ranges. Probably
  best not to do that within the scope of this ticket however.

  ---

  Lastly... we can do the same thing via the OSAPI using 'nova secgroup-
  add-rule' although it should be noted that you'd have to slow down
  things a bit on the OSAPI side due to rate limiting of the POST
  requests... So the OSAPI would slow down an attack on this front but
  wouldn't prevent it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/969545/+subscriptions