yahoo-eng-team team mailing list archive

[Alan Pevec <1393925@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron/icehouse
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1393925

Title:
  Race condition adding a security group rule when another is in-
  progress

Status in OpenStack Neutron (virtual network service):
  Fix Released
Status in neutron icehouse series:
  Fix Released
Status in neutron juno series:
  Fix Released
Status in OpenStack Security Advisories:
  Won't Fix

Bug description:
  I've come across a race condition where I sometimes see a security
  group rule is never added to iptables, if the OVS agent is in the
  middle of applying another security group rule when the RPC arrives.

  Here's an example scenario:

  nova boot --flavor 1 --image $nova_image  dev_server1
  sleep 4
  neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 1111 --port_range_max 1111 default
  neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 1112 --port_range_max 1112 default

  Wait for VM to complete booting, then check iptables:

  $ sudo iptables-save | grep 111
  -A neutron-openvswi-i741ff910-1 -p tcp -m tcp --dport 1111 -j RETURN

  The second rule is missing, and will only get added if you either add
  another rule, or restart the agent.

  My config is just devstack, running with the latest openstack bits as
  of today.  OVS agent w/vxlan and DVR enabled, nothing fancy.

  I've been able to track this down to the following code (i'll attach
  the complete log as a file due to line wraps):

  OVS agent receives RPC to setup port
      Port info is gathered for devices and filters for security groups are created
          Iptables "apply" is called
          New security group rule is added, triggering RPC message
          RPC received, and agent seems to add device to list that needs refresh

              Security group rule updated on remote: [u'5f0f5036-d14c-4b57-a855-ed39deaea256'] security_groups_rule_updated
              Security group rule updated [u'5f0f5036-d14c-4b57-a855-ed39deaea256']
              Adding [u'741ff910-12ba-4c1e-9dc9-38f7cbde0dc4'] devices to the list of devices for which firewall needs to be refreshed _security_group_updated

          Iptables "apply" is finished

  rpc_loop() in OVS agent does not notice there is more work to do on
  next loop, so rule never gets added

  At this point I'm thinking it could be that self.devices_to_refilter
  is modified in both _security_group_updated() and setup_port_filters()
  without any lock/semaphore, but the log doesn't explicity implicate it
  (perhaps we trust the timestamps too much?).

  I will continue to investigate, but if someone has an "aha!" moment
  after reading this far please add a note.

  A colleague here has also been able to duplicate this on his own
  devstack install, so it wasn't my fat-fingering that caused it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1393925/+subscriptions