yahoo-eng-team team mailing list archive

[OpenStack Infra <1719711@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/523319
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=37bd42e4f5d1be49689032822aca339523cfda33
Submitter: Zuul
Branch:    master

commit 37bd42e4f5d1be49689032822aca339523cfda33
Author: Jens Harbott <j.harbott@xxxxxxxx>
Date:   Tue Nov 28 07:39:04 2017 +0000

    Fix error when using protocol number in security groups
    
    When the support of protocol numbers in security groups
    was fixed in [1], it introduced two deficiencies in the
    iptables code:
    
    - it was missing some protocols, for example, 'icmp', 'tcp'
      and 'udp', so when rules were added by number we did not
      use their name as iptables expects
    - it used a dictionary to map numbers to names, but protocol
      numbers are stored as strings (i.e. '1' != 1)
    
    Updated the iptables number mapping dict to have all
    currently-known values, even those that are already well-known
    and should have been using a string instead of a number.
    
    Also changed the iptables number mapping dict to use
    strings as the keys instead of numbers, since that's
    what will be passed from the security group code.
    
    Removed IPTABLES_PROTOCOL_MAP as it lives in neutron-lib,
    and accidentally snuck-in in [1].
    
    [1] I5895250b47ddf664d214cf085be693c3897e0c87
    
    Change-Id: I6b7575eb531b4f35579960c3feb47000cd259b86
    Closes-Bug: 1719711


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1719711

Title:
  iptables failed to apply when binding a port with
  AGENT.debug_iptables_rules enabled

Status in neutron:
  Fix Released

Bug description:
  http://logs.openstack.org/21/504021/2/check/gate-tempest-dsvm-neutron-
  scenario-linuxbridge-ubuntu-xenial-nv/e47a3f3/testr_results.html.gz

  
  Traceback (most recent call last):
    File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_security_groups.py", line 127, in test_two_sec_groups
      num_servers=1, security_groups=security_groups_list)
    File "/opt/stack/new/neutron/neutron/tests/tempest/scenario/test_security_groups.py", line 54, in create_vm_testing_sec_grp
      const.SERVER_STATUS_ACTIVE)
    File "tempest/common/waiters.py", line 76, in wait_for_server_status
      server_id=server_id)
  tempest.exceptions.BuildErrorException: Server e1120d99-f0eb-43eb-a38b-847843a838b5 failed to build and is in ERROR status
  Details: {u'message': u'Build of instance e1120d99-f0eb-43eb-a38b-847843a838b5 aborted: Failed to allocate the network(s), not rescheduling.', u'code': 500, u'created': u'2017-09-26T09:23:42Z'}

  In linuxbridge agent log: http://logs.openstack.org/21/504021/2/check
  /gate-tempest-dsvm-neutron-scenario-linuxbridge-ubuntu-xenial-
  nv/e47a3f3/logs/screen-q-agt.txt.gz?level=TRACE#_Sep_26_09_16_30_623747

  Sep 26 09:16:30.623747 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.agent.linux.iptables_manager [None req-78fc6bc1-a089-4d5f-91d8-e5191e45978c None None] IPTables Rules did not converge. Diff: # Generated by iptables_manager
  Sep 26 09:16:30.623936 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: *filter
  Sep 26 09:16:30.624117 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: -D neutron-linuxbri-ibc1a22b9-e 6
  Sep 26 09:16:30.624316 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: -I neutron-linuxbri-ibc1a22b9-e 6 -p 1 -j RETURN
  Sep 26 09:16:30.624482 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: COMMIT
  Sep 26 09:16:30.624955 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: # Completed by iptables_manager
  Sep 26 09:16:30.635308 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent [None req-78fc6bc1-a089-4d5f-91d8-e5191e45978c None None] Error in agent loop. Devices info: {'current': set(['tapbc1a22b9-ef', 'tapc9488f0f-ae', 'tape2d2e245-96', 'tap93881b27-41', 'tapb265ee77-37', 'tapbadc6b64-69', 'tapa813220a-1d', 'tapa376782a-75', 'tap395ccf4d-c9', 'tapca94a412-e7', 'tap58f740f2-aa', 'tapb2444941-9f']), 'timestamps': {'tap93881b27-41': 56, 'tapc9488f0f-ae': 62, 'tape2d2e245-96': 11, 'tapbc1a22b9-ef': 68, 'tapb265ee77-37': 9, 'tapbadc6b64-69': 55, 'tapa813220a-1d': 66, 'tapa376782a-75': 65, 'tap395ccf4d-c9': 67, 'tapca94a412-e7': 6, 'tap58f740f2-aa': 59, 'tapb2444941-9f': 10}, 'removed': set([]), 'added': set([]), 'updated': set([])}: IpTablesApplyException: IPTables Rules did not converge. Diff: # Generated by iptables_manager
  Sep 26 09:16:30.636316 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: *filter
  Sep 26 09:16:30.636510 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: -D neutron-linuxbri-ibc1a22b9-e 6
  Sep 26 09:16:30.636700 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: -I neutron-linuxbri-ibc1a22b9-e 6 -p 1 -j RETURN
  Sep 26 09:16:30.636898 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: COMMIT
  Sep 26 09:16:30.637075 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: # Completed by iptables_manager
  Sep 26 09:16:30.637269 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
  Sep 26 09:16:30.637683 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 453, in daemon_loop
  Sep 26 09:16:30.637962 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     sync = self.process_network_devices(device_info)
  Sep 26 09:16:30.638211 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 157, in wrapper
  Sep 26 09:16:30.638373 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
  Sep 26 09:16:30.638538 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 200, in process_network_devices
  Sep 26 09:16:30.638728 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     device_info.get('updated'))
  Sep 26 09:16:30.639034 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 256, in setup_port_filters
  Sep 26 09:16:30.639220 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.refresh_firewall(updated_devices)
  Sep 26 09:16:30.639702 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 110, in decorated_function
  Sep 26 09:16:30.639993 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     *args, **kwargs)
  Sep 26 09:16:30.640390 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 209, in refresh_firewall
  Sep 26 09:16:30.640671 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._apply_port_filter(device_ids, update_filter=True)
  Sep 26 09:16:30.640925 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/securitygroups_rpc.py", line 145, in _apply_port_filter
  Sep 26 09:16:30.641159 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.firewall.process_trusted_ports(trusted_devices)
  Sep 26 09:16:30.641466 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  Sep 26 09:16:30.641926 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.gen.next()
  Sep 26 09:16:30.642208 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/firewall.py", line 145, in defer_apply
  Sep 26 09:16:30.642454 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.filter_defer_apply_off()
  Sep 26 09:16:30.642701 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/linux/iptables_firewall.py", line 852, in filter_defer_apply_off
  Sep 26 09:16:30.642990 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.iptables.defer_apply_off()
  Sep 26 09:16:30.643349 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/linux/iptables_manager.py", line 429, in defer_apply_off
  Sep 26 09:16:30.643647 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._apply()
  Sep 26 09:16:30.643920 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/stack/new/neutron/neutron/agent/linux/iptables_manager.py", line 454, in _apply
  Sep 26 09:16:30.644170 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise n_exc.IpTablesApplyException(msg)
  Sep 26 09:16:30.644519 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent IpTablesApplyException: IPTables Rules did not converge. Diff: # Generated by iptables_manager
  Sep 26 09:16:30.644842 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent *filter
  Sep 26 09:16:30.645208 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent -D neutron-linuxbri-ibc1a22b9-e 6
  Sep 26 09:16:30.645479 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent -I neutron-linuxbri-ibc1a22b9-e 6 -p 1 -j RETURN
  Sep 26 09:16:30.645847 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent COMMIT
  Sep 26 09:16:30.646182 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent # Completed by iptables_manager
  Sep 26 09:16:30.646457 ubuntu-xenial-ovh-gra1-11134533 neutron-linuxbridge-agent[24363]: ERROR neutron.plugins.ml2.drivers.agent._common_agent 

  This happens in scenario job when iptables manager is used. Despite
  the fact that it doesn't happen in ovs flavor of the job, it's not
  clear whether it affects ovs setups using iptables, because ovs
  scenario job uses 'openvswitch' flow based firewall driver instead of
  iptables.

  This happens on a patch that adds a new scenario test case targeting
  security groups, so may be related:
  https://review.openstack.org/#/c/504021/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1719711/+subscriptions