yahoo-eng-team team mailing list archive

[OpenStack Infra <1623958@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/370902
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=35e386f6165a431a066ea92cd317406322053ff7
Submitter: Jenkins
Branch:    master

commit 35e386f6165a431a066ea92cd317406322053ff7
Author: Brian Haley <brian.haley@xxxxxxx>
Date:   Thu Sep 15 10:40:04 2016 -0400

    Fix metering-agent iptables restore failure
    
    When we removed the trailing space from iptables rules, we
    broke the code that adds/restores them since it was assuming
    it could split() based on that space to get the rule.
    
    Just detect if the two values are equal, denoting there was
    no rule instead.  This is fine since a self-referencing rule
    is invalid anyways.
    
    Change-Id: I9157ced10de3099790dea7f94aa4e614ebf7f25c
    Closes-bug: #1623958


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1623958

Title:
  Metering-agent fails to restore iptables with 'Bad argument'

Status in neutron:
  Fix Released

Bug description:
  iptables-restore fails with 'Bad argument' when we call 'add_rule' with '' as rule,
  because IptableManager try to add line below which is invalid format.
  ===
  -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a
  ===

  This behavior is occurred after the modification.
  https://github.com/openstack/neutron/commit/5b7c71a327d735134fa0eeb4427d0e1bd1f7d1e5

  Created blank by rule is stripped during initializing IptablesRule.
  https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L124

  This stripping blank changes result in _generate_chain_diff_iptables_commands as follows.
  https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L767

  before:  '' is set in rule variable.
  after: chain name(e.g.neutron-meter-l-e648a667-c21) is set as rule variable. 

  Additional string is added and invalid format is generated.
  Thus, iptable-restore fails.

  trace in metering-agent.log
  ==============================
  2016-09-15 12:50:39.202 22676 ERROR neutron.agent.linux.utils [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761c
  e - - -] Exit code: 2; Stdin: # Generated by iptables_manager
  *filter
  :neutron-meter-FORWARD - [0:0]
  :neutron-meter-INPUT - [0:0]
  :neutron-meter-OUTPUT - [0:0]
  :neutron-meter-l-ba70a353-f3a - [0:0]
  :neutron-meter-local - [0:0]
  :neutron-meter-r-ba70a353-f3a - [0:0]
  -I FORWARD 2 -j neutron-meter-FORWARD
  -I INPUT 1 -j neutron-meter-INPUT
  -I OUTPUT 2 -j neutron-meter-OUTPUT
  -I neutron-filter-top 1 -j neutron-meter-local
  -I neutron-meter-FORWARD 1 -j neutron-meter-r-ba70a353-f3a
  -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a
  COMMIT
  # Completed by iptables_manager
  # Generated by iptables_manager
  *raw
  :neutron-meter-OUTPUT - [0:0]
  :neutron-meter-PREROUTING - [0:0]
  -I OUTPUT 1 -j neutron-meter-OUTPUT
  -I PREROUTING 1 -j neutron-meter-PREROUTING
  COMMIT
  # Completed by iptables_manager
  ; Stdout: ; Stderr: Bad argument `neutron-meter-l-ba70a353-f3a'
  Error occurred at line: 14
  Try `iptables-restore -h' or 'iptables-restore --help' for more information.

  2016-09-15 12:50:39.204 22676 ERROR neutron.agent.linux.iptables_manager [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761ce - - -] IPTablesManager.apply failed to apply the following set of iptables rules:
        1. # Generated by iptables_manager
        2. *filter
        3. :neutron-meter-FORWARD - [0:0]
        4. :neutron-meter-INPUT - [0:0]
        5. :neutron-meter-OUTPUT - [0:0]
        6. :neutron-meter-l-ba70a353-f3a - [0:0]
        7. :neutron-meter-local - [0:0]
        8. :neutron-meter-r-ba70a353-f3a - [0:0]
        9. -I FORWARD 2 -j neutron-meter-FORWARD
       10. -I INPUT 1 -j neutron-meter-INPUT
       11. -I OUTPUT 2 -j neutron-meter-OUTPUT
       12. -I neutron-filter-top 1 -j neutron-meter-local
       13. -I neutron-meter-FORWARD 1 -j neutron-meter-r-ba70a353-f3a
       14. -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a
       15. COMMIT
       16. # Completed by iptables_manager
       17. # Generated by iptables_manager
       18. *raw
       19. :neutron-meter-OUTPUT - [0:0]
       20. :neutron-meter-PREROUTING - [0:0]
       21. -I OUTPUT 1 -j neutron-meter-OUTPUT
       22. -I PREROUTING 1 -j neutron-meter-PREROUTING
       23. COMMIT
       24. # Completed by iptables_manager
       25.
  2016-09-15 12:50:39.204 22676 DEBUG oslo_concurrency.lockutils [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761ce - - -] Releasing semaphore "iptables-qrouter-98b59b73-4490-4834-a02c-ae2e1ea16d64" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent [req-b3191be2-6daf-42bf-bc3a-e792481da48b da7194b4e98b4c8badc5912bbcd7aea4 9384ef06bc9d4af08a384692c92761ce - - -] Driver neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeteringDriver:add_metering_label runtime error
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last):
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/agents/metering_agent.py", line 166, in _invoke_driver
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     return getattr(self.metering_driver, func_name)(context, meterings)
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     return method(*args, **kwargs)
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_driver.py", line 262, in add_metering_label
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     self._process_associate_metering_label(router)
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_driver.py", line 231, in _process_associate_metering_label
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     rm.metering_labels[label_id] = label
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_driver.py", line 58, in __exit__
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     self.im.apply()
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 438, in apply
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     return self._apply()
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 446, in _apply
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     first = self._apply_synchronized()
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 532, in _apply_synchronized
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     '\n'.join(log_lines))
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     self.force_reraise()
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     six.reraise(self.type_, self.value, self.tb)
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 511, in _apply_synchronized
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     run_as_root=True)
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 138, in execute
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent     raise RuntimeError(msg)
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 2; Stdin: # Generated by iptables_manager
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent *filter
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-FORWARD - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-INPUT - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-OUTPUT - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-l-ba70a353-f3a - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-local - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-r-ba70a353-f3a - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I FORWARD 2 -j neutron-meter-FORWARD
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I INPUT 1 -j neutron-meter-INPUT
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I OUTPUT 2 -j neutron-meter-OUTPUT
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I neutron-filter-top 1 -j neutron-meter-local
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I neutron-meter-FORWARD 1 -j neutron-meter-r-ba70a353-f3a
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I neutron-meter-l-ba70a353-f3a 1 neutron-meter-l-ba70a353-f3a
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent COMMIT
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent # Completed by iptables_manager
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent # Generated by iptables_manager
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent *raw
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-OUTPUT - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent :neutron-meter-PREROUTING - [0:0]
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I OUTPUT 1 -j neutron-meter-OUTPUT
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent -I PREROUTING 1 -j neutron-meter-PREROUTING
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent COMMIT
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent # Completed by iptables_manager
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent ; Stdout: ; Stderr: Bad argument `neutron-meter-l-ba70a353-f3a'
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent Error occurred at line: 14
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent Try `iptables-restore -h' or 'iptables-restore --help' for more information.
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent
  2016-09-15 12:50:39.205 22676 ERROR neutron.services.metering.agents.metering_agent
  =================================

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1623958/+subscriptions