yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1573505] [NEW] create ovs vm, ovs agent log is abnormal

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: QunyingRan <ran.qunying@xxxxxxxxxx>
Date: Fri, 22 Apr 2016 09:54:57 -0000
Reply-to: Bug 1573505 <1573505@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

 1. create ovs vm find error log in neutron-openvswitch-agent, the port staus is build.
---------------------------------------------------------------------------
2016-04-22 04:47:47.624 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-90c9a4d9-358b-4590-bad4-61580d0683f4 None None] Error while processing VIF ports
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1992, in rpc_loop
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, ovs_restarted)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1623, in process_network_ports
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 294, in setup_port_filters
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 149, in decorated_function
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 174, in prepare_devices_filter
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/firewall.py", line 129, in defer_apply
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 823, in filter_defer_apply_off
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.unfiltered_ports)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 214, in _setup_chains_apply
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._setup_chain(port, firewall.INGRESS_DIRECTION)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 241, in _setup_chain
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._add_rules_by_security_group(port, DIRECTION)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 493, in _add_rules_by_security_group
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._update_ipset_members(remote_sg_ids)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 529, in _update_ipset_members
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ipset.set_members(sg_id, ip_version, current_ips)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 85, in set_members
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.set_members_mutate(set_name, ethertype, member_ips)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 94, in set_members_mutate
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._create_set(set_name, ethertype)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 140, in _create_set
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply(cmd)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 150, in _apply
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     check_exit_code=fail_on_errors)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 114, in execute
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 100, in execute_rootwrap_daemon
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return client.execute(cmd, process_input)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line 128, in execute
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     res = proxy.run_one_command(cmd, stdin)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "<string>", line 2, in run_one_command
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise convert_to_error(kind, result)
2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError: 

2. Add debug information in iptables_manager.py :remove_rule:

          try:
            if '$' in rule:
                rule = ' '.join(
                    self._wrap_target_chain(e, wrap) for e in rule.split(' '))

            self.rules.remove(IptablesRule(chain, rule, wrap, top,
                                           self.wrap_name,
                                           comment=comment))
            if not wrap:
                self.remove_rules.append(str(IptablesRule(chain, rule, wrap,
                                                          top, self.wrap_name,
                                                          comment=comment)))
        except ValueError as e:
            LOG.info(" e %s", e)
   
       e's vlaue is  "list.remove(x): x not in list"

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1573505

Title:
  create ovs vm,ovs  agent log is abnormal

Status in neutron:
  New

Bug description:
   1. create ovs vm find error log in neutron-openvswitch-agent, the port staus is build.
  ---------------------------------------------------------------------------
  2016-04-22 04:47:47.624 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-90c9a4d9-358b-4590-bad4-61580d0683f4 None None] Error while processing VIF ports
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1992, in rpc_loop
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, ovs_restarted)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1623, in process_network_ports
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 294, in setup_port_filters
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 149, in decorated_function
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 174, in prepare_devices_filter
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.gen.next()
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/firewall.py", line 129, in defer_apply
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 823, in filter_defer_apply_off
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.unfiltered_ports)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 214, in _setup_chains_apply
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._setup_chain(port, firewall.INGRESS_DIRECTION)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 241, in _setup_chain
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._add_rules_by_security_group(port, DIRECTION)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 493, in _add_rules_by_security_group
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._update_ipset_members(remote_sg_ids)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 529, in _update_ipset_members
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ipset.set_members(sg_id, ip_version, current_ips)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 85, in set_members
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.set_members_mutate(set_name, ethertype, member_ips)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 94, in set_members_mutate
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._create_set(set_name, ethertype)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 140, in _create_set
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply(cmd)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 150, in _apply
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     check_exit_code=fail_on_errors)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 114, in execute
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 100, in execute_rootwrap_daemon
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return client.execute(cmd, process_input)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line 128, in execute
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     res = proxy.run_one_command(cmd, stdin)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "<string>", line 2, in run_one_command
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise convert_to_error(kind, result)
  2016-04-22 04:47:47.624 TRACE neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError: 

  2. Add debug information in iptables_manager.py :remove_rule:

            try:
              if '$' in rule:
                  rule = ' '.join(
                      self._wrap_target_chain(e, wrap) for e in rule.split(' '))

              self.rules.remove(IptablesRule(chain, rule, wrap, top,
                                             self.wrap_name,
                                             comment=comment))
              if not wrap:
                  self.remove_rules.append(str(IptablesRule(chain, rule, wrap,
                                                            top, self.wrap_name,
                                                            comment=comment)))
          except ValueError as e:
              LOG.info(" e %s", e)
     
         e's vlaue is  "list.remove(x): x not in list"

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1573505/+subscriptions
Follow ups

[Bug 1573505] Re: create ovs vm, ovs agent log is abnormal
From: QunyingRan, 2016-06-06