yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1315495] Re: Openvswitch throwing a table missing error?

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1315495@xxxxxxxxxxxxxxxxxx>
Date: Thu, 02 Oct 2014 04:17:13 -0000
Reply-to: Bug 1315495 <1315495@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for neutron because there has been no activity for 60 days.]

** Changed in: neutron
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1315495

Title:
  Openvswitch throwing a table missing error?

Status in OpenStack Neutron (virtual network service):
  Expired

Bug description:
  I am running IceHouse with CentOS 6.4 and I followed the redhat
  deployment guide for my install.  It is a 3 node setup with
  controller, network and compute nodes.  My issue is happening on the
  compute node.

  My /var/log/neutron/openvswitch-agent.log file is continually throwing
  this error:

  2014-05-02 15:11:44.195 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
  2014-05-02 15:11:44.277 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo93d39ce1-92', [u'map', [[u'attached-mac', u'fa:16:3e:37:26:f9'], [u'iface-id', u'93d39ce1-9209-49ee-96b5-01ff479b71c5'], [u'iface-status', u'active'], [u'vm-uuid', u'74187c71-8fed-4214-abb2-42c301489deb']]], -1]
  2014-05-02 15:11:44.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo02722581-14', [u'map', [[u'attached-mac', u'fa:16:3e:1f:59:19'], [u'iface-id', u'02722581-14f4-4e6f-bfd2-8fc6594440da'], [u'iface-status', u'active'], [u'vm-uuid', u'3402975f-6a90-45d9-86e4-0f4ed00b2762']]], -1]
  2014-05-02 15:11:44.358 2003 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'831403fc-8d48-4949-8a4f-4c4f3455d81a', u'fd5bb883-a415-49ec-849a-07442e3c7f1c'])
  2014-05-02 15:11:44.796 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port fd5bb883-a415-49ec-849a-07442e3c7f1c updated. Details: {u'admin_state_up': True, u'network_id': u'5e377240-4297-44c5-aec3-90e7db89b7a2', u'segmentation_id': 2, u'physical_network': None, u'device': u'fd5bb883-a415-49ec-849a-07442e3c7f1c', u'port_id': u'fd5bb883-a415-49ec-849a-07442e3c7f1c', u'network_type': u'gre'}
  2014-05-02 15:11:44.854 2003 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error while processing VIF ports
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1226, in rpc_loop
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1080, in process_network_ports
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     devices_added_updated)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 985, in treat_devices_added_or_updated
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.context, device, self.agent_id, cfg.CONF.host)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/rpc.py", line 107, in update_device_up
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     topic=self.topic)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/proxy.py", line 125, in call
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     result = rpc.call(context, real_topic, msg, timeout)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/__init__.py", line 112, in call
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return _get_impl().call(CONF, context, topic, msg, timeout)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 784, in call
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     rpc_amqp.get_connection_pool(conf, Connection))
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 575, in call
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     rv = list(rv)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 540, in __iter__
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise result
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: OperationalError (OperationalError) no such table: ml2_port_bindings u"SELECT ml2_port_bindings.port_id AS ml2_port_bindings_port_id, ml2_port_bindings.host AS ml2_port_bindings_host, ml2_port_bindings.vnic_type AS ml2_port_bindings_vnic_type, ml2_port_bindings.profile AS ml2_port_bindings_profile, ml2_port_bindings.vif_type AS ml2_port_bindings_vif_type, ml2_port_bindings.vif_details AS ml2_port_bindings_vif_details, ml2_port_bindings.driver AS ml2_port_bindings_driver, ml2_port_bindings.segment AS ml2_port_bindings_segment \nFROM ml2_port_bindings \nWHERE ml2_port_bindings.port_id LIKE ? || '%%'" (u'fd5bb883-a415-49ec-849a-07442e3c7f1c',)
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 462, in _process_data\n    **args)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch\n    neutron_ctxt, version, method, namespace, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/rpc.py", line 196, in update_device_up\n    if (host and not plugin.port_bound_to_host(port_id, host)):\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 782, in port_bound_to_host\n    port_host = db.get_port_binding_host(port_id)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/db.py", line 130, in get_port_binding_host\n    filter(models.PortBinding.port_id.startswith(port_id)).\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2184, in one\n    ret = list(self)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances\n    result = conn.execute(querycontext.statement, self._params)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute\n    params)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement\n    compiled_sql, distilled_params\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context\n    context)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context\n    context)\n', u'  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute\n    cursor.execute(statement, parameters)\n', u'OperationalError: (OperationalError) no such table: ml2_port_bindings u"SELECT ml2_port_bindings.port_id AS ml2_port_bindings_port_id, ml2_port_bindings.host AS ml2_port_bindings_host, ml2_port_bindings.vnic_type AS ml2_port_bindings_vnic_type, ml2_port_bindings.profile AS ml2_port_bindings_profile, ml2_port_bindings.vif_type AS ml2_port_bindings_vif_type, ml2_port_bindings.vif_details AS ml2_port_bindings_vif_details, ml2_port_bindings.driver AS ml2_port_bindings_driver, ml2_port_bindings.segment AS ml2_port_bindings_segment \\nFROM ml2_port_bindings \\nWHERE ml2_port_bindings.port_id LIKE ? || \'%%\'" (u\'fd5bb883-a415-49ec-849a-07442e3c7f1c\',)\n'].
  2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
  2014-05-02 15:11:46.196 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
  2014-05-02 15:11:46.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo93d39ce1-92', [u'map', [[u'attached-mac', u'fa:16:3e:37:26:f9'], [u'iface-id', u'93d39ce1-9209-49ee-96b5-01ff479b71c5'], [u'iface-status', u'active'], [u'vm-uuid', u'74187c71-8fed-4214-abb2-42c301489deb']]], -1]
  2014-05-02 15:11:46.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo02722581-14', [u'map', [[u'attached-mac', u'fa:16:3e:1f:59:19'], [u'iface-id', u'02722581-14f4-4e6f-bfd2-8fc6594440da'], [u'iface-status', u'active'], [u'vm-uuid', u'3402975f-6a90-45d9-86e4-0f4ed00b2762']]], -1]
  2014-05-02 15:11:46.359 2003 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'831403fc-8d48-4949-8a4f-4c4f3455d81a', u'fd5bb883-a415-49ec-849a-07442e3c7f1c'])

  however my ml2_port_bindings table does exist and has data:

  mysql> desc ml2_port_bindings
      -> ;
  +-------------+---------------+------+-----+---------+-------+
  | Field       | Type          | Null | Key | Default | Extra |
  +-------------+---------------+------+-----+---------+-------+
  | port_id     | varchar(36)   | NO   | PRI | NULL    |       |
  | host        | varchar(255)  | NO   |     | NULL    |       |
  | vnic_type   | varchar(64)   | NO   |     | NULL    |       |
  | profile     | varchar(4095) | NO   |     | NULL    |       |
  | vif_type    | varchar(64)   | NO   |     | NULL    |       |
  | vif_details | varchar(4095) | NO   |     | NULL    |       |
  | driver      | varchar(64)   | YES  |     | NULL    |       |
  | segment     | varchar(36)   | YES  | MUL | NULL    |       |
  +-------------+---------------+------+-----+---------+-------+
  8 rows in set (0.00 sec)

  mysql> select * from ml2_port_bindings;
  +--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
  | port_id                              | host                  | vnic_type | profile | vif_type | vif_details                                    | driver      | segment                              |
  +--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
  | 1c525688-451c-457d-bfc4-b65ebcffcc5a |                       | normal    | {}      | unbound  |                                                | NULL        | NULL                                 |
  | 27c6fae0-f7a4-4a7d-a39f-c6b8946d2cf2 | nmtg-net001.cisco.com | normal    | {}      | ovs      | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
  | 4f2d43cb-35f9-48f7-a715-6217aebf49fd | nmtg-cpu001.cisco.com | normal    |         | ovs      | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
  | 831403fc-8d48-4949-8a4f-4c4f3455d81a | nmtg-cpu001.cisco.com | normal    |         | ovs      | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
  | 8e24653d-1e5f-4fa1-91a4-c85434c4659b | nmtg-net001.cisco.com | normal    | {}      | ovs      | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | 9321dddb-75fb-496b-bf9f-ecf85ee202a2 |
  | fd5bb883-a415-49ec-849a-07442e3c7f1c | nmtg-cpu001.cisco.com | normal    |         | ovs      | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
  +--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
  6 rows in set (0.00 sec)

  mysql>

  The compute node is able to ping and connect to the neutron database
  manually on the command line.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1315495/+subscriptions
References

[Bug 1315495] [NEW] Openvswitch throwing a table missing error?
From: J. Kary, 2014-05-02