← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1399249] [NEW] Neutron openvswitch-agent doesn't recover ports from binding_failed status

 

Public bug reported:

Ports created when neutron-openvswitch-agent is down are in status down
and "binding:vif_type=binding_failed" which is as it should be. When the
agent is rebooted it should be able to recreate the ports according to
the DB, but instead it logs a WARNING and creates the port with status
DOWN. only solution is to delete the port and create it again

>From agent log:
2014-12-04 16:53:00.559 16319 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2dcc9141-7439-450a-bb2a-fe31ab577f47 None] Device 3dc73917-93b1-4f6d-a2e1-90c74cea6de7 not defined on plugin


Recreation steps:

shut down ovs-agent and wait for neutron to notice:

[root@RHEL7Server ~]# systemctl stop neutron-openvswitch-agent.service 
[root@RHEL7Server ~(keystone_admin)]# neutron agent-list | grep open
| 2d97bbd1-b937-4b19-8205-4167bbcb659d | Open vSwitch agent | node_29  | xxx   | True           | neutron-openvswitch-agent |

create router and attach it to network
[root@RHEL7Server ~(keystone_admin)]# neutron router-create myrouter --ha False
Created a new router:
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| distributed           | False                                |
| external_gateway_info |                                      |
| ha                    | False                                |
| id                    | 8210f453-2a17-400e-ae32-74aa1503d0a5 |
| name                  | myrouter                             |
| routes                |                                      |
| status                | ACTIVE                               |
| tenant_id             | 183611eb84204b839e43d97c081973c0     |
+-----------------------+--------------------------------------+
[root@RHEL7Server ~(keystone_admin)]# neutron router-interface-add myrouter private
Added interface 3dc73917-93b1-4f6d-a2e1-90c74cea6de7 to router myrouter.
[root@RHEL7Server ~(keystone_admin)]# neutron l3-agent-list-hosting-router myrouter
+--------------------------------------+---------+----------------+-------+
| id                                   | host    | admin_state_up | alive |
+--------------------------------------+---------+----------------+-------+
| 0110d49c-59dd-496c-a2a3-549a2ad4ba4d | node_29 | True           | :-)   |
+--------------------------------------+---------+----------------+-------+

Port will show status DOWN, and "binding_failed"
[root@RHEL7Server ~(keystone_admin)]# neutron port-show 3dc73917-93b1-4f6d-a2e1-90c74cea6de7
+-----------------------+---------------------------------------------------------------------------------+
| Field                 | Value                                                                           |
+-----------------------+---------------------------------------------------------------------------------+
| admin_state_up        | True                                                                            |
| allowed_address_pairs |                                                                                 |
| binding:host_id       | node_29                                                                         |
| binding:profile       | {}                                                                              |
| binding:vif_details   | {}                                                                              |
| binding:vif_type      | binding_failed                                                                  |
| binding:vnic_type     | normal                                                                          |
| device_id             | 8210f453-2a17-400e-ae32-74aa1503d0a5                                            |
| device_owner          | network:router_interface                                                        |
| extra_dhcp_opts       |                                                                                 |
| fixed_ips             | {"subnet_id": "d8881a14-bd8b-4595-b497-8da6587a46c1", "ip_address": "10.0.0.1"} |
| id                    | 3dc73917-93b1-4f6d-a2e1-90c74cea6de7                                            |
| mac_address           | fa:16:3e:db:0f:9b                                                               |
| name                  |                                                                                 |
| network_id            | 6091abc0-4fdf-402d-aaf0-3a955fabd6b7                                            |
| security_groups       |                                                                                 |
| status                | DOWN                                                                            |
| tenant_id             | 183611eb84204b839e43d97c081973c0                                                |
+-----------------------+---------------------------------------------------------------------------------+


start ovs-agent
systemctl start neutron-openvswitch-agent.service
[root@RHEL7Server ~(keystone_admin)]# neutron agent-list | grep open
| 2d97bbd1-b937-4b19-8205-4167bbcb659d | Open vSwitch agent | node_29  | :-)   | True           | neutron-openvswitch-agent |

Port will be forever down and even when restarting the agent won't bring
it up

>From agent log:
2014-12-04 16:53:00.559 16319 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2dcc9141-7439-450a-bb2a-fe31ab577f47 None] Device 3dc73917-93b1-4f6d-a2e1-90c74cea6de7 not defined on plugin

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1399249

Title:
  Neutron openvswitch-agent doesn't recover ports from binding_failed
  status

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Ports created when neutron-openvswitch-agent is down are in status
  down and "binding:vif_type=binding_failed" which is as it should be.
  When the agent is rebooted it should be able to recreate the ports
  according to the DB, but instead it logs a WARNING and creates the
  port with status DOWN. only solution is to delete the port and create
  it again

  From agent log:
  2014-12-04 16:53:00.559 16319 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2dcc9141-7439-450a-bb2a-fe31ab577f47 None] Device 3dc73917-93b1-4f6d-a2e1-90c74cea6de7 not defined on plugin

  
  Recreation steps:

  shut down ovs-agent and wait for neutron to notice:

  [root@RHEL7Server ~]# systemctl stop neutron-openvswitch-agent.service 
  [root@RHEL7Server ~(keystone_admin)]# neutron agent-list | grep open
  | 2d97bbd1-b937-4b19-8205-4167bbcb659d | Open vSwitch agent | node_29  | xxx   | True           | neutron-openvswitch-agent |

  create router and attach it to network
  [root@RHEL7Server ~(keystone_admin)]# neutron router-create myrouter --ha False
  Created a new router:
  +-----------------------+--------------------------------------+
  | Field                 | Value                                |
  +-----------------------+--------------------------------------+
  | admin_state_up        | True                                 |
  | distributed           | False                                |
  | external_gateway_info |                                      |
  | ha                    | False                                |
  | id                    | 8210f453-2a17-400e-ae32-74aa1503d0a5 |
  | name                  | myrouter                             |
  | routes                |                                      |
  | status                | ACTIVE                               |
  | tenant_id             | 183611eb84204b839e43d97c081973c0     |
  +-----------------------+--------------------------------------+
  [root@RHEL7Server ~(keystone_admin)]# neutron router-interface-add myrouter private
  Added interface 3dc73917-93b1-4f6d-a2e1-90c74cea6de7 to router myrouter.
  [root@RHEL7Server ~(keystone_admin)]# neutron l3-agent-list-hosting-router myrouter
  +--------------------------------------+---------+----------------+-------+
  | id                                   | host    | admin_state_up | alive |
  +--------------------------------------+---------+----------------+-------+
  | 0110d49c-59dd-496c-a2a3-549a2ad4ba4d | node_29 | True           | :-)   |
  +--------------------------------------+---------+----------------+-------+

  Port will show status DOWN, and "binding_failed"
  [root@RHEL7Server ~(keystone_admin)]# neutron port-show 3dc73917-93b1-4f6d-a2e1-90c74cea6de7
  +-----------------------+---------------------------------------------------------------------------------+
  | Field                 | Value                                                                           |
  +-----------------------+---------------------------------------------------------------------------------+
  | admin_state_up        | True                                                                            |
  | allowed_address_pairs |                                                                                 |
  | binding:host_id       | node_29                                                                         |
  | binding:profile       | {}                                                                              |
  | binding:vif_details   | {}                                                                              |
  | binding:vif_type      | binding_failed                                                                  |
  | binding:vnic_type     | normal                                                                          |
  | device_id             | 8210f453-2a17-400e-ae32-74aa1503d0a5                                            |
  | device_owner          | network:router_interface                                                        |
  | extra_dhcp_opts       |                                                                                 |
  | fixed_ips             | {"subnet_id": "d8881a14-bd8b-4595-b497-8da6587a46c1", "ip_address": "10.0.0.1"} |
  | id                    | 3dc73917-93b1-4f6d-a2e1-90c74cea6de7                                            |
  | mac_address           | fa:16:3e:db:0f:9b                                                               |
  | name                  |                                                                                 |
  | network_id            | 6091abc0-4fdf-402d-aaf0-3a955fabd6b7                                            |
  | security_groups       |                                                                                 |
  | status                | DOWN                                                                            |
  | tenant_id             | 183611eb84204b839e43d97c081973c0                                                |
  +-----------------------+---------------------------------------------------------------------------------+


  start ovs-agent
  systemctl start neutron-openvswitch-agent.service
  [root@RHEL7Server ~(keystone_admin)]# neutron agent-list | grep open
  | 2d97bbd1-b937-4b19-8205-4167bbcb659d | Open vSwitch agent | node_29  | :-)   | True           | neutron-openvswitch-agent |

  Port will be forever down and even when restarting the agent won't
  bring it up

  From agent log:
  2014-12-04 16:53:00.559 16319 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2dcc9141-7439-450a-bb2a-fe31ab577f47 None] Device 3dc73917-93b1-4f6d-a2e1-90c74cea6de7 not defined on plugin

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1399249/+subscriptions


Follow ups

References