yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1695995] [NEW] of_interface=native makes openvswitch agents fail when run on the same node

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Simonin <matthieu.simonin@xxxxxxxx>
Date: Mon, 05 Jun 2017 21:09:36 -0000
Reply-to: Bug 1695995 <1695995@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

For eval purpose I used to run several openvswitch agents on the same machine.
Kolla-ansible proposes such a thing (https://docs.openstack.org/developer/kolla-ansible/nova-fake-driver.html) but this feature is nowaday broken.
In this context each openvswitch agent are running on its own dedicated 
docker container and are configured to use a different ovs bridge (br-int-1, br-int-2, …).

Some preliminary tests shows that using the default
`of_interface=native` led to some troubles.

Environment : 
-------------

$) docker exec -ti neutron_openvswitch_agent_1 pip freeze | grep neutron
neutron==11.0.0.0b2.dev408
neutron-fwaas==11.0.0.0b2.dev25
neutron-lib==1.7.0
python-neutronclient==6.2.0

$) docker exec -ti neutron_openvswitch_agent_1 ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.6.1
DB Schema 7.14.0

Observation with of_interface=native
-------------------------------------

On the nodes where several openvswitch agents run only one runs correctly. 
The others fail (3 agents in this example) showing the following log :

2017-06-05 20:24:16.056 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:17.117 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:21.061 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:22.123 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:26.064 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:27.125 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:31.065 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:32.128 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:36.072 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:37.131 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:41.077 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:42.136 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
2017-06-05 20:24:42.163 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout


$) openstack network agent list
+--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                               | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
| 143b3b54-0f5e-4a2a-a441-fed5e5cc496f | L3 agent           | enos-126-neutron-l3-agent          | nova              | True  | UP    | neutron-l3-agent          |
| a6d5e1d9-3102-4395-bd52-977372b0cfeb | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None              | True | UP    | neutron-openvswitch-agent |
| cd8d49c6-aaa6-4de7-a07f-68ef3b117e36 | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None              | True  | UP    | neutron-openvswitch-agent |
| d951874e-d449-459e-bf47-bd746a45b16a | Metadata agent     | enos-126-neutron-metadata-agent    | None              | True  | UP    | neutron-metadata-agent    |
| fe2cf638-ff26-4662-969e-5f49e6b57d4d | DHCP agent         | enos-126-neutron-dhcp-agent        | nova              | True  | UP    | neutron-dhcp-agent        |
+--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+


Observation with of_interface=ovs-ofctl
----------------------------------------

When switching to `of_interface=ovs-ofctl` everything works as expected : all the agents 
are running correctly : 

$) openstack network agent list
+--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                               | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
| 21a45a3a-4a2c-4c3d-9246-26adfd6cc890 | Open vSwitch agent | enos-125-3                         | None              | True  | UP    | neutron-openvswitch-agent |
| 563cb6d3-7527-4d99-a80a-bf278fbde57e | Open vSwitch agent | enos-125-1                         | None              | True  | UP    | neutron-openvswitch-agent |
| 596ec4f1-0fe9-42a0-be8d-602deb8a4a4e | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None              | True  | UP    | neutron-openvswitch-agent |
| 5bbb6722-4df9-465a-a433-fb8153f595b5 | Open vSwitch agent | enos-125-2                         | None              | True  | UP    | neutron-openvswitch-agent |
| 797fdec3-6967-46e5-9d81-6280c888066d | L3 agent           | enos-126-neutron-l3-agent          | nova              | True  | UP    | neutron-l3-agent          |
| b56549f1-c755-45dd-b2c9-072fdd418e78 | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None              | True  | UP    | neutron-openvswitch-agent |
| cef1c889-0851-44f4-b1d3-e6339320473a | DHCP agent         | enos-126-neutron-dhcp-agent        | nova              | True  | UP    | neutron-dhcp-agent        |
| df718b50-305c-4fcc-bce3-3dd252942dd3 | Metadata agent     | enos-126-neutron-metadata-agent    | None              | True  | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+


I think I can enforce `of_interface=ovs-ofctl` in the configuration file, but it doesn't seem a viable solution dur to its future deprecation.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1695995

Title:
  of_interface=native makes openvswitch agents fail when run on the same
  node

Status in neutron:
  New

Bug description:
  For eval purpose I used to run several openvswitch agents on the same machine.
  Kolla-ansible proposes such a thing (https://docs.openstack.org/developer/kolla-ansible/nova-fake-driver.html) but this feature is nowaday broken.
  In this context each openvswitch agent are running on its own dedicated 
  docker container and are configured to use a different ovs bridge (br-int-1, br-int-2, …).

  Some preliminary tests shows that using the default
  `of_interface=native` led to some troubles.

  Environment : 
  -------------

  $) docker exec -ti neutron_openvswitch_agent_1 pip freeze | grep neutron
  neutron==11.0.0.0b2.dev408
  neutron-fwaas==11.0.0.0b2.dev25
  neutron-lib==1.7.0
  python-neutronclient==6.2.0

  $) docker exec -ti neutron_openvswitch_agent_1 ovs-vsctl --version
  ovs-vsctl (Open vSwitch) 2.6.1
  DB Schema 7.14.0

  Observation with of_interface=native
  -------------------------------------

  On the nodes where several openvswitch agents run only one runs correctly. 
  The others fail (3 agents in this example) showing the following log :

  2017-06-05 20:24:16.056 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:17.117 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:21.061 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:22.123 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:26.064 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:27.125 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:31.065 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:32.128 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:36.072 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:37.131 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:41.077 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:42.136 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246
  2017-06-05 20:24:42.163 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout

  
  $) openstack network agent list
  +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
  | ID                                   | Agent Type         | Host                               | Availability Zone | Alive | State | Binary                    |
  +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
  | 143b3b54-0f5e-4a2a-a441-fed5e5cc496f | L3 agent           | enos-126-neutron-l3-agent          | nova              | True  | UP    | neutron-l3-agent          |
  | a6d5e1d9-3102-4395-bd52-977372b0cfeb | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None              | True | UP    | neutron-openvswitch-agent |
  | cd8d49c6-aaa6-4de7-a07f-68ef3b117e36 | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None              | True  | UP    | neutron-openvswitch-agent |
  | d951874e-d449-459e-bf47-bd746a45b16a | Metadata agent     | enos-126-neutron-metadata-agent    | None              | True  | UP    | neutron-metadata-agent    |
  | fe2cf638-ff26-4662-969e-5f49e6b57d4d | DHCP agent         | enos-126-neutron-dhcp-agent        | nova              | True  | UP    | neutron-dhcp-agent        |
  +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+

  
  Observation with of_interface=ovs-ofctl
  ----------------------------------------

  When switching to `of_interface=ovs-ofctl` everything works as expected : all the agents 
  are running correctly : 

  $) openstack network agent list
  +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
  | ID                                   | Agent Type         | Host                               | Availability Zone | Alive | State | Binary                    |
  +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+
  | 21a45a3a-4a2c-4c3d-9246-26adfd6cc890 | Open vSwitch agent | enos-125-3                         | None              | True  | UP    | neutron-openvswitch-agent |
  | 563cb6d3-7527-4d99-a80a-bf278fbde57e | Open vSwitch agent | enos-125-1                         | None              | True  | UP    | neutron-openvswitch-agent |
  | 596ec4f1-0fe9-42a0-be8d-602deb8a4a4e | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None              | True  | UP    | neutron-openvswitch-agent |
  | 5bbb6722-4df9-465a-a433-fb8153f595b5 | Open vSwitch agent | enos-125-2                         | None              | True  | UP    | neutron-openvswitch-agent |
  | 797fdec3-6967-46e5-9d81-6280c888066d | L3 agent           | enos-126-neutron-l3-agent          | nova              | True  | UP    | neutron-l3-agent          |
  | b56549f1-c755-45dd-b2c9-072fdd418e78 | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None              | True  | UP    | neutron-openvswitch-agent |
  | cef1c889-0851-44f4-b1d3-e6339320473a | DHCP agent         | enos-126-neutron-dhcp-agent        | nova              | True  | UP    | neutron-dhcp-agent        |
  | df718b50-305c-4fcc-bce3-3dd252942dd3 | Metadata agent     | enos-126-neutron-metadata-agent    | None              | True  | UP    | neutron-metadata-agent    |
  +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+

  
  I think I can enforce `of_interface=ovs-ofctl` in the configuration file, but it doesn't seem a viable solution dur to its future deprecation.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1695995/+subscriptions
Follow ups

[Bug 1695995] Re: of_interface=native makes openvswitch agents fail when run on the same node
From: Trevor McCasland, 2017-06-09