← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1661717] [NEW] L2pop filters out port info for HA router internal interface when sending out notification

 

Public bug reported:

Branch: stable/newton

Setup:
l3_ha = True
l2_population = True
arp_responder = True
enable_distributed_routing = False

Node1: controller/network
Node2: controller/network
Node3: compute

step 1: created external network
step 2: created a router
step 3: set router gateway to external network
step 4: created internal network with subnet, for example: 2.2.2.0/24
step 5: added internal network subnet to router using command: neutron router-interface-add <router> <subnet-id>
step 6: created an instance (VM1) on the internal network
Note: VM1 scheduled to boot on compute host or Node1

Symptoms:
Missing arp entry on Node1 for the internal interface on router
VM1 does not know where to forward out going traffic

Debugging:

# neutron port-list | grep 2.2.2.
| 1a593597-72de-4309-ada0-8e2dacba36ce |                                                 | fa:16:3e:c3:f8:fb | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.2"}        |
| 2007ca70-8cde-43bf-9254-66fd2e7fb327 |                                                 | fa:16:3e:ca:2a:c0 | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.1"}        |
| ba642200-e389-436d-aee8-a2198414f221 |                                                 | fa:16:3e:0b:5d:3d | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.3"}        |


# neutron port-show 2007ca70-8cde-43bf-9254-66fd2e7fb327
+-----------------------+--------------------------------------------------------------------------------+
| Field                 | Value                                                                          |
+-----------------------+--------------------------------------------------------------------------------+
| admin_state_up        | True                                                                           |
| allowed_address_pairs |                                                                                |
| binding:host_id       | controller2                                                                    |
| binding:profile       | {}                                                                             |
| binding:vif_details   | {"port_filter": true}                                                          |
| binding:vif_type      | bridge                                                                         |
| binding:vnic_type     | normal                                                                         |
| created_at            | 2017-02-01T22:10:39Z                                                           |
| description           |                                                                                |
| device_id             | c0f504ff-575e-4e7e-b25e-1f5ddc29a390                                           |
| device_owner          | network:ha_router_replicated_interface                                         |
| extra_dhcp_opts       |                                                                                |
| fixed_ips             | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.1"} |
| id                    | 2007ca70-8cde-43bf-9254-66fd2e7fb327                                           |
| mac_address           | fa:16:3e:ca:2a:c0                                                              |
| name                  |                                                                                |
| network_id            | 048938b7-b108-4f43-9222-3560f6d91fef                                           |
| port_security_enabled | False                                                                          |
| project_id            | 58a54da3f0404bc4ad7a266e73c9e7cc                                               |
| revision_number       | 68                                                                             |
| security_groups       |                                                                                |
| status                | ACTIVE                                                                         |
| tenant_id             | 58a54da3f0404bc4ad7a266e73c9e7cc                                               |
| updated_at            | 2017-02-03T05:09:49Z                                                           |
+-----------------------+--------------------------------------------------------------------------------+


In the log, the following debug message was seen and only 2.2.2.2 and 2.2.2.3 IPs are sent out by the l2pop notification agent.
/var/log/neutron/neutron-server.log
2017-02-01 22:20:03.925 35239 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req-f3f31e84-b43e-4393-9ac1-8da9c5c5952e - - - - -] Notify l2population agent compute1 at q-agent-notifier the message add_fdb_entries with {u'048938b7-b108-4f43-9222-3560f6d91fef': {'ports': {u'10.153.36.74': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:c3:f8:fb', ip_address=u'2.2.2.2')], u'10.153.36.75': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:0b:5d:3d', ip_address=u'2.2.2.3')]}, 'network_type': u'vxlan', 'segment_id': 18}} _notification_host /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/rpc.py:57

[root@compute1 ] # arp -an | grep 2.2.2
? (2.2.2.2) at fa:16:3e:c3:f8:fb [ether] PERM on vxlan-18
? (2.2.2.3) at fa:16:3e:0b:5d:3d [ether] PERM on vxlan-18

Looking at the code:
L2pop notification for these ports is sent here:
https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/l2pop/rpc.py#L51

Port info is gathered here for non-distributed HA router:
https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/l2pop/db.py#L99
Trace down from this line, it filters out ports that are in HA_ROUTER_PORTS, which HA_ROUTER_PORTS = (const.DEVICE_OWNER_HA_REPLICATED_INT, const.DEVICE_OWNER_ROUTER_SNAT)
>From the above we see that the port's device_owner is network:ha_router_replicated_interface, which would get filtered out by this.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1661717

Title:
  L2pop filters out port info for HA router internal interface when
  sending out notification

Status in neutron:
  New

Bug description:
  Branch: stable/newton

  Setup:
  l3_ha = True
  l2_population = True
  arp_responder = True
  enable_distributed_routing = False

  Node1: controller/network
  Node2: controller/network
  Node3: compute

  step 1: created external network
  step 2: created a router
  step 3: set router gateway to external network
  step 4: created internal network with subnet, for example: 2.2.2.0/24
  step 5: added internal network subnet to router using command: neutron router-interface-add <router> <subnet-id>
  step 6: created an instance (VM1) on the internal network
  Note: VM1 scheduled to boot on compute host or Node1

  Symptoms:
  Missing arp entry on Node1 for the internal interface on router
  VM1 does not know where to forward out going traffic

  Debugging:

  # neutron port-list | grep 2.2.2.
  | 1a593597-72de-4309-ada0-8e2dacba36ce |                                                 | fa:16:3e:c3:f8:fb | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.2"}        |
  | 2007ca70-8cde-43bf-9254-66fd2e7fb327 |                                                 | fa:16:3e:ca:2a:c0 | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.1"}        |
  | ba642200-e389-436d-aee8-a2198414f221 |                                                 | fa:16:3e:0b:5d:3d | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.3"}        |

  
  # neutron port-show 2007ca70-8cde-43bf-9254-66fd2e7fb327
  +-----------------------+--------------------------------------------------------------------------------+
  | Field                 | Value                                                                          |
  +-----------------------+--------------------------------------------------------------------------------+
  | admin_state_up        | True                                                                           |
  | allowed_address_pairs |                                                                                |
  | binding:host_id       | controller2                                                                    |
  | binding:profile       | {}                                                                             |
  | binding:vif_details   | {"port_filter": true}                                                          |
  | binding:vif_type      | bridge                                                                         |
  | binding:vnic_type     | normal                                                                         |
  | created_at            | 2017-02-01T22:10:39Z                                                           |
  | description           |                                                                                |
  | device_id             | c0f504ff-575e-4e7e-b25e-1f5ddc29a390                                           |
  | device_owner          | network:ha_router_replicated_interface                                         |
  | extra_dhcp_opts       |                                                                                |
  | fixed_ips             | {"subnet_id": "5f60e3c1-b176-4cc7-ba44-b236b3001b35", "ip_address": "2.2.2.1"} |
  | id                    | 2007ca70-8cde-43bf-9254-66fd2e7fb327                                           |
  | mac_address           | fa:16:3e:ca:2a:c0                                                              |
  | name                  |                                                                                |
  | network_id            | 048938b7-b108-4f43-9222-3560f6d91fef                                           |
  | port_security_enabled | False                                                                          |
  | project_id            | 58a54da3f0404bc4ad7a266e73c9e7cc                                               |
  | revision_number       | 68                                                                             |
  | security_groups       |                                                                                |
  | status                | ACTIVE                                                                         |
  | tenant_id             | 58a54da3f0404bc4ad7a266e73c9e7cc                                               |
  | updated_at            | 2017-02-03T05:09:49Z                                                           |
  +-----------------------+--------------------------------------------------------------------------------+

  
  In the log, the following debug message was seen and only 2.2.2.2 and 2.2.2.3 IPs are sent out by the l2pop notification agent.
  /var/log/neutron/neutron-server.log
  2017-02-01 22:20:03.925 35239 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req-f3f31e84-b43e-4393-9ac1-8da9c5c5952e - - - - -] Notify l2population agent compute1 at q-agent-notifier the message add_fdb_entries with {u'048938b7-b108-4f43-9222-3560f6d91fef': {'ports': {u'10.153.36.74': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:c3:f8:fb', ip_address=u'2.2.2.2')], u'10.153.36.75': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address=u'fa:16:3e:0b:5d:3d', ip_address=u'2.2.2.3')]}, 'network_type': u'vxlan', 'segment_id': 18}} _notification_host /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/rpc.py:57

  [root@compute1 ] # arp -an | grep 2.2.2
  ? (2.2.2.2) at fa:16:3e:c3:f8:fb [ether] PERM on vxlan-18
  ? (2.2.2.3) at fa:16:3e:0b:5d:3d [ether] PERM on vxlan-18

  Looking at the code:
  L2pop notification for these ports is sent here:
  https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/l2pop/rpc.py#L51

  Port info is gathered here for non-distributed HA router:
  https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/l2pop/db.py#L99
  Trace down from this line, it filters out ports that are in HA_ROUTER_PORTS, which HA_ROUTER_PORTS = (const.DEVICE_OWNER_HA_REPLICATED_INT, const.DEVICE_OWNER_ROUTER_SNAT)
  From the above we see that the port's device_owner is network:ha_router_replicated_interface, which would get filtered out by this.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1661717/+subscriptions


Follow ups