← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1830349] [NEW] Router external gateway wrongly marked as DOWN

 

Public bug reported:

neutron version: 2:8.4.0-0ubuntu7.3~cloud0
openstack version: cloud:trusty-mitaka

In bootstack a customer had a non-ha router.
After updating the router to HA mode,
it is external gateway is wrongly marked as Down,
but we can see traffic going through the interface:

openstack router show  7d7a37e0-33f3-474f-adbf-ab27033c6bc8
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                                                                                                                                                                                                                                                                          |
| availability_zone_hints |                                                                                                                                                                                                                                                                                                                                                                                                                             |
| availability_zones      | nova                                                                                                                                                                                                                                                                                                                                                                                                                        |
| created_at              | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
| description             |                                                                                                                                                                                                                                                                                                                                                                                                                             |
| distributed             | False                                                                                                                                                                                                                                                                                                                                                                                                                       |
| external_gateway_info   | {"enable_snat": true, "external_fixed_ips": [{"subnet_id": "dbfee73f-7094-4596-a79c-e05c2ce7d738", "ip_address": "185.170.7.198"}], "network_id": "43c6a5c6-d44c-43d9-a0e9-1c0311b41626"}                                                                                                                                                                                                                                   |
| flavor_id               | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ha                      | True                                                                                                                                                                                                                                                                                                                                                                                                                        |
| id                      | 7d7a37e0-33f3-474f-adbf-ab27033c6bc8                                                                                                                                                                                                                                                                                                                                                                                        |
| interfaces_info         | [{"ip_address": "169.254.192.6", "port_id": "bd24f0cc-8f8c-4e79-8a5d-eec0ad841153", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}, {"ip_address": "169.254.192.4", "port_id": "d0bbad51-5755-4715-989b-2e2f81377185", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}, {"ip_address": "169.254.192.5", "port_id": "fac3d9b3-fc31-4b29-9fef-45778ec9f37f", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}] |
| name                    | hatestrouter                                                                                                                                                                                                                                                                                                                                                                                                                |
| project_id              | 1452d7e0d0df4e94bf35543cc78f157a                                                                                                                                                                                                                                                                                                                                                                                            |
| revision_number         | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
| routes                  |                                                                                                                                                                                                                                                                                                                                                                                                                             |
| status                  | ACTIVE                                                                                                                                                                                                                                                                                                                                                                                                                      |
| tags                    |                                                                                                                                                                                                                                                                                                                                                                                                                             |
| updated_at              | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack port show aafef3b0-08dd-4cb7-97b4-be18d1aa95b2
+-----------------------+------------------------------------------------------------------------------+
| Field                 | Value                                                                        |
+-----------------------+------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                           |
| allowed_address_pairs |                                                                              |
| binding_host_id       | juju-9a8e29-2-lxd-6                                                          |
| binding_profile       |                                                                              |
| binding_vif_details   | ovs_hybrid_plug='True', port_filter='True'                                   |
| binding_vif_type      | ovs                                                                          |
| binding_vnic_type     | normal                                                                       |
| created_at            | 2019-05-14T12:15:04                                                          |
| data_plane_status     | None                                                                         |
| description           |                                                                              |
| device_id             | 7d7a37e0-33f3-474f-adbf-ab27033c6bc8                                         |
| device_owner          | network:router_gateway                                                       |
| dns_assignment        | None                                                                         |
| dns_domain            | None                                                                         |
| dns_name              | None                                                                         |
| extra_dhcp_opts       |                                                                              |
| fixed_ips             | ip_address='185.170.7.198', subnet_id='dbfee73f-7094-4596-a79c-e05c2ce7d738' |
| id                    | aafef3b0-08dd-4cb7-97b4-be18d1aa95b2                                         |
| mac_address           | fa:16:3e:29:53:c0                                                            |
| name                  |                                                                              |
| network_id            | 43c6a5c6-d44c-43d9-a0e9-1c0311b41626                                         |
| port_security_enabled | False                                                                        |
| project_id            |                                                                              |
| qos_policy_id         | None                                                                         |
| revision_number       | None                                                                         |
| security_group_ids    |                                                                              |
| status                | DOWN                                                                         |
| tags                  |                                                                              |
| trunk_details         | None                                                                         |
| updated_at            | 2019-05-24T08:17:44                                                          |
+-----------------------+------------------------------------------------------------------------------+
$ openstack subnet show dbfee73f-7094-4596-a79c-e05c2ce7d738
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 185.170.6.2-185.170.7.253            |
| cidr              | 185.170.4.0/22                       |
| created_at        | 2017-02-07T00:39:23                  |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | False                                |
| gateway_ip        | 185.170.6.1                          |
| host_routes       |                                      |
| id                | dbfee73f-7094-4596-a79c-e05c2ce7d738 |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | subnet_ext_net                       |
| network_id        | 43c6a5c6-d44c-43d9-a0e9-1c0311b41626 |
| project_id        | 5f182a7b537a4aa7b40832ed4bbbfbed     |
| revision_number   | None                                 |
| segment_id        | None                                 |
| service_types     | None                                 |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2017-02-07T00:39:23                  |
+-------------------+--------------------------------------+

$ ping -c 5 185.170.7.198
PING 185.170.7.198 (185.170.7.198) 56(84) bytes of data.
64 bytes from 185.170.7.198: icmp_seq=1 ttl=62 time=0.594 ms
64 bytes from 185.170.7.198: icmp_seq=2 ttl=62 time=0.354 ms
64 bytes from 185.170.7.198: icmp_seq=3 ttl=62 time=0.379 ms
64 bytes from 185.170.7.198: icmp_seq=4 ttl=62 time=0.366 ms
64 bytes from 185.170.7.198: icmp_seq=5 ttl=62 time=0.308 ms

--- 185.170.7.198 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4000ms
rtt min/avg/max/mdev = 0.308/0.400/0.594/0.100 ms

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

+ neutron version: 2:8.4.0-0ubuntu7.3~cloud0
+ openstack version: cloud:trusty-mitaka
+ 
  In bootstack a customer had a non-ha router.
  After updating the router to HA mode,
  it is external gateway is wrongly marked as Down,
  but we can see traffic going through the interface:
  
  openstack router show  7d7a37e0-33f3-474f-adbf-ab27033c6bc8
  +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field                   | Value                                                                                                                                                                                                                                                                                                                                                                                                                       |
  +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | admin_state_up          | UP                                                                                                                                                                                                                                                                                                                                                                                                                          |
  | availability_zone_hints |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | availability_zones      | nova                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | created_at              | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | description             |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | distributed             | False                                                                                                                                                                                                                                                                                                                                                                                                                       |
  | external_gateway_info   | {"enable_snat": true, "external_fixed_ips": [{"subnet_id": "dbfee73f-7094-4596-a79c-e05c2ce7d738", "ip_address": "185.170.7.198"}], "network_id": "43c6a5c6-d44c-43d9-a0e9-1c0311b41626"}                                                                                                                                                                                                                                   |
  | flavor_id               | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | ha                      | True                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | id                      | 7d7a37e0-33f3-474f-adbf-ab27033c6bc8                                                                                                                                                                                                                                                                                                                                                                                        |
  | interfaces_info         | [{"ip_address": "169.254.192.6", "port_id": "bd24f0cc-8f8c-4e79-8a5d-eec0ad841153", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}, {"ip_address": "169.254.192.4", "port_id": "d0bbad51-5755-4715-989b-2e2f81377185", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}, {"ip_address": "169.254.192.5", "port_id": "fac3d9b3-fc31-4b29-9fef-45778ec9f37f", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}] |
  | name                    | hatestrouter                                                                                                                                                                                                                                                                                                                                                                                                                |
  | project_id              | 1452d7e0d0df4e94bf35543cc78f157a                                                                                                                                                                                                                                                                                                                                                                                            |
  | revision_number         | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | routes                  |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | status                  | ACTIVE                                                                                                                                                                                                                                                                                                                                                                                                                      |
  | tags                    |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | updated_at              | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- 
  
  openstack port show aafef3b0-08dd-4cb7-97b4-be18d1aa95b2
  +-----------------------+------------------------------------------------------------------------------+
  | Field                 | Value                                                                        |
  +-----------------------+------------------------------------------------------------------------------+
  | admin_state_up        | UP                                                                           |
  | allowed_address_pairs |                                                                              |
  | binding_host_id       | juju-9a8e29-2-lxd-6                                                          |
  | binding_profile       |                                                                              |
  | binding_vif_details   | ovs_hybrid_plug='True', port_filter='True'                                   |
  | binding_vif_type      | ovs                                                                          |
  | binding_vnic_type     | normal                                                                       |
  | created_at            | 2019-05-14T12:15:04                                                          |
  | data_plane_status     | None                                                                         |
  | description           |                                                                              |
  | device_id             | 7d7a37e0-33f3-474f-adbf-ab27033c6bc8                                         |
  | device_owner          | network:router_gateway                                                       |
  | dns_assignment        | None                                                                         |
  | dns_domain            | None                                                                         |
  | dns_name              | None                                                                         |
  | extra_dhcp_opts       |                                                                              |
  | fixed_ips             | ip_address='185.170.7.198', subnet_id='dbfee73f-7094-4596-a79c-e05c2ce7d738' |
  | id                    | aafef3b0-08dd-4cb7-97b4-be18d1aa95b2                                         |
  | mac_address           | fa:16:3e:29:53:c0                                                            |
  | name                  |                                                                              |
  | network_id            | 43c6a5c6-d44c-43d9-a0e9-1c0311b41626                                         |
  | port_security_enabled | False                                                                        |
  | project_id            |                                                                              |
  | qos_policy_id         | None                                                                         |
  | revision_number       | None                                                                         |
  | security_group_ids    |                                                                              |
  | status                | DOWN                                                                         |
  | tags                  |                                                                              |
  | trunk_details         | None                                                                         |
  | updated_at            | 2019-05-24T08:17:44                                                          |
  +-----------------------+------------------------------------------------------------------------------+
  $ openstack subnet show dbfee73f-7094-4596-a79c-e05c2ce7d738
  +-------------------+--------------------------------------+
  | Field             | Value                                |
  +-------------------+--------------------------------------+
  | allocation_pools  | 185.170.6.2-185.170.7.253            |
  | cidr              | 185.170.4.0/22                       |
  | created_at        | 2017-02-07T00:39:23                  |
  | description       |                                      |
  | dns_nameservers   |                                      |
  | enable_dhcp       | False                                |
  | gateway_ip        | 185.170.6.1                          |
  | host_routes       |                                      |
  | id                | dbfee73f-7094-4596-a79c-e05c2ce7d738 |
  | ip_version        | 4                                    |
  | ipv6_address_mode | None                                 |
  | ipv6_ra_mode      | None                                 |
  | name              | subnet_ext_net                       |
  | network_id        | 43c6a5c6-d44c-43d9-a0e9-1c0311b41626 |
  | project_id        | 5f182a7b537a4aa7b40832ed4bbbfbed     |
  | revision_number   | None                                 |
  | segment_id        | None                                 |
  | service_types     | None                                 |
  | subnetpool_id     | None                                 |
  | tags              |                                      |
  | updated_at        | 2017-02-07T00:39:23                  |
  +-------------------+--------------------------------------+
  
  $ ping -c 5 185.170.7.198
  PING 185.170.7.198 (185.170.7.198) 56(84) bytes of data.
  64 bytes from 185.170.7.198: icmp_seq=1 ttl=62 time=0.594 ms
  64 bytes from 185.170.7.198: icmp_seq=2 ttl=62 time=0.354 ms
  64 bytes from 185.170.7.198: icmp_seq=3 ttl=62 time=0.379 ms
  64 bytes from 185.170.7.198: icmp_seq=4 ttl=62 time=0.366 ms
  64 bytes from 185.170.7.198: icmp_seq=5 ttl=62 time=0.308 ms
  
  --- 185.170.7.198 ping statistics ---
  5 packets transmitted, 5 received, 0% packet loss, time 4000ms
  rtt min/avg/max/mdev = 0.308/0.400/0.594/0.100 ms

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1830349

Title:
  Router external gateway wrongly marked as DOWN

Status in neutron:
  New

Bug description:
  neutron version: 2:8.4.0-0ubuntu7.3~cloud0
  openstack version: cloud:trusty-mitaka

  In bootstack a customer had a non-ha router.
  After updating the router to HA mode,
  it is external gateway is wrongly marked as Down,
  but we can see traffic going through the interface:

  openstack router show  7d7a37e0-33f3-474f-adbf-ab27033c6bc8
  +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field                   | Value                                                                                                                                                                                                                                                                                                                                                                                                                       |
  +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | admin_state_up          | UP                                                                                                                                                                                                                                                                                                                                                                                                                          |
  | availability_zone_hints |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | availability_zones      | nova                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | created_at              | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | description             |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | distributed             | False                                                                                                                                                                                                                                                                                                                                                                                                                       |
  | external_gateway_info   | {"enable_snat": true, "external_fixed_ips": [{"subnet_id": "dbfee73f-7094-4596-a79c-e05c2ce7d738", "ip_address": "185.170.7.198"}], "network_id": "43c6a5c6-d44c-43d9-a0e9-1c0311b41626"}                                                                                                                                                                                                                                   |
  | flavor_id               | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | ha                      | True                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | id                      | 7d7a37e0-33f3-474f-adbf-ab27033c6bc8                                                                                                                                                                                                                                                                                                                                                                                        |
  | interfaces_info         | [{"ip_address": "169.254.192.6", "port_id": "bd24f0cc-8f8c-4e79-8a5d-eec0ad841153", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}, {"ip_address": "169.254.192.4", "port_id": "d0bbad51-5755-4715-989b-2e2f81377185", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}, {"ip_address": "169.254.192.5", "port_id": "fac3d9b3-fc31-4b29-9fef-45778ec9f37f", "subnet_id": "4cdfece4-4ef0-4172-9151-6177ebe6390b"}] |
  | name                    | hatestrouter                                                                                                                                                                                                                                                                                                                                                                                                                |
  | project_id              | 1452d7e0d0df4e94bf35543cc78f157a                                                                                                                                                                                                                                                                                                                                                                                            |
  | revision_number         | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  | routes                  |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | status                  | ACTIVE                                                                                                                                                                                                                                                                                                                                                                                                                      |
  | tags                    |                                                                                                                                                                                                                                                                                                                                                                                                                             |
  | updated_at              | None                                                                                                                                                                                                                                                                                                                                                                                                                        |
  +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

  openstack port show aafef3b0-08dd-4cb7-97b4-be18d1aa95b2
  +-----------------------+------------------------------------------------------------------------------+
  | Field                 | Value                                                                        |
  +-----------------------+------------------------------------------------------------------------------+
  | admin_state_up        | UP                                                                           |
  | allowed_address_pairs |                                                                              |
  | binding_host_id       | juju-9a8e29-2-lxd-6                                                          |
  | binding_profile       |                                                                              |
  | binding_vif_details   | ovs_hybrid_plug='True', port_filter='True'                                   |
  | binding_vif_type      | ovs                                                                          |
  | binding_vnic_type     | normal                                                                       |
  | created_at            | 2019-05-14T12:15:04                                                          |
  | data_plane_status     | None                                                                         |
  | description           |                                                                              |
  | device_id             | 7d7a37e0-33f3-474f-adbf-ab27033c6bc8                                         |
  | device_owner          | network:router_gateway                                                       |
  | dns_assignment        | None                                                                         |
  | dns_domain            | None                                                                         |
  | dns_name              | None                                                                         |
  | extra_dhcp_opts       |                                                                              |
  | fixed_ips             | ip_address='185.170.7.198', subnet_id='dbfee73f-7094-4596-a79c-e05c2ce7d738' |
  | id                    | aafef3b0-08dd-4cb7-97b4-be18d1aa95b2                                         |
  | mac_address           | fa:16:3e:29:53:c0                                                            |
  | name                  |                                                                              |
  | network_id            | 43c6a5c6-d44c-43d9-a0e9-1c0311b41626                                         |
  | port_security_enabled | False                                                                        |
  | project_id            |                                                                              |
  | qos_policy_id         | None                                                                         |
  | revision_number       | None                                                                         |
  | security_group_ids    |                                                                              |
  | status                | DOWN                                                                         |
  | tags                  |                                                                              |
  | trunk_details         | None                                                                         |
  | updated_at            | 2019-05-24T08:17:44                                                          |
  +-----------------------+------------------------------------------------------------------------------+
  $ openstack subnet show dbfee73f-7094-4596-a79c-e05c2ce7d738
  +-------------------+--------------------------------------+
  | Field             | Value                                |
  +-------------------+--------------------------------------+
  | allocation_pools  | 185.170.6.2-185.170.7.253            |
  | cidr              | 185.170.4.0/22                       |
  | created_at        | 2017-02-07T00:39:23                  |
  | description       |                                      |
  | dns_nameservers   |                                      |
  | enable_dhcp       | False                                |
  | gateway_ip        | 185.170.6.1                          |
  | host_routes       |                                      |
  | id                | dbfee73f-7094-4596-a79c-e05c2ce7d738 |
  | ip_version        | 4                                    |
  | ipv6_address_mode | None                                 |
  | ipv6_ra_mode      | None                                 |
  | name              | subnet_ext_net                       |
  | network_id        | 43c6a5c6-d44c-43d9-a0e9-1c0311b41626 |
  | project_id        | 5f182a7b537a4aa7b40832ed4bbbfbed     |
  | revision_number   | None                                 |
  | segment_id        | None                                 |
  | service_types     | None                                 |
  | subnetpool_id     | None                                 |
  | tags              |                                      |
  | updated_at        | 2017-02-07T00:39:23                  |
  +-------------------+--------------------------------------+

  $ ping -c 5 185.170.7.198
  PING 185.170.7.198 (185.170.7.198) 56(84) bytes of data.
  64 bytes from 185.170.7.198: icmp_seq=1 ttl=62 time=0.594 ms
  64 bytes from 185.170.7.198: icmp_seq=2 ttl=62 time=0.354 ms
  64 bytes from 185.170.7.198: icmp_seq=3 ttl=62 time=0.379 ms
  64 bytes from 185.170.7.198: icmp_seq=4 ttl=62 time=0.366 ms
  64 bytes from 185.170.7.198: icmp_seq=5 ttl=62 time=0.308 ms

  --- 185.170.7.198 ping statistics ---
  5 packets transmitted, 5 received, 0% packet loss, time 4000ms
  rtt min/avg/max/mdev = 0.308/0.400/0.594/0.100 ms

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1830349/+subscriptions


Follow ups