← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #39154

[Bug 1499647] [NEW] L3 HA: extra L3HARouterAgentPortBinding created for routers

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I have tested work of L3 HA on environment with 3 controllers and 1
compute (Kilo) keepalived v1.2.13 I create 50 nets with 50 subnets and
50 routers with interface is set for each subnet(Note: I've seem the
same errors with just one router and net). I've got the following
errors:

root@node-6:~# neutron l3-agent-list-hosting-router router-1
Request Failed: internal server error while processing your request.
 
In neutron-server error log:  http://paste.openstack.org/show/473760/

When I fixed _get_agents_dict_for_router to skip None for further
testing, so then I was able to see:

root@node-6:~# neutron l3-agent-list-hosting-router router-1
+--------------------------------------+-------------------+----------------+-------+----------+
| id                                   | host              | admin_state_up | alive | ha_state |
+--------------------------------------+-------------------+----------------+-------+----------+
| f3baba98-ef5d-41f8-8c74-a91b7016ba62 | node-6.domain.tld | True           | :-)   | active   |
| c9159f09-34d4-404f-b46c-a8c18df677f3 | node-7.domain.tld | True           | :-)   | standby  |
| b458ab49-c294-4bdb-91bf-ae375d87ff20 | node-8.domain.tld | True           | :-)   | standby  |
| f3baba98-ef5d-41f8-8c74-a91b7016ba62 | node-6.domain.tld | True           | :-)   | active   |
+--------------------------------------+-------------------+----------------+-------+----------+

root@node-6:~# neutron port-list --device_id=fcf150c0-f690-4265-974d-8db370e345c4
+--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name                                            | mac_address       | fixed_ips                                                                              |
+--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+
| 0834f8a2-f109-4060-9312-edebac84aba5 |                                                 | fa:16:3e:73:9f:33 | {"subnet_id": "0c7a2cfa-1cfd-4ecc-a196-ab9e97139352", "ip_address": "172.18.161.223"}  |
| 2b5a7a15-98a2-4ff1-9128-67d098fa3439 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:b8:f6:35 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.149"} |
| 48c887c1-acc3-4804-a993-b99060fa2c75 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:e7:70:13 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.151"} |
| 82ab62d6-7dd1-4294-a0dc-f5ebfbcbb4ca |                                                 | fa:16:3e:c6:fc:74 | {"subnet_id": "c4cc21c9-3b3a-407c-b4a7-b22f783377e7", "ip_address": "10.0.40.1"}       |
| bbca8575-51f1-4b42-b074-96e15aeda420 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:84:4c:fc | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.150"} |
| bee5c6d4-7e0a-4510-bb19-2ef9d60b9faf | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:09:a1:ae | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.193.11"}  |
| f8945a1d-b359-4c36-a8f8-e78c1ba992f0 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:c4:54:b5 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.193.12"}  |
+--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+
mysql root@192.168.0.2:neutron> SELECT * FROM ha_router_agent_port_bindings WHERE router_id='fcf150c0-f690-4265-974d-8db370e345c4';
+--------------------------------------+--------------------------------------+--------------------------------------+---------+
| port_id                              | router_id                            | l3_agent_id                          | state   |
|--------------------------------------+--------------------------------------+--------------------------------------+---------|
| 2b5a7a15-98a2-4ff1-9128-67d098fa3439 | fcf150c0-f690-4265-974d-8db370e345c4 | c9159f09-34d4-404f-b46c-a8c18df677f3 | standby |
| 48c887c1-acc3-4804-a993-b99060fa2c75 | fcf150c0-f690-4265-974d-8db370e345c4 | b458ab49-c294-4bdb-91bf-ae375d87ff20 | standby |
| bbca8575-51f1-4b42-b074-96e15aeda420 | fcf150c0-f690-4265-974d-8db370e345c4 | <null>                               | standby |
| bee5c6d4-7e0a-4510-bb19-2ef9d60b9faf | fcf150c0-f690-4265-974d-8db370e345c4 | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | active  |
| f8945a1d-b359-4c36-a8f8-e78c1ba992f0 | fcf150c0-f690-4265-974d-8db370e345c4 | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | active  |
+--------------------------------------+--------------------------------------+--------------------------------------+---------+

So extra L3HARouterAgentPortBinding was created for routers. This issue does not reproduced all the time.
During sync_routers the following errors in logs appeared:

http://paste.openstack.org/show/473839/
http://paste.openstack.org/show/473840/

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-ha

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1499647

Title:
  L3 HA: extra L3HARouterAgentPortBinding created for routers

Status in neutron:
  New

Bug description:
  I have tested work of L3 HA on environment with 3 controllers and 1
  compute (Kilo) keepalived v1.2.13 I create 50 nets with 50 subnets and
  50 routers with interface is set for each subnet(Note: I've seem the
  same errors with just one router and net). I've got the following
  errors:

  root@node-6:~# neutron l3-agent-list-hosting-router router-1
  Request Failed: internal server error while processing your request.
   
  In neutron-server error log:  http://paste.openstack.org/show/473760/

  When I fixed _get_agents_dict_for_router to skip None for further
  testing, so then I was able to see:

  root@node-6:~# neutron l3-agent-list-hosting-router router-1
  +--------------------------------------+-------------------+----------------+-------+----------+
  | id                                   | host              | admin_state_up | alive | ha_state |
  +--------------------------------------+-------------------+----------------+-------+----------+
  | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | node-6.domain.tld | True           | :-)   | active   |
  | c9159f09-34d4-404f-b46c-a8c18df677f3 | node-7.domain.tld | True           | :-)   | standby  |
  | b458ab49-c294-4bdb-91bf-ae375d87ff20 | node-8.domain.tld | True           | :-)   | standby  |
  | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | node-6.domain.tld | True           | :-)   | active   |
  +--------------------------------------+-------------------+----------------+-------+----------+

  root@node-6:~# neutron port-list --device_id=fcf150c0-f690-4265-974d-8db370e345c4
  +--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+
  | id                                   | name                                            | mac_address       | fixed_ips                                                                              |
  +--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+
  | 0834f8a2-f109-4060-9312-edebac84aba5 |                                                 | fa:16:3e:73:9f:33 | {"subnet_id": "0c7a2cfa-1cfd-4ecc-a196-ab9e97139352", "ip_address": "172.18.161.223"}  |
  | 2b5a7a15-98a2-4ff1-9128-67d098fa3439 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:b8:f6:35 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.149"} |
  | 48c887c1-acc3-4804-a993-b99060fa2c75 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:e7:70:13 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.151"} |
  | 82ab62d6-7dd1-4294-a0dc-f5ebfbcbb4ca |                                                 | fa:16:3e:c6:fc:74 | {"subnet_id": "c4cc21c9-3b3a-407c-b4a7-b22f783377e7", "ip_address": "10.0.40.1"}       |
  | bbca8575-51f1-4b42-b074-96e15aeda420 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:84:4c:fc | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.192.150"} |
  | bee5c6d4-7e0a-4510-bb19-2ef9d60b9faf | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:09:a1:ae | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.193.11"}  |
  | f8945a1d-b359-4c36-a8f8-e78c1ba992f0 | HA port tenant aef8d13bad9d42df9f25d8ee54c80ad6 | fa:16:3e:c4:54:b5 | {"subnet_id": "1915ccb8-9d0f-4f1a-9811-9a196d1e495e", "ip_address": "169.254.193.12"}  |
  +--------------------------------------+-------------------------------------------------+-------------------+----------------------------------------------------------------------------------------+
  mysql root@192.168.0.2:neutron> SELECT * FROM ha_router_agent_port_bindings WHERE router_id='fcf150c0-f690-4265-974d-8db370e345c4';
  +--------------------------------------+--------------------------------------+--------------------------------------+---------+
  | port_id                              | router_id                            | l3_agent_id                          | state   |
  |--------------------------------------+--------------------------------------+--------------------------------------+---------|
  | 2b5a7a15-98a2-4ff1-9128-67d098fa3439 | fcf150c0-f690-4265-974d-8db370e345c4 | c9159f09-34d4-404f-b46c-a8c18df677f3 | standby |
  | 48c887c1-acc3-4804-a993-b99060fa2c75 | fcf150c0-f690-4265-974d-8db370e345c4 | b458ab49-c294-4bdb-91bf-ae375d87ff20 | standby |
  | bbca8575-51f1-4b42-b074-96e15aeda420 | fcf150c0-f690-4265-974d-8db370e345c4 | <null>                               | standby |
  | bee5c6d4-7e0a-4510-bb19-2ef9d60b9faf | fcf150c0-f690-4265-974d-8db370e345c4 | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | active  |
  | f8945a1d-b359-4c36-a8f8-e78c1ba992f0 | fcf150c0-f690-4265-974d-8db370e345c4 | f3baba98-ef5d-41f8-8c74-a91b7016ba62 | active  |
  +--------------------------------------+--------------------------------------+--------------------------------------+---------+

  So extra L3HARouterAgentPortBinding was created for routers. This issue does not reproduced all the time.
  During sync_routers the following errors in logs appeared:

  http://paste.openstack.org/show/473839/
  http://paste.openstack.org/show/473840/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1499647/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next