yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #38395
[Bug 1494866] [NEW] router allocation doesn't match the record in neutron db when l3_ha is true
Public bug reported:
We are using kilo. In our setup, we have 3 neutron controllers, l3
agents are running on all the 3 neutron controllers. We make l3_ha =
true in all the 3 neutron.conf.
We notice that when we attach a network to a router, the gateway
namespace is allocated to a controller node which doesn't match the
record in neutron db. Following is one example.
Create a router, a network, attach the network to the router.
1. neutron tells that the gateway ip 1.1.1.1 is at controller-1
[stack@c5220-01 ~]$ neutron port-show 3306c360-5a3d-4a08-aa92-017498758963
+-----------------------+--------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | overcloud-controller-1.localdomain |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| device_id | 934f0b90-2d98-4d54-b9ca-5222aac2199d |
| device_owner | network:router_interface |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "463c2f0c-5d56-4abb-8b30-8450d8306f46", "ip_address": "1.1.1.1"} |
| id | 3306c360-5a3d-4a08-aa92-017498758963 |
| mac_address | fa:16:3e:72:34:4c |
| name | |
| network_id | 98f125b6-6d4d-4417-a0b3-e8d9ff530d6f |
| security_groups | |
| status | ACTIVE |
| tenant_id | 4ef11838925940eb9d177ae9345711ee |
+-----------------------+--------------------------------------------------------------------------------+
2. However, the gateway ip is at controller-2
[heat-admin@overcloud-controller-2 ~]$ sudo ip netns exec qrouter-934f0b90-2d98-4d54-b9ca-5222aac2199d ifconfig
ha-6d47f13a-b7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.6 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::f816:3eff:fe43:9b80 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:43:9b:80 txqueuelen 1000 (Ethernet)
RX packets 20 bytes 1638 (1.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 309 bytes 16926 (16.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-22431202-eb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.8.87.25 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::f816:3eff:febd:56ad prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:bd:56:ad txqueuelen 1000 (Ethernet)
RX packets 36 bytes 2746 (2.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 43 bytes 2890 (2.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3306c360-5a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.1.1.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::f816:3eff:fe72:344c prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:72:34:4c txqueuelen 1000 (Ethernet)
RX packets 95 bytes 5856 (5.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 90 bytes 4200 (4.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
3. On controller-1, there is no such ip
[heat-admin@overcloud-controller-1 ~]$ sudo ip netns exec qrouter-934f0b90-2d98-4d54-b9ca-5222aac2199d ifconfig
ha-7ff9abd2-bd: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.5 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::f816:3eff:fe9d:275c prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:9d:27:5c txqueuelen 1000 (Ethernet)
RX packets 321 bytes 19678 (19.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 1008 (1008.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-22431202-eb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:bd:56:ad txqueuelen 1000 (Ethernet)
RX packets 42 bytes 3360 (3.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3306c360-5a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:72:34:4c txqueuelen 1000 (Ethernet)
RX packets 105 bytes 6456 (6.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
4. On controller-0, there is no such ip
[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qrouter-934f0b90-2d98-4d54-b9ca-5222aac2199d ifconfig
ha-8dccf24a-2e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.4 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::f816:3eff:fe98:83dd prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:98:83:dd txqueuelen 1000 (Ethernet)
RX packets 1140 bytes 68618 (67.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 1008 (1008.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-22431202-eb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:bd:56:ad txqueuelen 1000 (Ethernet)
RX packets 42 bytes 3244 (3.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3306c360-5a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:72:34:4c txqueuelen 1000 (Ethernet)
RX packets 1753 bytes 105336 (102.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1494866
Title:
router allocation doesn't match the record in neutron db when l3_ha is
true
Status in neutron:
New
Bug description:
We are using kilo. In our setup, we have 3 neutron controllers, l3
agents are running on all the 3 neutron controllers. We make l3_ha =
true in all the 3 neutron.conf.
We notice that when we attach a network to a router, the gateway
namespace is allocated to a controller node which doesn't match the
record in neutron db. Following is one example.
Create a router, a network, attach the network to the router.
1. neutron tells that the gateway ip 1.1.1.1 is at controller-1
[stack@c5220-01 ~]$ neutron port-show 3306c360-5a3d-4a08-aa92-017498758963
+-----------------------+--------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | overcloud-controller-1.localdomain |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| device_id | 934f0b90-2d98-4d54-b9ca-5222aac2199d |
| device_owner | network:router_interface |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "463c2f0c-5d56-4abb-8b30-8450d8306f46", "ip_address": "1.1.1.1"} |
| id | 3306c360-5a3d-4a08-aa92-017498758963 |
| mac_address | fa:16:3e:72:34:4c |
| name | |
| network_id | 98f125b6-6d4d-4417-a0b3-e8d9ff530d6f |
| security_groups | |
| status | ACTIVE |
| tenant_id | 4ef11838925940eb9d177ae9345711ee |
+-----------------------+--------------------------------------------------------------------------------+
2. However, the gateway ip is at controller-2
[heat-admin@overcloud-controller-2 ~]$ sudo ip netns exec qrouter-934f0b90-2d98-4d54-b9ca-5222aac2199d ifconfig
ha-6d47f13a-b7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.6 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::f816:3eff:fe43:9b80 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:43:9b:80 txqueuelen 1000 (Ethernet)
RX packets 20 bytes 1638 (1.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 309 bytes 16926 (16.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-22431202-eb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.8.87.25 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::f816:3eff:febd:56ad prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:bd:56:ad txqueuelen 1000 (Ethernet)
RX packets 36 bytes 2746 (2.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 43 bytes 2890 (2.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3306c360-5a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.1.1.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::f816:3eff:fe72:344c prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:72:34:4c txqueuelen 1000 (Ethernet)
RX packets 95 bytes 5856 (5.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 90 bytes 4200 (4.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
3. On controller-1, there is no such ip
[heat-admin@overcloud-controller-1 ~]$ sudo ip netns exec qrouter-934f0b90-2d98-4d54-b9ca-5222aac2199d ifconfig
ha-7ff9abd2-bd: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.5 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::f816:3eff:fe9d:275c prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:9d:27:5c txqueuelen 1000 (Ethernet)
RX packets 321 bytes 19678 (19.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 1008 (1008.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-22431202-eb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:bd:56:ad txqueuelen 1000 (Ethernet)
RX packets 42 bytes 3360 (3.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3306c360-5a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:72:34:4c txqueuelen 1000 (Ethernet)
RX packets 105 bytes 6456 (6.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
4. On controller-0, there is no such ip
[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qrouter-934f0b90-2d98-4d54-b9ca-5222aac2199d ifconfig
ha-8dccf24a-2e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.4 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::f816:3eff:fe98:83dd prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:98:83:dd txqueuelen 1000 (Ethernet)
RX packets 1140 bytes 68618 (67.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 1008 (1008.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-22431202-eb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:bd:56:ad txqueuelen 1000 (Ethernet)
RX packets 42 bytes 3244 (3.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-3306c360-5a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fa:16:3e:72:34:4c txqueuelen 1000 (Ethernet)
RX packets 1753 bytes 105336 (102.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 110 (110.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1494866/+subscriptions
Follow ups
