yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #93026
[Bug 2041612] [NEW] [n-d-r] DRAgent removed from bgp speaker automatically
Public bug reported:
Hello everyone,
I noticed a BUG in the Neutron agent reschedule process when I'm running
the 'BGP dynamic routing agent'.
I tried to found some n-d-r related documentation about the scheduler
process for L3 agents, but I've just saw routers context topics [1].
The main problem is the FSM of the L3 reschedule method, since it was
developed for router specifc purposes, and the n-d-r has a different
context. To be clear, when this reschedule fails for the DRAgent, an
administrative action (bgp_speaker association in a DRAgent) will be
automatically removed from the Neutron configuration, and even if the
agent comes back, it will no longer be associated with the speaker.
The reschedule_resources_from_down_agents method was develop to
"Reschedule routers from down l3 agents if admin state is up", and this
same method calls reschedule_resources_from_down_agents [2].
Basically, the problem ocurrs after '4 * agent_down_time' because the
reschedule process starts '2 * agent_down_time' without agent
communication [3], and try to reschedule until more 2 * agent_down_time
[4].
Ofcouse this problem only occurs because the DRAgent has not established
communication with Neutron server in the meantime of the reschedule
process. The root cause was a RabbitMQ and/or MySQL issue, but it does
make sense to me that the speaker is removed from the DRAgent,
considering that the link between the BGP speaker and the DRAgent is an
administrative action.
Logs about the occurrence of the BUG below:
For reference: agent_down_time = 1200
Oct 20 09:06:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:06:15.563 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 09:07:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:07:05.630 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
Oct 20 09:16:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:16:15.563 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 09:17:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:17:05.588 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
Oct 20 09:26:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:26:15.572 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 09:27:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:27:05.591 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
..........
Oct 20 10:56:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 10:56:15.758 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 10:57:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 10:57:05.781 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
After that, DRAgent was permanently removed from the BGP speaker.
[1] https://docs.openstack.org/neutron/latest/admin/config-services-agent.html#l3-agent
[2] https://opendev.org/openstack/neutron/src/commit/53f4fd6b9fcb4f8ba907bfbace342bf902fc55f7/neutron/db/l3_agentschedulers_db.py#L65
[3] https://opendev.org/openstack/neutron/src/commit/f90d7d2a9e49688f7baee13268c3ba34f59d5a16/neutron/db/agentschedulers_db.py#L123
[4] https://opendev.org/openstack/neutron/src/commit/f90d7d2a9e49688f7baee13268c3ba34f59d5a16/neutron/db/agentschedulers_db.py#L146
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
Hello everyone,
I noticed a BUG in the Neutron agent reschedule process when I'm running
the 'BGP dynamic routing agent'.
I tried to found some n-d-r related documentation about the scheduler
process for L3 agents, but I've just saw routers context topics [1].
The main problem is the FSM of the L3 reschedule method, since it was
developed for router specifc purposes, and the n-d-r has a different
context. To be clear, when this reschedule fails for the DRAgent, an
administrative action (bgp_speaker association in a DRAgent) will be
automatically removed from the Neutron configuration, and even if the
agent comes back, it will no longer be associated with the speaker.
The reschedule_resources_from_down_agents method was develop to
"Reschedule routers from down l3 agents if admin state is up", and this
same method calls reschedule_resources_from_down_agents [2].
Basically, the problem ocurrs after '4 * agent_down_time' because the
reschedule process starts '2 * agent_down_time' without agent
communication [3], and try to reschedule until more 2 * agent_down_time
[4].
Ofcouse this problem only occurs because the DRAgent has not established
communication with Neutron server in the meantime of the reschedule
process. The root cause was a RabbitMQ and/or MySQL issue, but it does
make sense to me that the speaker is removed from the DRAgent,
considering that the link between the BGP speaker and the DRAgent is an
administrative action.
-
Logs about the occurrence of the BUG below:
For reference: agent_down_time = 1200
Oct 20 09:06:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:06:15.563 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 09:07:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:07:05.630 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
Oct 20 09:16:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:16:15.563 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 09:17:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:17:05.588 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
Oct 20 09:26:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:26:15.572 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 09:27:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 09:27:05.591 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
..........
-
Oct 20 10:56:15 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 10:56:15.758 113 WARNING
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Rescheduling bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089
from agent 424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did
not report to the server in the last 2400 seconds.
Oct 20 10:57:05 srv-neutron-server-container-9b119379.1a.ne.jaxyendy.com
neutron-server[113]: 2023-10-20 10:57:05.781 113 ERROR
neutron.db.agentschedulers_db [req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392
- - - - -] Exception encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id = %(agent_id_1)s]#012[parameters:
{'bgp_speaker_id_1': '6addb3dc-7282-4be2-9f40-b234563c1089',
'agent_id_1': '424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
After that, DRAgent was permanently removed from the BGP speaker.
+
+
+ [1] https://docs.openstack.org/neutron/latest/admin/config-services-agent.html#l3-agent
+ [2] https://opendev.org/openstack/neutron/src/commit/53f4fd6b9fcb4f8ba907bfbace342bf902fc55f7/neutron/db/l3_agentschedulers_db.py#L65
+ [3] https://opendev.org/openstack/neutron/src/commit/f90d7d2a9e49688f7baee13268c3ba34f59d5a16/neutron/db/agentschedulers_db.py#L123
+ [4] https://opendev.org/openstack/neutron/src/commit/f90d7d2a9e49688f7baee13268c3ba34f59d5a16/neutron/db/agentschedulers_db.py#L146
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2041612
Title:
[n-d-r] DRAgent removed from bgp speaker automatically
Status in neutron:
New
Bug description:
Hello everyone,
I noticed a BUG in the Neutron agent reschedule process when I'm
running the 'BGP dynamic routing agent'.
I tried to found some n-d-r related documentation about the scheduler
process for L3 agents, but I've just saw routers context topics [1].
The main problem is the FSM of the L3 reschedule method, since it was
developed for router specifc purposes, and the n-d-r has a different
context. To be clear, when this reschedule fails for the DRAgent, an
administrative action (bgp_speaker association in a DRAgent) will be
automatically removed from the Neutron configuration, and even if the
agent comes back, it will no longer be associated with the speaker.
The reschedule_resources_from_down_agents method was develop to
"Reschedule routers from down l3 agents if admin state is up", and
this same method calls reschedule_resources_from_down_agents [2].
Basically, the problem ocurrs after '4 * agent_down_time' because the
reschedule process starts '2 * agent_down_time' without agent
communication [3], and try to reschedule until more 2 *
agent_down_time [4].
Ofcouse this problem only occurs because the DRAgent has not
established communication with Neutron server in the meantime of the
reschedule process. The root cause was a RabbitMQ and/or MySQL issue,
but it does make sense to me that the speaker is removed from the
DRAgent, considering that the link between the BGP speaker and the
DRAgent is an administrative action.
Logs about the occurrence of the BUG below:
For reference: agent_down_time = 1200
Oct 20 09:06:15 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
09:06:15.563 113 WARNING neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Rescheduling
bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089 from agent
424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did not report
to the server in the last 2400 seconds.
Oct 20 09:07:05 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
09:07:05.630 113 ERROR neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Exception
encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id =
%(agent_id_1)s]#012[parameters: {'bgp_speaker_id_1':
'6addb3dc-7282-4be2-9f40-b234563c1089', 'agent_id_1':
'424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
Oct 20 09:16:15 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
09:16:15.563 113 WARNING neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Rescheduling
bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089 from agent
424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did not report
to the server in the last 2400 seconds.
Oct 20 09:17:05 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
09:17:05.588 113 ERROR neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Exception
encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id =
%(agent_id_1)s]#012[parameters: {'bgp_speaker_id_1':
'6addb3dc-7282-4be2-9f40-b234563c1089', 'agent_id_1':
'424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
Oct 20 09:26:15 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
09:26:15.572 113 WARNING neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Rescheduling
bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089 from agent
424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did not report
to the server in the last 2400 seconds.
Oct 20 09:27:05 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
09:27:05.591 113 ERROR neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Exception
encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id =
%(agent_id_1)s]#012[parameters: {'bgp_speaker_id_1':
'6addb3dc-7282-4be2-9f40-b234563c1089', 'agent_id_1':
'424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
..........
Oct 20 10:56:15 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
10:56:15.758 113 WARNING neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Rescheduling
bgp_speaker 6addb3dc-7282-4be2-9f40-b234563c1089 from agent
424ad86f-164c-454b-ad64-6f9f7ab2fb72 because the agent did not report
to the server in the last 2400 seconds.
Oct 20 10:57:05 srv-neutron-server-
container-9b119379.1a.ne.jaxyendy.com neutron-server[113]: 2023-10-20
10:57:05.781 113 ERROR neutron.db.agentschedulers_db
[req-8e46c4f2-bfeb-4fe7-8726-ea3fd1d76392 - - - - -] Exception
encountered during bgp_speaker rescheduling.:
oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205,
'Lock wait timeout exceeded; try restarting transaction')#012[SQL:
DELETE FROM bgp_speaker_dragent_bindings WHERE
bgp_speaker_dragent_bindings.bgp_speaker_id = %(bgp_speaker_id_1)s AND
bgp_speaker_dragent_bindings.agent_id =
%(agent_id_1)s]#012[parameters: {'bgp_speaker_id_1':
'6addb3dc-7282-4be2-9f40-b234563c1089', 'agent_id_1':
'424ad86f-164c-454b-ad64-6f9f7ab2fb72'}]
After that, DRAgent was permanently removed from the BGP speaker.
[1] https://docs.openstack.org/neutron/latest/admin/config-services-agent.html#l3-agent
[2] https://opendev.org/openstack/neutron/src/commit/53f4fd6b9fcb4f8ba907bfbace342bf902fc55f7/neutron/db/l3_agentschedulers_db.py#L65
[3] https://opendev.org/openstack/neutron/src/commit/f90d7d2a9e49688f7baee13268c3ba34f59d5a16/neutron/db/agentschedulers_db.py#L123
[4] https://opendev.org/openstack/neutron/src/commit/f90d7d2a9e49688f7baee13268c3ba34f59d5a16/neutron/db/agentschedulers_db.py#L146
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2041612/+subscriptions