← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #88756

[Bug 1970759] [NEW] Race during removal of the network from DHCP agent

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

It may happen that 2 neutron-server workers will try to remove network
from the DHCP agent in almost the same way. On one of them all will work
fine:

2022-04-19 18:51:26.579 50 WARNING neutron.db.agentschedulers_db
[req-73439ae5-d6a4-432c-92f0-872cf5d19bec - - - - -] Removing network
d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to
the server in the last 150 seconds.

but on second one it will fail with error:

2022-04-19 18:51:26.634 50 WARNING neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to the server in the last 150 seconds.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Unexpected exception occurred while removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947: neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db Traceback (most recent call last):
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 342, in remove_networks_from_down_agents
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     notify=False)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 431, in remove_network_from_dhcp_agent
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     binding_obj.delete()
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 912, in delete
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     self._get_composite_keys()))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 94, in delete_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     db_obj = _safe_get_object(obj_cls, context, **kwargs)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 80, in _safe_get_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     id="%s(%s)" % (obj_cls.db_model.__name__, key))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db

** Affects: neutron
     Importance: Medium
     Assignee: Slawek Kaplonski (slaweq)
         Status: In Progress


** Tags: db l3-ipam-dhcp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1970759

Title:
  Race during removal of the network from DHCP agent

Status in neutron:
  In Progress

Bug description:
  It may happen that 2 neutron-server workers will try to remove network
  from the DHCP agent in almost the same way. On one of them all will
  work fine:

  2022-04-19 18:51:26.579 50 WARNING neutron.db.agentschedulers_db
  [req-73439ae5-d6a4-432c-92f0-872cf5d19bec - - - - -] Removing network
  d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
  1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report
  to the server in the last 150 seconds.

  but on second one it will fail with error:

  2022-04-19 18:51:26.634 50 WARNING neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to the server in the last 150 seconds.
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Unexpected exception occurred while removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947: neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db Traceback (most recent call last):
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 342, in remove_networks_from_down_agents
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     notify=False)
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 431, in remove_network_from_dhcp_agent
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     binding_obj.delete()
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 912, in delete
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     self._get_composite_keys()))
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 94, in delete_object
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     db_obj = _safe_get_object(obj_cls, context, **kwargs)
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 80, in _safe_get_object
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     id="%s(%s)" % (obj_cls.db_model.__name__, key))
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1970759/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next