yahoo-eng-team team mailing list archive

[OpenStack Infra <1970759@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/839779
Committed: https://opendev.org/openstack/neutron/commit/8df2f69b6f93fd964cc96efb42d085063b90a8e6
Submitter: "Zuul (22348)"
Branch:    master

commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6
Author: Slawek Kaplonski <skaplons@xxxxxxxxxx>
Date:   Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent
    
    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.
    
    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1970759

Title:
  Race during removal of the network from DHCP agent

Status in neutron:
  Fix Released

Bug description:
  It may happen that 2 neutron-server workers will try to remove network
  from the DHCP agent in almost the same way. On one of them all will
  work fine:

  2022-04-19 18:51:26.579 50 WARNING neutron.db.agentschedulers_db
  [req-73439ae5-d6a4-432c-92f0-872cf5d19bec - - - - -] Removing network
  d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
  1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report
  to the server in the last 150 seconds.

  but on second one it will fail with error:

  2022-04-19 18:51:26.634 50 WARNING neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to the server in the last 150 seconds.
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Unexpected exception occurred while removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947: neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db Traceback (most recent call last):
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 342, in remove_networks_from_down_agents
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     notify=False)
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 431, in remove_network_from_dhcp_agent
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     binding_obj.delete()
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 912, in delete
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     self._get_composite_keys()))
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 94, in delete_object
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     db_obj = _safe_get_object(obj_cls, context, **kwargs)
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db   File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 80, in _safe_get_object
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db     id="%s(%s)" % (obj_cls.db_model.__name__, key))
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
  2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1970759/+subscriptions