yahoo-eng-team team mailing list archive

[OpenStack Infra <1944201@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/810592
Committed: https://opendev.org/openstack/neutron/commit/57629dc05122b01a9ba76606b8d75cce9da40776
Submitter: "Zuul (22348)"
Branch:    master

commit 57629dc05122b01a9ba76606b8d75cce9da40776
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Thu Sep 23 09:15:09 2021 +0000

    Add retry when executing OF commands if "InvalidDatapath"
    
    When using the OF API (currently Neutron only uses native
    implementation via "os-ken" librarr), retry the command in case of
    "InvalidDatapath" exception.
    
    As commented in the related bug, some operations could restart the
    OF controller (set the OF procols, set the bridge fail mode). During
    the controller restart, a command can return a "InvalidDatapath"
    exception.
    
    Closes-Bug: #1944201
    Change-Id: Ia8d202f8a38362272e9519c1cbd9d6ba9359e0a1


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1944201

Title:
  neutron-openvswitch-agent crashes on start with firewall config of br-
  int

Status in neutron:
  Fix Released

Bug description:
  In upstream CI, Ironic jobs have been encountering failures where we
  never find the networking to be stood up by neutron. Investigation
  into what was going on led us to finding the neutron-openvswitch-agent
  in failed state, exited due to RuntimeError, just a few seconds after
  the service was started.

  neutron-openvswitch-agent[78787]: DEBUG neutron.agent.securitygroups_rpc [None req-b18a79b7-7258-44f0-9a69-fa92a490bc26 None None] Init firewall settings (driver=openvswitch) {{(pid=78787) init_firewall /opt/stack/neutron/neutron/agent/securitygroups_rpc.py:118}}
  neutron-openvswitch-agent[78787]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbAddCommand(table=Bridge, record=br-int, column=protocols, values=('OpenFlow10', 'OpenFlow11', 'OpenFlow12', 'OpenFlow13', 'OpenFlow14')) {{(pid=78787) do_commit /usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}}
  neutron-openvswitch-agent[78787]: ERROR OfctlService [-] unknown dpid 90695823979334
  neutron-openvswitch-agent[78787]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [None req-b18a79b7-7258-44f0-9a69-fa92a490bc26 None None] ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=71,type=1) error Datapath Invalid 90695823979334: os_ken.app.ofctl.exception.InvalidDatapath: Datapath Invalid 90695823979334
  neutron-openvswitch-agent[78787]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-b18a79b7-7258-44f0-9a69-fa92a490bc26 None None] ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=71,type=1) error Datapath Invalid 90695823979334 agent terminated!: RuntimeError: ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=71,type=1) error Datapath Invalid 90695823979334
  systemd[1]: devstack@q-agt.service: Main process exited, code=exited, status=1/FAILURE
  systemd[1]: devstack@q-agt.service: Failed with result 'exit-code'.

  Originally, this was thought to be related to
  https://bugs.launchpad.net/neutron/+bug/1817022, however this is upon
  service startup on a relatively low load machine where the only action
  really is truly just neutron starting at that time. Also, starting,
  the connections have not been able to exist long enough for inactivity
  idle triggers to occur.

  Investigation into allowed us to identify the general path of what is
  occurring, yet why we don't understand, at least in the Ironic
  community.

  init_firewall() invocation: https://github.com/openstack/neutron/blob/79445f12be3a9ca892672fe0e016336ef60877a2/neutron/agent/securitygroups_rpc.py#L70
  Firewall class launch: https://github.com/openstack/neutron/blob/79445f12be3a9ca892672fe0e016336ef60877a2/neutron/agent/securitygroups_rpc.py#L121

  As the default for the firewall driver ends up sending us into
  openvswitch's firewall code:

  https://github.com/openstack/neutron/blob/79445f12be3a9ca892672fe0e016336ef60877a2/neutron/agent/linux/openvswitch_firewall/firewall.py#L548
  https://github.com/openstack/neutron/blob/79445f12be3a9ca892672fe0e016336ef60877a2/neutron/agent/linux/openvswitch_firewall/firewall.py#L628

  Which eventually ends up in
  https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py#L91
  where it raises a RuntimeError and the service exits out.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1944201/+subscriptions