yahoo-eng-team team mailing list archive

[OpenStack Infra <2012104@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/881204
Committed: https://opendev.org/openstack/neutron/commit/b31453af477bc7421f81886fc1f4cba230dad425
Submitter: "Zuul (22348)"
Branch:    master

commit b31453af477bc7421f81886fc1f4cba230dad425
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Thu Apr 20 20:52:33 2023 +0200

    [OVN] Admin procedure for duplicated or deleted OVN agents
    
    This patch documents how to detect that the system has duplicated
    "Chassis" and "Chassis_Private" registers or when a "Chassis_Private"
    register is orphaned, and how to proceed to health the OVN Southbound
    database.
    
    Closes-Bug: #2012104
    Change-Id: I926e6b9fe5fbad2968fc92e65082b7bb0d8571a9


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2012104

Title:
  Neutron picking incorrect ovn records

Status in neutron:
  Fix Released

Bug description:
  For one of our compute machines I'm seeing two network agents that
  appear unhealthy:

  ```
  $ os network agent list | fgrep "register deleted"
  | compute1                              | OVN Controller agent | ("Chassis" register deleted) |                   | XXX   | UP    | ovn-controller             |
  | c085d57a-3a2b-4f97-8250-23d3f914b078 | OVN Metadata agent   | ("Chassis" register deleted) |                   | XXX   | UP    | neutron-ovn-metadata-agent |
  ```

  The ("Chassis" register deleted) message appears to come from the fix
  for this: https://bugs.launchpad.net/neutron/+bug/1951149

  Searching for that external id I can find this private chassis and
  it's chassis indeed seems empty:

  ```
  $ sudo ovn-sbctl find chassis-private | grep -A 5 e621e0fb-83d3-4a18-82b3-c842996548ed'
  _uuid               : e621e0fb-83d3-4a18-82b3-c842996548ed
  chassis             : []
  external_ids        : {"neutron:liveness_check_at"="2022-06-17T08:43:33.393639+00:00", "neutron:metadata_liveness_check_at"="2022-06-17T02:27:21.309718+00:00", "neutron:ovn-metadata-id"="c085d57a-3a2b-4f97-8250-23d3f914b078", "ne
  utron:ovn-metadata-sb-cfg"="150397"}
  name                : compute1
  nb_cfg              : 150397
  nb_cfg_timestamp    : 1657729945956
  ```

  But there's also:

  ```
  $ sudo ovn-sbctl find chassis hostname=compute1.stack
  _uuid               : 164cb56b-1a3c-4401-bc52-6fa5e58d8f2a
  encaps              : [c442312a-9dfa-4ffe-9db7-afe5f9055962]
  external_ids        : {datapath-type=system, iface-types="bareudp,erspan,geneve,gre,gtpu,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", "neutron:ovn-metadata-sb-cfg"="250161", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-enable-lflow-cache="true", ovn-limit-lflow-cache="", ovn-memlimit-lflow-cache-kb="", ovn-monitor-all="false", ovn-trim-limit-lflow-cache="", ovn-trim-timeout-ms="", ovn-trim-wmark-perc-lflow-cache="", port-up-notif="true"}
  hostname            : compute1.stack
  name                : compute1.stack
  nb_cfg              : 0
  other_config        : {datapath-type=system, iface-types="bareudp,erspan,geneve,gre,gtpu,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-enable-lflow-cache="true", ovn-limit-lflow-cache="", ovn-memlimit-lflow-cache-kb="", ovn-monitor-all="false", ovn-trim-limit-lflow-cache="", ovn-trim-timeout-ms="", ovn-trim-wmark-perc-lflow-cache="", port-up-notif="true"}
  transport_zones     : []
  vtep_logical_switches: []

  $ sudo ovn-sbctl find chassis-private chassis=164cb56b-1a3c-4401-bc52-6fa5e58d8f2a
  _uuid               : cbec617d-19dc-481c-ba99-b4132244773c
  chassis             : 164cb56b-1a3c-4401-bc52-6fa5e58d8f2a
  external_ids        : {"neutron:ovn-metadata-id"="3328a0c7-081b-58a9-9e91-baf5c8c259cd", "neutron:ovn-metadata-sb-cfg"="312321"}
  name                : compute1.stack
  nb_cfg              : 312321
  nb_cfg_timestamp    : 1679042105359
  ```

  
  Which seems to be a correct entry -- should neutron not pick up this entry rather than the one with "chassis             : []"?

  Software versions:

  ii  neutron-server                  2:20.2.0-0ubuntu1~cloud0
  all          Neutron is a virtual network service for Openstack -
  server

  ii  ovn-central                    22.03.0-0ubuntu1~cloud0
  amd64        OVN central components

  Distributor ID: Ubuntu
  Description:    Ubuntu 20.04.4 LTS
  Release:        20.04
  Codename:       focal

  
  Please let me know if I can provide more diagnostics.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2012104/+subscriptions