yahoo-eng-team team mailing list archive

[yatin <1978938@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

These FIPS jobs running on centos-9-stream fails randomly at different points due to DNS issue.
In these jobs after configuring fips, node get's rebooted. The unbound service may take some time to be ready after the reboot and until then any DNS resolution fails.

Fails like below:-
2022-06-16 02:36:58.829027 | controller | + ./stack.sh:_install_rdo:314              :   sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos9-master/delorean-deps.repo
2022-06-16 02:36:58.940595 | controller |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-06-16 02:36:58.940689 | controller |                                  Dload  Upload   Total   Spent    Left  Speed
2022-06-16 02:36:58.941657 | controller | 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: trunk.rdoproject.org
2022-06-16 02:36:58.946989 | controller | + ./stack.sh:_install_rdo:316              :   sudo dnf -y update
2022-06-16 02:36:59.649717 | controller | CentOS Stream 9 - BaseOS                        0.0  B/s |   0  B     00:00
2022-06-16 02:36:59.650078 | controller | Errors during downloading metadata for repository 'baseos':
2022-06-16 02:36:59.650117 | controller |   - Curl error (6): Couldn't resolve host name for https://mirror-int.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml [Could not resolve host: mirror-int.ord.rax.opendev.org]

2022-06-16 02:40:30.411317 | controller | + functions-common:sudo_with_proxies:2384  :   sudo http_proxy= https_proxy= no_proxy= dnf install -y rabbitmq-server
2022-06-16 02:40:31.225502 | controller | Last metadata expiration check: 0:03:30 ago on Thu 16 Jun 2022 02:37:01 AM UTC.
2022-06-16 02:40:31.278762 | controller | No match for argument: rabbitmq-server
2022-06-16 02:40:31.286118 | controller | Error: Unable to find a match: rabbitmq-server

Job builds:-
- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-ovs-release-fips&project=openstack/neutron
- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-fips&project=openstack/neutron

** Affects: neutron
     Importance: High
     Assignee: yatin (yatinkarel)
         Status: In Progress

** Changed in: neutron
       Status: New => Triaged

** Changed in: neutron
   Importance: Undecided => High

** Changed in: neutron
     Assignee: (unassigned) => yatin (yatinkarel)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1978938

Title:
  [9-stream][master][yoga] ovs/ovn fips jobs fails randomly with DNS
  issue

Status in neutron:
  In Progress

Bug description:
  These FIPS jobs running on centos-9-stream fails randomly at different points due to DNS issue.
  In these jobs after configuring fips, node get's rebooted. The unbound service may take some time to be ready after the reboot and until then any DNS resolution fails.

  Fails like below:-
  2022-06-16 02:36:58.829027 | controller | + ./stack.sh:_install_rdo:314              :   sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos9-master/delorean-deps.repo
  2022-06-16 02:36:58.940595 | controller |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  2022-06-16 02:36:58.940689 | controller |                                  Dload  Upload   Total   Spent    Left  Speed
  2022-06-16 02:36:58.941657 | controller | 
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: trunk.rdoproject.org
  2022-06-16 02:36:58.946989 | controller | + ./stack.sh:_install_rdo:316              :   sudo dnf -y update
  2022-06-16 02:36:59.649717 | controller | CentOS Stream 9 - BaseOS                        0.0  B/s |   0  B     00:00
  2022-06-16 02:36:59.650078 | controller | Errors during downloading metadata for repository 'baseos':
  2022-06-16 02:36:59.650117 | controller |   - Curl error (6): Couldn't resolve host name for https://mirror-int.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml [Could not resolve host: mirror-int.ord.rax.opendev.org]

  2022-06-16 02:40:30.411317 | controller | + functions-common:sudo_with_proxies:2384  :   sudo http_proxy= https_proxy= no_proxy= dnf install -y rabbitmq-server
  2022-06-16 02:40:31.225502 | controller | Last metadata expiration check: 0:03:30 ago on Thu 16 Jun 2022 02:37:01 AM UTC.
  2022-06-16 02:40:31.278762 | controller | No match for argument: rabbitmq-server
  2022-06-16 02:40:31.286118 | controller | Error: Unable to find a match: rabbitmq-server

  Job builds:-
  - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-ovs-release-fips&project=openstack/neutron
  - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-fips&project=openstack/neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1978938/+subscriptions