yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2045383] [NEW] [xena] nftables periodic jobs fails with RETRY_LIMIT

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2045383@xxxxxxxxxxxxxxxxxx>
Date: Fri, 01 Dec 2023 07:49:41 -0000
Reply-to: Bug 2045383 <2045383@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Since 18th Nov nftables jobs failing with RETRY_LIMIT and no logs are
available. These jobs quickly fails and last task seen running in
console is "Preparing job workspace".

This looks a regression with zuul change
https://review.opendev.org/c/zuul/zuul/+/900489.

@fungi checked executor logs and it fails as:-
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: [e: a4e70e9b208f479e98334c09305d2013] [build: 3c41d75624274d2e8d7fd62ae332c31d] Exception while executing job
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   Traceback (most recent call last):
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1125, in __getitem__
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       return getattr(self, index)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:              ^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1114, in __getattr__
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       return list.__getattribute__(self, attr)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   AttributeError: 'IterableList' object has no attribute '2.3.0'
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   The above exception was the direct cause of the following exception:
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   Traceback (most recent call last):
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 1185, in do_execute
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self._execute()
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 1556, in _execute
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.preparePlaybooks(args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2120, in preparePlaybooks
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.preparePlaybook(jobdir_playbook, playbook, args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2177, in preparePlaybook
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.prepareRole(jobdir_playbook, role, args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2367, in prepareRole
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.prepareZuulRole(jobdir_playbook, role, args, role_info)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2423, in prepareZuulRole
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       path = self.checkoutUntrustedProject(project, branch, args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2295, in checkoutUntrustedProject
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       repo.getBranchHead(branch).hexsha
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       ^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/merger/merger.py", line 465, in getBranchHead
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       branch_head = repo.heads[branch]
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:                     ~~~~~~~~~~^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1127, in __getitem__
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       raise IndexError("No item found with id %r" % (self._prefix + index)) from e
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   IndexError: No item found with id '2.3.0'

The parent job[1] which does pins neutron-tempest-plugin to 2.3.0 also
do pass but not the inherited ones, same can be seen in the test
patch[2]

Builds:- https://zuul.openstack.org/builds?job_name=neutron-linuxbridge-
tempest-plugin-scenario-nftables&job_name=neutron-ovs-tempest-plugin-
scenario-iptables_hybrid-
nftables&project=openstack%2Fneutron&branch=stable%2Fxena&skip=0

[1] https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-xena
[2] https://review.opendev.org/c/openstack/neutron/+/902296

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2045383

Title:
  [xena] nftables periodic jobs fails with RETRY_LIMIT

Status in neutron:
  New

Bug description:
  Since 18th Nov nftables jobs failing with RETRY_LIMIT and no logs are
  available. These jobs quickly fails and last task seen running in
  console is "Preparing job workspace".

  This looks a regression with zuul change
  https://review.opendev.org/c/zuul/zuul/+/900489.

  @fungi checked executor logs and it fails as:-
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: [e: a4e70e9b208f479e98334c09305d2013] [build: 3c41d75624274d2e8d7fd62ae332c31d] Exception while executing job
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   Traceback (most recent call last):
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1125, in __getitem__
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       return getattr(self, index)
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:              ^^^^^^^^^^^^^^^^^^^^
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1114, in __getattr__
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       return list.__getattribute__(self, attr)
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   AttributeError: 'IterableList' object has no attribute '2.3.0'
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   The above exception was the direct cause of the following exception:
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   Traceback (most recent call last):
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 1185, in do_execute
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self._execute()
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 1556, in _execute
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.preparePlaybooks(args)
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2120, in preparePlaybooks
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.preparePlaybook(jobdir_playbook, playbook, args)
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2177, in preparePlaybook
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.prepareRole(jobdir_playbook, role, args)
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2367, in prepareRole
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       self.prepareZuulRole(jobdir_playbook, role, args, role_info)
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2423, in prepareZuulRole
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       path = self.checkoutUntrustedProject(project, branch, args)
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2295, in checkoutUntrustedProject
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       repo.getBranchHead(branch).hexsha
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/zuul/merger/merger.py", line 465, in getBranchHead
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       branch_head = repo.heads[branch]
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:                     ~~~~~~~~~~^^^^^^^^
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:     File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1127, in __getitem__
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:       raise IndexError("No item found with id %r" % (self._prefix + index)) from e
  2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:   IndexError: No item found with id '2.3.0'

  The parent job[1] which does pins neutron-tempest-plugin to 2.3.0 also
  do pass but not the inherited ones, same can be seen in the test
  patch[2]

  Builds:- https://zuul.openstack.org/builds?job_name=neutron-
  linuxbridge-tempest-plugin-scenario-nftables&job_name=neutron-ovs-
  tempest-plugin-scenario-iptables_hybrid-
  nftables&project=openstack%2Fneutron&branch=stable%2Fxena&skip=0

  [1] https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-xena
  [2] https://review.opendev.org/c/openstack/neutron/+/902296

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2045383/+subscriptions
Follow ups

[Bug 2045383] Re: [xena] nftables periodic jobs fails with RETRY_LIMIT
From: yatin, 2023-12-04