yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1883601] [NEW] ovn based neutron gate jobs failing 100% of times

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1883601@xxxxxxxxxxxxxxxxxx>
Date: Mon, 15 Jun 2020 20:33:23 -0000
Reply-to: Bug 1883601 <1883601@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

It seems that since last few days our tempest jobs with ovn are failing
all the time. Example of failure:

https://zuul.opendev.org/t/openstack/build/d3a28cb7a40c46dea20fb96a4d116c98

Error:

2020-06-15 16:43:44.093113 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/exthdrs_core.o
2020-06-15 16:43:44.132270 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/geneve.o
2020-06-15 16:43:44.145219 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/gre.o
2020-06-15 16:43:44.520322 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/gso.o
2020-06-15 16:43:44.613210 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/genetlink-openvswitch.o
2020-06-15 16:43:44.837038 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/inet_fragment.o
2020-06-15 16:43:44.871982 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_gre.o
2020-06-15 16:43:44.929289 | controller | /opt/stack/ovs/datapath/linux/geneve.c: In function 'geneve_get_v6_dst':
2020-06-15 16:43:44.929366 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_fragment.o
2020-06-15 16:43:44.929479 | controller | /opt/stack/ovs/datapath/linux/geneve.c:966:17: error: 'const struct ipv6_stub' has no member named 'ipv6_dst_lookup'; did you mean 'ipv6_dst_lookup_flow'?
2020-06-15 16:43:44.929520 | controller |   if (ipv6_stub->ipv6_dst_lookup(geneve->net, gs6->sock->sk, &dst, fl6)) {
2020-06-15 16:43:44.929549 | controller |                  ^~~~~~~~~~~~~~~
2020-06-15 16:43:44.930026 | controller |                  ipv6_dst_lookup_flow
2020-06-15 16:43:44.961658 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_tunnel.o
2020-06-15 16:43:44.964932 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_output.o
2020-06-15 16:43:44.983199 | controller | scripts/Makefile.build:330: recipe for target '/opt/stack/ovs/datapath/linux/geneve.o' failed
2020-06-15 16:43:44.983267 | controller | make[5]: *** [/opt/stack/ovs/datapath/linux/geneve.o] Error 1
2020-06-15 16:43:44.983296 | controller | make[5]: *** Waiting for unfinished jobs....
2020-06-15 16:43:46.142816 | controller | Makefile:1577: recipe for target '_module_/opt/stack/ovs/datapath/linux' failed
2020-06-15 16:43:46.142890 | controller | make[4]: *** [_module_/opt/stack/ovs/datapath/linux] Error 2
2020-06-15 16:43:46.142915 | controller | make[4]: Leaving directory '/usr/src/linux-headers-4.15.0-106-generic'
2020-06-15 16:43:46.142946 | controller | Makefile.main:71: recipe for target 'default' failed
2020-06-15 16:43:46.142968 | controller | make[3]: *** [default] Error 2
2020-06-15 16:43:46.142988 | controller | make[3]: Leaving directory '/opt/stack/ovs/datapath/linux'
2020-06-15 16:43:46.143818 | controller | Makefile:579: recipe for target 'all-recursive' failed
2020-06-15 16:43:46.143846 | controller | make[2]: *** [all-recursive] Error 1
2020-06-15 16:43:46.143867 | controller | make[2]: Leaving directory '/opt/stack/ovs/datapath'
2020-06-15 16:43:46.144917 | controller | Makefile:4990: recipe for target 'all-recursive' failed
2020-06-15 16:43:46.144967 | controller | make[1]: *** [all-recursive] Error 1
2020-06-15 16:43:46.144990 | controller | make[1]: Leaving directory '/opt/stack/ovs'
2020-06-15 16:43:46.145010 | controller | Makefile:2750: recipe for target 'all' failed
2020-06-15 16:43:46.145030 | controller | make: *** [all] Error 2

** Affects: neutron
     Importance: Critical
         Status: Confirmed


** Tags: gate-failure ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1883601

Title:
  ovn based neutron gate jobs failing 100% of times

Status in neutron:
  Confirmed

Bug description:
  It seems that since last few days our tempest jobs with ovn are
  failing all the time. Example of failure:

  https://zuul.opendev.org/t/openstack/build/d3a28cb7a40c46dea20fb96a4d116c98

  Error:

  2020-06-15 16:43:44.093113 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/exthdrs_core.o
  2020-06-15 16:43:44.132270 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/geneve.o
  2020-06-15 16:43:44.145219 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/gre.o
  2020-06-15 16:43:44.520322 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/gso.o
  2020-06-15 16:43:44.613210 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/genetlink-openvswitch.o
  2020-06-15 16:43:44.837038 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/inet_fragment.o
  2020-06-15 16:43:44.871982 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_gre.o
  2020-06-15 16:43:44.929289 | controller | /opt/stack/ovs/datapath/linux/geneve.c: In function 'geneve_get_v6_dst':
  2020-06-15 16:43:44.929366 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_fragment.o
  2020-06-15 16:43:44.929479 | controller | /opt/stack/ovs/datapath/linux/geneve.c:966:17: error: 'const struct ipv6_stub' has no member named 'ipv6_dst_lookup'; did you mean 'ipv6_dst_lookup_flow'?
  2020-06-15 16:43:44.929520 | controller |   if (ipv6_stub->ipv6_dst_lookup(geneve->net, gs6->sock->sk, &dst, fl6)) {
  2020-06-15 16:43:44.929549 | controller |                  ^~~~~~~~~~~~~~~
  2020-06-15 16:43:44.930026 | controller |                  ipv6_dst_lookup_flow
  2020-06-15 16:43:44.961658 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_tunnel.o
  2020-06-15 16:43:44.964932 | controller |   CC [M]  /opt/stack/ovs/datapath/linux/ip_output.o
  2020-06-15 16:43:44.983199 | controller | scripts/Makefile.build:330: recipe for target '/opt/stack/ovs/datapath/linux/geneve.o' failed
  2020-06-15 16:43:44.983267 | controller | make[5]: *** [/opt/stack/ovs/datapath/linux/geneve.o] Error 1
  2020-06-15 16:43:44.983296 | controller | make[5]: *** Waiting for unfinished jobs....
  2020-06-15 16:43:46.142816 | controller | Makefile:1577: recipe for target '_module_/opt/stack/ovs/datapath/linux' failed
  2020-06-15 16:43:46.142890 | controller | make[4]: *** [_module_/opt/stack/ovs/datapath/linux] Error 2
  2020-06-15 16:43:46.142915 | controller | make[4]: Leaving directory '/usr/src/linux-headers-4.15.0-106-generic'
  2020-06-15 16:43:46.142946 | controller | Makefile.main:71: recipe for target 'default' failed
  2020-06-15 16:43:46.142968 | controller | make[3]: *** [default] Error 2
  2020-06-15 16:43:46.142988 | controller | make[3]: Leaving directory '/opt/stack/ovs/datapath/linux'
  2020-06-15 16:43:46.143818 | controller | Makefile:579: recipe for target 'all-recursive' failed
  2020-06-15 16:43:46.143846 | controller | make[2]: *** [all-recursive] Error 1
  2020-06-15 16:43:46.143867 | controller | make[2]: Leaving directory '/opt/stack/ovs/datapath'
  2020-06-15 16:43:46.144917 | controller | Makefile:4990: recipe for target 'all-recursive' failed
  2020-06-15 16:43:46.144967 | controller | make[1]: *** [all-recursive] Error 1
  2020-06-15 16:43:46.144990 | controller | make[1]: Leaving directory '/opt/stack/ovs'
  2020-06-15 16:43:46.145010 | controller | Makefile:2750: recipe for target 'all' failed
  2020-06-15 16:43:46.145030 | controller | make: *** [all] Error 2

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1883601/+subscriptions
Follow ups

[Bug 1883601] Re: ovn based neutron gate jobs failing 100% of times
From: OpenStack Infra, 2020-06-19