yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #92538
[Bug 2024976] [NEW] iptable rules restoring error in l3-agent and openvswitch-agent
Public bug reported:
Openstack version: zed/stable
OS version: Ubuntu 22.04.2 LTS
Kernel version: 5.15.0-75-generic #82-Ubuntu
Deployment: kolla-ansible
iptable rules restoring error in l3-agent and openvswitch-agent:
openvswitch-agnet log:
2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager
*filter
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
-I FORWARD 1 -j neutron-filter-top
-I FORWARD 2 -j neutron-openvswi-FORWARD
-I INPUT 1 -j neutron-openvswi-INPUT
-I OUTPUT 1 -j neutron-filter-top
-I OUTPUT 2 -j neutron-openvswi-OUTPUT
-I neutron-filter-top 1 -j neutron-openvswi-local
-I neutron-openvswi-FORWARD 1 -m physdev --physdev-out tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-FORWARD 2 -m physdev --physdev-in tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-FORWARD 3 -m physdev --physdev-out tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-FORWARD 4 -m physdev --physdev-in tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-sg-chain 1 -j ACCEPT
-I neutron-openvswi-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*raw
:OUTPUT - [0:0]
:PREROUTING - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
-I OUTPUT 1 -j neutron-openvswi-OUTPUT
-I PREROUTING 1 -j neutron-openvswi-PREROUTING
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `physdev':No such file or directory
Error occurred at line: 19
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
l3-agent log:
2023-06-23 16:15:49.545 33 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['ip', 'netns', 'exec', 'qrouter-0f0e60d0-bf51-4361-901b-4b998201b44b', 'iptables-restore', '-n']; Stdin: # Generated by iptables_manager
*filter
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
-I FORWARD 1 -j neutron-filter-top
-I FORWARD 2 -j neutron-l3-agent-FORWARD
-I INPUT 1 -j neutron-l3-agent-INPUT
-I OUTPUT 1 -j neutron-filter-top
-I OUTPUT 2 -j neutron-l3-agent-OUTPUT
-I neutron-filter-top 1 -j neutron-l3-agent-local
-I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope
-I neutron-l3-agent-scope 1 -m mark --mark 0x1/0xffff -j DROP
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:POSTROUTING - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-floatingip - [0:0]
:neutron-l3-agent-mark - [0:0]
:neutron-l3-agent-scope - [0:0]
-I FORWARD 1 -j neutron-l3-agent-FORWARD
-I INPUT 1 -j neutron-l3-agent-INPUT
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
-I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-mark
-I neutron-l3-agent-PREROUTING 2 -j neutron-l3-agent-scope
-I neutron-l3-agent-PREROUTING 3 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-I neutron-l3-agent-PREROUTING 4 -j neutron-l3-agent-floatingip
-I neutron-l3-agent-PREROUTING 5 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
-I neutron-l3-agent-float-snat 1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*nat
:OUTPUT - [0:0]
:POSTROUTING - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING
-I POSTROUTING 2 -j neutron-postrouting-bottom
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
-I neutron-l3-agent-POSTROUTING 1 ! -o rfp-0f0e60d0-b -m conntrack ! --ctstate DNAT -j ACCEPT
-I neutron-l3-agent-PREROUTING 1 -d 137.175.31.207/32 -i rfp-0f0e60d0-b -j DNAT --to-destination 10.10.0.246
-I neutron-l3-agent-float-snat 1 -s 10.10.0.246/32 -j SNAT --to-source 137.175.31.207 --random-fully
-I neutron-l3-agent-snat 1 -j neutron-l3-agent-float-snat
-I neutron-postrouting-bottom 1 -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*raw
:OUTPUT - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `mark':No such file or directory
Error occurred at line: 19
And we check the system the x_tables kernel module were loaded:
# lsmod | grep x_tablesx_tables 53248 12
xt_conntrack,nft_compat,xt_tcpudp,xt_physdev,xt_nat,xt_comment,ip6_tables,xt_connmark,xt_CT,ip_tables,xt_REDIRECT,xt_mark
(neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*mark.so"
./lib/x86_64-linux-gnu/xtables/libxt_connmark.so
./lib/x86_64-linux-gnu/xtables/libxt_mark.so
./lib/x86_64-linux-gnu/xtables/libebt_mark.so
(neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*physdev.so"
./lib/x86_64-linux-gnu/xtables/libxt_physdev.so
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2024976
Title:
iptable rules restoring error in l3-agent and openvswitch-agent
Status in neutron:
New
Bug description:
Openstack version: zed/stable
OS version: Ubuntu 22.04.2 LTS
Kernel version: 5.15.0-75-generic #82-Ubuntu
Deployment: kolla-ansible
iptable rules restoring error in l3-agent and openvswitch-agent:
openvswitch-agnet log:
2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager
*filter
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
-I FORWARD 1 -j neutron-filter-top
-I FORWARD 2 -j neutron-openvswi-FORWARD
-I INPUT 1 -j neutron-openvswi-INPUT
-I OUTPUT 1 -j neutron-filter-top
-I OUTPUT 2 -j neutron-openvswi-OUTPUT
-I neutron-filter-top 1 -j neutron-openvswi-local
-I neutron-openvswi-FORWARD 1 -m physdev --physdev-out tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-FORWARD 2 -m physdev --physdev-in tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-FORWARD 3 -m physdev --physdev-out tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-FORWARD 4 -m physdev --physdev-in tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT
-I neutron-openvswi-sg-chain 1 -j ACCEPT
-I neutron-openvswi-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*raw
:OUTPUT - [0:0]
:PREROUTING - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
-I OUTPUT 1 -j neutron-openvswi-OUTPUT
-I PREROUTING 1 -j neutron-openvswi-PREROUTING
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `physdev':No such file or directory
Error occurred at line: 19
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
l3-agent log:
2023-06-23 16:15:49.545 33 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['ip', 'netns', 'exec', 'qrouter-0f0e60d0-bf51-4361-901b-4b998201b44b', 'iptables-restore', '-n']; Stdin: # Generated by iptables_manager
*filter
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
-I FORWARD 1 -j neutron-filter-top
-I FORWARD 2 -j neutron-l3-agent-FORWARD
-I INPUT 1 -j neutron-l3-agent-INPUT
-I OUTPUT 1 -j neutron-filter-top
-I OUTPUT 2 -j neutron-l3-agent-OUTPUT
-I neutron-filter-top 1 -j neutron-l3-agent-local
-I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope
-I neutron-l3-agent-scope 1 -m mark --mark 0x1/0xffff -j DROP
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:POSTROUTING - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-floatingip - [0:0]
:neutron-l3-agent-mark - [0:0]
:neutron-l3-agent-scope - [0:0]
-I FORWARD 1 -j neutron-l3-agent-FORWARD
-I INPUT 1 -j neutron-l3-agent-INPUT
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
-I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-mark
-I neutron-l3-agent-PREROUTING 2 -j neutron-l3-agent-scope
-I neutron-l3-agent-PREROUTING 3 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-I neutron-l3-agent-PREROUTING 4 -j neutron-l3-agent-floatingip
-I neutron-l3-agent-PREROUTING 5 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
-I neutron-l3-agent-float-snat 1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*nat
:OUTPUT - [0:0]
:POSTROUTING - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING
-I POSTROUTING 2 -j neutron-postrouting-bottom
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
-I neutron-l3-agent-POSTROUTING 1 ! -o rfp-0f0e60d0-b -m conntrack ! --ctstate DNAT -j ACCEPT
-I neutron-l3-agent-PREROUTING 1 -d 137.175.31.207/32 -i rfp-0f0e60d0-b -j DNAT --to-destination 10.10.0.246
-I neutron-l3-agent-float-snat 1 -s 10.10.0.246/32 -j SNAT --to-source 137.175.31.207 --random-fully
-I neutron-l3-agent-snat 1 -j neutron-l3-agent-float-snat
-I neutron-postrouting-bottom 1 -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*raw
:OUTPUT - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables): Couldn't load match `mark':No such file or directory
Error occurred at line: 19
And we check the system the x_tables kernel module were loaded:
# lsmod | grep x_tablesx_tables 53248 12
xt_conntrack,nft_compat,xt_tcpudp,xt_physdev,xt_nat,xt_comment,ip6_tables,xt_connmark,xt_CT,ip_tables,xt_REDIRECT,xt_mark
(neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*mark.so"
./lib/x86_64-linux-gnu/xtables/libxt_connmark.so
./lib/x86_64-linux-gnu/xtables/libxt_mark.so
./lib/x86_64-linux-gnu/xtables/libebt_mark.so
(neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*physdev.so"
./lib/x86_64-linux-gnu/xtables/libxt_physdev.so
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2024976/+subscriptions
