| Thread Previous • Date Previous • Date Next • Thread Next |
Public bug reported:
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Linux 4.4.0-96-generic on AMD64
Neutron 2:10.0.4-0ubuntu2~cloud0 from Cloud Archive xenial-updates/ocata
OpenVSwitch 2.6.1-0ubuntu5.2~cloud0 from Cloud Archive xenial-upates/ocata
In an environment with three bare-metal Neutron deployments, hosting
upward of 300 routers, with approximately the same number of instances,
typically one router per instance, packet loss on instances accessed via
floating IPs, including complete connectivity loss, is experienced. The
problem is exacerbated by enabling L3HA, likely due to the increase in
router namespaces to be scheduled and managed, and the additional
scheduling work of bringing up keepalived and monitoring the keepalived
VIP.
Reducing the number of routers and rescheduling routers on new hosts,
causing the routers to undergo a full recreation of namespace, iptables
rules, and replugging of interfaces into OVS will correct packet loss or
connectivity loss on impacted routers.
On Neutron hosts in this environment, we have used systemtap to trace
calls to kfree_skb which reveals the majority of dropped packets occur
in the openvswitch module, notably on the br-int bridge. Inspecting the
state of OVS shows many qtap interfaces which are no longer present on
the Neutron host which are still plugged in to OVS.
Diagnostic outputs in following comments.
** Affects: neutron
Importance: Undecided
Status: New
** Affects: openvswitch (Ubuntu)
Importance: Undecided
Status: New
** Also affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1749425
Title:
Neutron integrated with OpenVSwitch drops packets and fails to
plug/unplug interfaces from OVS on router interfaces at scale
Status in neutron:
New
Status in openvswitch package in Ubuntu:
New
Bug description:
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Linux 4.4.0-96-generic on AMD64
Neutron 2:10.0.4-0ubuntu2~cloud0 from Cloud Archive xenial-updates/ocata
OpenVSwitch 2.6.1-0ubuntu5.2~cloud0 from Cloud Archive xenial-upates/ocata
In an environment with three bare-metal Neutron deployments, hosting
upward of 300 routers, with approximately the same number of
instances, typically one router per instance, packet loss on instances
accessed via floating IPs, including complete connectivity loss, is
experienced. The problem is exacerbated by enabling L3HA, likely due
to the increase in router namespaces to be scheduled and managed, and
the additional scheduling work of bringing up keepalived and
monitoring the keepalived VIP.
Reducing the number of routers and rescheduling routers on new hosts,
causing the routers to undergo a full recreation of namespace,
iptables rules, and replugging of interfaces into OVS will correct
packet loss or connectivity loss on impacted routers.
On Neutron hosts in this environment, we have used systemtap to trace
calls to kfree_skb which reveals the majority of dropped packets occur
in the openvswitch module, notably on the br-int bridge. Inspecting
the state of OVS shows many qtap interfaces which are no longer
present on the Neutron host which are still plugged in to OVS.
Diagnostic outputs in following comments.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1749425/+subscriptions
| Thread Previous • Date Previous • Date Next • Thread Next |
