yahoo-eng-team team mailing list archive

[LIU Yulong <2051863@xxxxxxxxxxxxxxxxxx>]

Looks more like a libvirt error, or nova side problem. Neutron does not
take responsibilities to create the tap-XXX device. It is plugged by
nova-compute. Need to find out why the tap device is not created before
TC rules creating.

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2051863

Title:
  Race condition during VM creation - could not open network device
  tapXXX (No such device)

Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  [High-Level Description]
  While creating an amphora from the Octavia service, we are encountering a race condition. Nova-compute is unable to add a QC ingress rule because the tap interface does not exist yet. This situation results in the service becoming temporarily unavailable, failing in approximately 50% of cases.

  [Pre-conditions]
  Create a simple octavia loadbalancer with environment deployed by kolla-environment. For amphora server, use flavor with qos properties as from example: https://paste.openstack.org/show/bkx3RgEFqUwxCU0Qr8z2/

  [Step-by-Step Reproduction]
  Create a load balancer as part of a Heat stack, following the example provided at:  https://github.com/syseleven/heat-examples/tree/master/lbaas-octavia-http
  Use a qos flavor as shown in example: https://paste.openstack.org/show/bkx3RgEFqUwxCU0Qr8z2/

  [Expected Output]
  Expect 100% successful creation of the amphora and subsequent Nova instances serving as members for the load balancer.

  [Actual Output]
  During the creation process, it is observed that nova-compute fails due to:
  /var/log/kolla/libvirt/libvirtd.log:2024-01-31 09:24:55.852+0000: 4065628: error : virCommandWait:2748 : internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
  /var/log/kolla/nova/nova-compute.log:2024-01-31 10:24:56.086 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.

  Unfortunately ovs-vswitchd.log file says that tap interface was not created at this point:
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.469Z|00094|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.809Z|00095|bridge|INFO|bridge br-int: added interface tap5f365c23-89 on port 27
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.885Z|00096|bridge|INFO|bridge br-int: deleted interface tap5f365c23-89 on port 27
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.888Z|00097|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.927Z|00098|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.934Z|00099|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.943Z|00100|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
  /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:56.024Z|00101|bridge|WARN|could not open network device tap5f365c23-89 (No such device)

  [Version]
  OpenStack Zed, deployed by kolla-ansible with all defaults
  Ubuntu 22.04 LTS
  libvirt: 8.0.0-1ubuntu7.7
  nova: zed
  python3-openvswitch: 3.0.3-0ubuntu0.22.10.3~cloud3
  neutron: zed

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2051863/+subscriptions