yahoo-eng-team team mailing list archive

[Joshua Genet <1923257@xxxxxxxxxxxxxxxxxx>]

The networking issue we are running into seems to be caused by this:
https://askubuntu.com/questions/1126037/netplan-generates-the-same-mac-address-for-bridges-on-two-different-machines

We believe it is a bug in the maas-image-builder scripts (meph2) that we
consume and will be filing a bug over there once we have more info.

** Changed in: cloud-init
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1923257

Title:
  diminished networking, packet issues during Bionic openstack deploys

Status in cloud-init:
  Invalid

Bug description:
  Test run here:
  https://solutions.qa.canonical.com/testruns/testRun/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b

  Artifacts/Logs/Bundles here:
  https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/index.html

  Juju Openstack model crashdump here:
  https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/generated/generated/openstack/juju-crashdump-openstack-2021-04-09-19.57.18.tar.gz

  ---

  We've seen this in several different environments. Some using Stein-
  Bionic, some using Ussuri-Bionic. Ussuri-Focal had no issues.

  Juju ssh-ing to any of these machines is incredibly slow and unresponsive. (an ls takes somewhere around 30-60seconds)
  As seen below, we're failing to grab lxd images for our instances. Likely due to the diminished networking. On an earlier manual run I was apt installing at 400B/s.

  ---

  Machine   State    DNS            Inst id  Series  AZ     Message
  0         started  <ourIP>        duision  bionic  zone1  Deployed
  0/lxd/0   pending                 pending  bionic         starting
  0/lxd/1   pending                 pending  bionic         starting
  0/lxd/2   pending                 pending  bionic         starting
  0/lxd/3   pending                 pending  bionic         acquiring LXD image
  0/lxd/4   pending                 pending  bionic         starting
  0/lxd/5   down                    pending  bionic         failed to start machine 0/lxd/5 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)
  0/lxd/6   pending                 pending  bionic         Creating container spec
  0/lxd/7   pending                 pending  bionic         starting
  0/lxd/8   pending                 pending  bionic         starting
  0/lxd/9   pending                 pending  bionic         starting
  0/lxd/10  pending                 pending  bionic         starting
  1         started  <ourIP>        azurill  bionic  zone1  Deployed
  1/lxd/0   pending                 pending  bionic         starting
  1/lxd/1   down                    pending  bionic         failed to start machine 1/lxd/1 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)
  1/lxd/2   pending                 pending  bionic         starting
  1/lxd/3   pending                 pending  bionic         starting
  1/lxd/4   pending                 pending  bionic         starting
  1/lxd/5   pending                 pending  bionic         starting
  1/lxd/6   pending                 pending  bionic         starting
  1/lxd/7   down                    pending  bionic         failed to start machine 1/lxd/7 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/server/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: proxyconnect tcp: Unable to connect to: squid.internal), retrying in 10s (10 more attempts)
  1/lxd/8   pending                 pending  bionic         starting
  1/lxd/9   pending                 pending  bionic         starting
  1/lxd/10  pending                 pending  bionic         starting
  2         started  <ourIP>        meowth   bionic  zone2  Deployed
  2/lxd/0   down                    pending  bionic         failed to start machine 2/lxd/0 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: read tcp: read: connection reset by peer), retrying in 10s (10 more attempts)

  ---

  kern.log from one of our machines shows a bunch of this:

  ---

  Apr  9 20:52:22 azurill kernel: [ 4196.092358] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:22 azurill kernel: [ 4196.317426] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:22 azurill kernel: [ 4196.387884] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:22 azurill kernel: [ 4196.423968] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:22 azurill kernel: [ 4196.488808] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:22 azurill kernel: [ 4196.919973] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:23 azurill kernel: [ 4197.116675] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:23 azurill kernel: [ 4197.331899] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:23 azurill kernel: [ 4197.447991] br-eth0: received packet on eth0 with own address as source address
  Apr  9 20:52:23 azurill kernel: [ 4197.513159] br-eth0: received packet on eth0 with own address as source address

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1923257/+subscriptions