yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #85756
[Bug 1923257] [NEW] diminished networking, packet issues during Bionic openstack deploys
Public bug reported:
Test run here:
https://solutions.qa.canonical.com/testruns/testRun/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b
Artifacts/Logs/Bundles here:
https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/index.html
Juju Openstack model crashdump here:
https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/generated/generated/openstack/juju-crashdump-openstack-2021-04-09-19.57.18.tar.gz
---
We've seen this in several different environments. Some using Stein-
Bionic, some using Ussuri-Bionic. Ussuri-Focal had no issues.
Juju ssh-ing to any of these machines is incredibly slow and unresponsive. (an ls takes somewhere around 30-60seconds)
As seen below, we're failing to grab lxd images for our instances. Likely due to the diminished networking. On an earlier manual run I was apt installing at 400B/s.
---
Machine State DNS Inst id Series AZ Message
0 started <ourIP> duision bionic zone1 Deployed
0/lxd/0 pending pending bionic starting
0/lxd/1 pending pending bionic starting
0/lxd/2 pending pending bionic starting
0/lxd/3 pending pending bionic acquiring LXD image
0/lxd/4 pending pending bionic starting
0/lxd/5 down pending bionic failed to start machine 0/lxd/5 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)
0/lxd/6 pending pending bionic Creating container spec
0/lxd/7 pending pending bionic starting
0/lxd/8 pending pending bionic starting
0/lxd/9 pending pending bionic starting
0/lxd/10 pending pending bionic starting
1 started <ourIP> azurill bionic zone1 Deployed
1/lxd/0 pending pending bionic starting
1/lxd/1 down pending bionic failed to start machine 1/lxd/1 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)
1/lxd/2 pending pending bionic starting
1/lxd/3 pending pending bionic starting
1/lxd/4 pending pending bionic starting
1/lxd/5 pending pending bionic starting
1/lxd/6 pending pending bionic starting
1/lxd/7 down pending bionic failed to start machine 1/lxd/7 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/server/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: proxyconnect tcp: Unable to connect to: squid.internal), retrying in 10s (10 more attempts)
1/lxd/8 pending pending bionic starting
1/lxd/9 pending pending bionic starting
1/lxd/10 pending pending bionic starting
2 started <ourIP> meowth bionic zone2 Deployed
2/lxd/0 down pending bionic failed to start machine 2/lxd/0 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: read tcp: read: connection reset by peer), retrying in 10s (10 more attempts)
---
kern.log from one of our machines shows a bunch of this:
---
Apr 9 20:52:22 azurill kernel: [ 4196.092358] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.317426] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.387884] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.423968] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.488808] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.919973] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.116675] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.331899] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.447991] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.513159] br-eth0: received packet on eth0 with own address as source address
** Affects: cloud-init
Importance: Undecided
Status: New
** Tags: cdo-qa cdo-release-blocker foundations-engine
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1923257
Title:
diminished networking, packet issues during Bionic openstack deploys
Status in cloud-init:
New
Bug description:
Test run here:
https://solutions.qa.canonical.com/testruns/testRun/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b
Artifacts/Logs/Bundles here:
https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/index.html
Juju Openstack model crashdump here:
https://oil-jenkins.canonical.com/artifacts/19f7492e-1a8c-4ec3-b0bb-0fc4f2453f7b/generated/generated/openstack/juju-crashdump-openstack-2021-04-09-19.57.18.tar.gz
---
We've seen this in several different environments. Some using Stein-
Bionic, some using Ussuri-Bionic. Ussuri-Focal had no issues.
Juju ssh-ing to any of these machines is incredibly slow and unresponsive. (an ls takes somewhere around 30-60seconds)
As seen below, we're failing to grab lxd images for our instances. Likely due to the diminished networking. On an earlier manual run I was apt installing at 400B/s.
---
Machine State DNS Inst id Series AZ Message
0 started <ourIP> duision bionic zone1 Deployed
0/lxd/0 pending pending bionic starting
0/lxd/1 pending pending bionic starting
0/lxd/2 pending pending bionic starting
0/lxd/3 pending pending bionic acquiring LXD image
0/lxd/4 pending pending bionic starting
0/lxd/5 down pending bionic failed to start machine 0/lxd/5 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)
0/lxd/6 pending pending bionic Creating container spec
0/lxd/7 pending pending bionic starting
0/lxd/8 pending pending bionic starting
0/lxd/9 pending pending bionic starting
0/lxd/10 pending pending bionic starting
1 started <ourIP> azurill bionic zone1 Deployed
1/lxd/0 pending pending bionic starting
1/lxd/1 down pending bionic failed to start machine 1/lxd/1 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)
1/lxd/2 pending pending bionic starting
1/lxd/3 pending pending bionic starting
1/lxd/4 pending pending bionic starting
1/lxd/5 pending pending bionic starting
1/lxd/6 pending pending bionic starting
1/lxd/7 down pending bionic failed to start machine 1/lxd/7 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/server/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: proxyconnect tcp: Unable to connect to: squid.internal), retrying in 10s (10 more attempts)
1/lxd/8 pending pending bionic starting
1/lxd/9 pending pending bionic starting
1/lxd/10 pending pending bionic starting
2 started <ourIP> meowth bionic zone2 Deployed
2/lxd/0 down pending bionic failed to start machine 2/lxd/0 (acquiring LXD image: Failed remote image download: Get https://cloud-images.ubuntu.com/releases/bionic/release-20210325/ubuntu-18.04-server-cloudimg-amd64.squashfs: read tcp: read: connection reset by peer), retrying in 10s (10 more attempts)
---
kern.log from one of our machines shows a bunch of this:
---
Apr 9 20:52:22 azurill kernel: [ 4196.092358] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.317426] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.387884] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.423968] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.488808] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:22 azurill kernel: [ 4196.919973] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.116675] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.331899] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.447991] br-eth0: received packet on eth0 with own address as source address
Apr 9 20:52:23 azurill kernel: [ 4197.513159] br-eth0: received packet on eth0 with own address as source address
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1923257/+subscriptions
Follow ups
