yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1704212] [NEW] MAAS marks node 'Deployed' before sshd is up

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jason Hobbs <jason.hobbs@xxxxxxxxxxxxx>
Date: Thu, 13 Jul 2017 20:30:57 -0000
Reply-to: Bug 1704212 <1704212@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

Public bug reported:

MAAS is marking one of my nodes as 'deployed' before SSHD comes up. My
code is watching for the MAAS node to change to Deployed state, attempts
SSH, and then fails because sshd is not listening yet.

After seeing the node turn to deployed state:
Node changed status - From 'Deploying' to 'Deployed' Thu, 13 Jul. 2017 20:17:34

my code attempts ssh to the node and fails:

ssh: connect to host 10.245.208.32 port 22: Connection refused
2017-07-13-20:17:41 ERROR Fatal: Could not establish SSH connection to ubuntu@10.245.208.32

According to logs from the node, ssh didn't start until 20:17:42:
Jul 13 20:17:42 swoobat systemd[1]: Starting OpenBSD Secure Shell server...
Jul 13 20:17:42 swoobat sshd[2230]: Server listening on 0.0.0.0 port 22.
Jul 13 20:17:42 swoobat sshd[2230]: Server listening on :: port 22.
Jul 13 20:17:42 swoobat systemd[1]: Started OpenBSD Secure Shell server.

In the cloud-init.log, you can see the request for user data at
20:17:35, 7 seconds before sshd starts up. That's when MAAS marks the
node deployed.

Here is /var/log/cloud-init.log from the node:
http://paste.ubuntu.com/25084186/

And /var/log/cloud-init-output.log from the node:
http://paste.ubuntu.com/25084192/

This is with maas 2.2.0

** Affects: cloud-init
Importance: Undecided
Status: New

** Affects: maas
Importance: Undecided
Status: New

** Tags: cdo-qa

** Description changed:

MAAS is marking one of my nodes as 'deployed' before SSHD comes up. My
code is watching for the MAAS node to change to Deployed state, attempts
SSH, and then fails because sshd is not listening yet.

After seeing the node turn to deployed state:
Node changed status - From 'Deploying' to 'Deployed' Thu, 13 Jul. 2017 20:17:34

my code attempts ssh to the node and fails:

ssh: connect to host 10.245.208.32 port 22: Connection refused
2017-07-13-20:17:41 ERROR Fatal: Could not establish SSH connection to ubuntu@10.245.208.32

Here is /var/log/cloud-init.log from the node:
http://paste.ubuntu.com/25084186/

And /var/log/cloud-init-output.log from the node:
http://paste.ubuntu.com/25084192/
+
+ This is with maas 2.2.0

** Also affects: cloud-init
Importance: Undecided
Status: New

--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1704212

Title:
MAAS marks node 'Deployed' before sshd is up

Status in cloud-init:
New
Status in MAAS:
New

Bug description:
MAAS is marking one of my nodes as 'deployed' before SSHD comes up.
My code is watching for the MAAS node to change to Deployed state,
attempts SSH, and then fails because sshd is not listening yet.

After seeing the node turn to deployed state:
Node changed status - From 'Deploying' to 'Deployed' Thu, 13 Jul. 2017 20:17:34

my code attempts ssh to the node and fails:

ssh: connect to host 10.245.208.32 port 22: Connection refused
2017-07-13-20:17:41 ERROR Fatal: Could not establish SSH connection to ubuntu@10.245.208.32

In the cloud-init.log, you can see the request for user data at
20:17:35, 7 seconds before sshd starts up. That's when MAAS marks the
node deployed.

Here is /var/log/cloud-init.log from the node:
http://paste.ubuntu.com/25084186/

And /var/log/cloud-init-output.log from the node:
http://paste.ubuntu.com/25084192/

This is with maas 2.2.0

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1704212/+subscriptions

Follow ups

[Bug 1704212] Re: MAAS marks node 'Deployed' before sshd is up
From: Dan Watkins, 2019-02-22
[Bug 1704212] Re: MAAS marks node 'Deployed' before sshd is up
From: Andres Rodriguez, 2017-07-20