yahoo-eng-team team mailing list archive

[Kevin Rogers <1911680@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Starting on the 6th January 2021 we started observing SSH authentication
issues in AWS AMI builds.

We have SSH configured with an absolute (i.e. rather than per-user)
authorised keys file, e.g.

   AuthorizedKeysFile  /etc/ssh/authorized_keys

We observed that the file and parent folder permissions had been
modified, to:

   /etc/ssh - 0700
   /etc/ssh/authorized_key - 0600

These permissions would be fine if the authorised keys file were in a
users home directory, but not for a centrally owned absolute file.

We investigated and identified that between the 4th and 6th January
2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-
0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included
the following fix, that led to the problem, although prior commits set
the scene for this commit to cause us trouble.

https://github.com/canonical/cloud-
init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c

While trying workarounds (e.g. oneshot service to revert permissions),
we then ran into another change that appended exit(142) to the command
option:

https://github.com/canonical/cloud-
init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f

which then meant, as root is disabled, that SSH would not work using the
authorised key pair for any user. This is because cloud-init first
writes the key for the user (e.g. ubuntu) and in our case writing the
key to /etc/ssh/authorized_keys, then writes the key for the disabled
root user to the same location, overwriting the previous write.

There are similarities to https://bugs.launchpad.net/cloud-
init/+bug/1839061, but this is a different issue.

** Affects: cloud-init
     Importance: Undecided
         Status: New

** Description changed:

  Starting on the 6th January 2021 we started observing SSH authentication
  issues in AWS AMI builds.
  
  We have SSH configured with an absolute (i.e. rather than per-user)
  authorised keys file, e.g.
  
-    AuthorizedKeysFile  /etc/ssh/authorized_keys
+    AuthorizedKeysFile  /etc/ssh/authorized_keys
  
  We observed that the file and parent folder permissions had been
  modified, to:
  
-    /etc/ssh - 0700
-    /etc/ssh/authorized_key - 0600
+    /etc/ssh - 0700
+    /etc/ssh/authorized_key - 0600
  
  These permissions would be fine if the authorised keys file were in a
  users home directory, but not for a centrally owned absolute file.
  
  We investigated and identified that between the 4th and 6th January
  2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-
  0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included
  the following fix, that led to the problem, although prior commits set
  the scene for this commit to cause us trouble.
  
  https://github.com/canonical/cloud-
  init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c
  
  While trying workarounds (e.g. oneshot service to revert permissions),
  we then ran into another change that appended exit(142) to the command
  option:
  
  https://github.com/canonical/cloud-
  init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f
  
  which then meant, as root is disabled, that SSH would not work using the
  authorised key pair for any user. This is because cloud-init first
  writes the key for the user (e.g. ubuntu) and in our case writing the
  key to /etc/ssh/authorized_keys, then writes the key for the disabled
  root user to the same location, overwriting the previous write.
+ 
+ There are similarities to https://bugs.launchpad.net/cloud-
+ init/+bug/1839061, but this is a different issue.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1911680

Title:
  Wrong access permissions of authorized keys file and parent directory
  when using absolute AuthorizedKeysFile

Status in cloud-init:
  New

Bug description:
  Starting on the 6th January 2021 we started observing SSH
  authentication issues in AWS AMI builds.

  We have SSH configured with an absolute (i.e. rather than per-user)
  authorised keys file, e.g.

     AuthorizedKeysFile  /etc/ssh/authorized_keys

  We observed that the file and parent folder permissions had been
  modified, to:

     /etc/ssh - 0700
     /etc/ssh/authorized_key - 0600

  These permissions would be fine if the authorised keys file were in a
  users home directory, but not for a centrally owned absolute file.

  We investigated and identified that between the 4th and 6th January
  2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-
  0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included
  the following fix, that led to the problem, although prior commits set
  the scene for this commit to cause us trouble.

  https://github.com/canonical/cloud-
  init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c

  While trying workarounds (e.g. oneshot service to revert permissions),
  we then ran into another change that appended exit(142) to the command
  option:

  https://github.com/canonical/cloud-
  init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f

  which then meant, as root is disabled, that SSH would not work using
  the authorised key pair for any user. This is because cloud-init first
  writes the key for the user (e.g. ubuntu) and in our case writing the
  key to /etc/ssh/authorized_keys, then writes the key for the disabled
  root user to the same location, overwriting the previous write.

  There are similarities to https://bugs.launchpad.net/cloud-
  init/+bug/1839061, but this is a different issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1911680/+subscriptions