cloud-init-dev team mailing list archive

[Balint Reczey <balint.reczey@xxxxxxxxxxxxx>]

On Thu, Feb 13, 2020 at 4:51 PM Ryan Harper <1835114@xxxxxxxxxxxxxxxxxx> wrote:
>
> On Thu, Feb 13, 2020 at 6:20 AM Balint Reczey <balint.reczey@xxxxxxxxxxxxx>
> wrote:
>
> > @raharper I agree with the concern regarding the manipulation of sshd
> > config. To minimize the collision with cloud-init this package does not
> > change /etc/ssh/sshd_config like cloud-init does, but overrides the
> > configuration value with a systemd drop-in. The drop-in is placed at the
> >
>
> This does not avoid a collision with cloud-init.  That is, the current
> implementation
> is to modify the arguments passed to sshd instead of making changes to sshd
> config.
> Multiple sources of configuration is still an issue.  Someone inspecting
> sshd config will
> not see that the AuthorizedKeyCommand is being passed via arguments,
> further they
> may not know where this additional override is coming from w.r.t the
> systemd drop-in
> config.  If a user (or program via cloud-init config) were to modify sshd
> config and
> set their own AuthorizedKeyCommand, this scenario will fail for them since
> the
> ec2-instance-connect drop in will always take precedence.
>
> Shouldn't the drop-in program examine sshd config?

A agree that having multiple sources of configuration can be confusing
for _some_ users, but using systemd drop-ins is a well established
practice for extending configuration of services:

$ apt-file search -x '^/lib/systemd/system/.*.d/.*.conf' | wc -l
21

Also the drop-in is and the changed configuration is reasonably easy
to discover in ssh's status:

$ service ssh status
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor
preset: enabled)
  Drop-In: /lib/systemd/system/ssh.service.d
           └─ec2-instance-connect.conf
   Active: active (running) since Mon 2020-02-24 16:58:25 UTC; 22h ago
 Main PID: 27725 (sshd)
    Tasks: 1 (limit: 1152)
   CGroup: /system.slice/ssh.service
           └─27725 /usr/sbin/sshd -D -o AuthorizedKeysCommand
/usr/share/ec2-instance-connect/eic_run_authorized_keys %u %f -o
AuthorizedKeysCommandUser ec2-instance-connect

The drop-in could in theory examine sshd_config, but sshd_config is
not the only possible source of the configuration and I see this
behaviour more confusing than the simple discoverable override. Users
not wanting to use the override can simply remove the
ec2-instance-connect -package or override the drop-in in
/etc/systemd/system/

> > Regarding the potential user confusion when the user also sets ssh keys
> > using cloud-init eic_run_authorized_keys is designed to _merge_ the keys
> > used by Instance Connect with the other keys in use thus the users can
> > continue to use their keys deployed by cloud-init or the ones deployed
> > by other means.
> >
>
> Could you elaborate (or point to documentation) on this merging?   I'm
> familiar
> with how AuthorizedKeyCommand works if it does not produce any output, sshd
> will fall back to AuthorizedKey files specifiedin sshd_config; but I've not
> seen any
> discussion on the merging you suggest.

I was pretty sure I saw the key I've set up on the AWS console to
access the machine showing up with the key for instance connect, but
it seems I was wrong.
Users can keep connecting using other keys, because
eic_run_authorized_keys fails when it queries unknown offered public
key and ssh falls back to AuthorizedKey files:

ssh -v -o ControlPath=none ec2-52-15-81-45.us-east-2.compute.amazonaws.com
...
debug1: Offering public key: /home/rbalint/.ssh/id_rsa RSA
SHA256:JkIF2zsiEkPmW29GdF5gzYTgPSYsxOtn1EPk2CEDS+Q agent
...

sudo tail -n40 /var/log/auth.log
...
Feb 25 15:11:39 ip-172-31-42-192 sshd[31076]: AuthorizedKeysCommand
/usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu
SHA256:JkIF2zsiEkPmW29GdF5gzYTgPSYsxOtn1EPk2CEDS+Q failed, status 22
Feb 25 15:11:39 ip-172-31-42-192 sshd[31076]: Accepted publickey for
ubuntu from 178.48.204.120 port 58898 ssh2: RSA
SHA256:JkIF2zsiEkPmW29GdF5gzYTgPSYsxOtn1EPk2CEDS+Q
...

So in effect the union of the Instance Connect key and the other keys
can be used causing no blocked logins.

> > I also agree that there is additional overhead for each ssh connection,
> > but while testing the package I have not found that excessive. We may
> >
>
> Instead of vague statements, capturing actual values would be most
> useful.

Please see my measurements below.

> > need further evaluation of the impact on the ssh service before adding
> > the package to the AMIs by default, but I think this can be done after
> > finishing the MIR process.
> >
>
> I generally disagree with letting something in to main that we want to
> support
> and "we can fix this after we agree to MIR".

Generally I'd disagree, too, but this package is integrating Ubuntu to
Amazon's cloud the way Amazon would like to have that integration and
the questioned part is a tradeoff between speed, security and ease of
use. I hope my measurements below show that the performance impact is
not severe and is ok to accept the package to main.

> > Upstream already answered @paelzer's caching proposal, and the package
> > is installed on Amazon Linux 2 by default already, thus I believe
> > upstream's attention is warranted regarding the overhead.
> >
>
> The response is dismissive; security and usability override any concern
> around
> overhead of the "security" feature.  This reinforces the need to benchmark
> the
> actual overhead per connection with synthetic and actual workloakds (I
> suspect
> something like a juju deployment to Ec2 of something like k8s or the like
> mode
> with lots of instances would be a reasonable workload to compare with and
> without
> the instance connect enabled.

I've tested ssh-ing to localhost without and then with the
ec2-instance-connect-package installed:
ssh-keygen -t rsa -b 4096

ubuntu@ip-172-31-45-228:~$ for i in $(seq 15); do (time ssh -o
ControlPath=none localhost echo ) 2>&1 | grep real ; done | sort
real 0m0.611s
real 0m0.612s
real 0m0.616s
real 0m0.616s
real 0m0.616s
real 0m0.616s
real 0m0.620s
real 0m0.620s
real 0m0.620s
real 0m0.620s
real 0m0.620s
real 0m0.620s
real 0m0.624s
real 0m0.624s
real 0m0.624s

ubuntu@ip-172-31-45-228:~$ sudo apt install ec2-instance-connect
...
Unpacking ec2-instance-connect (1.1.12+dfsg1-0ubuntu3~18.04.0) ...
Setting up ec2-instance-connect (1.1.12+dfsg1-0ubuntu3~18.04.0) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/ec2-instance-connect.service
→ /lib/systemd/system/ec2-instance-connect.service.
sshd override added, restarting daemon

ubuntu@ip-172-31-45-228:~$ for i in $(seq 15); do (time ssh -o
ControlPath=none localhost echo ) 2>&1 | grep real ; done | sort
real 0m0.680s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.684s
real 0m0.688s
real 0m0.688s
real 0m0.688s
real 0m0.959s

There is clearly a regression in connection speed, but IMO ~70ms is an
acceptable cost for the comfort of being able to connect to an
instance just by clicking on 'Connect' on the AWS Console.

Connecting to the instance from outside, from my coworking place is
~3.4s.

I also believe that a provisioner/configurator systemd should use
persistent connections where only the first connection suffers the
connection delay and further commands via ssh look like this:
ubuntu@ip-172-31-45-228:~$ time ssh -o
ControlPath=~/.ssh/ssh_control_socket_%C -o ControlMaster=auto -o
ControlPersist=600 localhost echo ok
ok

real 0m0.007s
user 0m0.000s
sys 0m0.004s

I have not tested AWS's scalability by connecting to many instances in
parallel, but checking Amazon's infrastructure that way seems out of
scope for the main inclusion process.

Balint

>
> Ryan
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1835114
>
> Title:
>   [MIR] ec2-instance-connect
>
> Status in ec2-instance-connect package in Ubuntu:
>   Incomplete
>
> Bug description:
>   [Availability]
>   ec2-instance-connect is in the Ubuntu archive, and available for all supported releases. It is available on all architectures despite only being useful on Amazon EC2 instances.
>
>   [Rationale]
>   This package is useful on Amazon EC2 instances to make use of a new feature:
>   Instance Connect; which allows storing SSH keys for access online in the Amazon systems. These SSH keys are then retrieved to be used by the system's SSH service, collated with pre-existing keys as deployed on the system.
>
>   Installing the package enables the use of Instance Connect on an
>   instance.
>
>   [Security]
>   This is a new package, and as such has no security history to speak of.
>
>   [Quality Assurance]
>   The package consists in a few shell scripts that are difficult to test by
>   themselves due to the high reliance on Amazon's Instance Connect service;
>   which is online and limited to use on Amazon instances.
>
>   Given that it's a new package, there are no long-term outstanding bugs in
>   Ubuntu or Debian. The package is only maintained in Ubuntu at the moment.
>
>   This package deals with special "hardware"; it is only useful on Amazon
>   instances, and its support is required as a default deployment on such
>   instances when deployed with Ubuntu.
>
>   [UI Standards]
>   Not applicable. This service is command-line only and has no configuration options.
>
>   [Dependencies]
>   There are no special dependencies to speak of.
>
>   [Standards Compliance]
>   This package has been thoroughly reviewed by a few Canonical engineers, there are no standards violations known.
>
>   [Maintenance]
>   This package is to be owned by the Ubuntu Foundations team.
>
>   [Background Information]
>   This is Amazon-specific, as previously mentioned.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/ec2-instance-connect/+bug/1835114/+subscriptions


--
Balint Reczey
Ubuntu & Debian Developer

-- 
You received this bug notification because you are a member of cloud-
init Commiters, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1835114

Title:
  [MIR] ec2-instance-connect

Status in ec2-instance-connect package in Ubuntu:
  Incomplete

Bug description:
  [Availability]
  ec2-instance-connect is in the Ubuntu archive, and available for all supported releases. It is available on all architectures despite only being useful on Amazon EC2 instances.

  [Rationale]
  This package is useful on Amazon EC2 instances to make use of a new feature:
  Instance Connect; which allows storing SSH keys for access online in the Amazon systems. These SSH keys are then retrieved to be used by the system's SSH service, collated with pre-existing keys as deployed on the system.

  Installing the package enables the use of Instance Connect on an
  instance.

  [Security]
  This is a new package, and as such has no security history to speak of.

  [Quality Assurance]
  The package consists in a few shell scripts that are difficult to test by
  themselves due to the high reliance on Amazon's Instance Connect service;
  which is online and limited to use on Amazon instances.

  Given that it's a new package, there are no long-term outstanding bugs in
  Ubuntu or Debian. The package is only maintained in Ubuntu at the moment.

  This package deals with special "hardware"; it is only useful on Amazon
  instances, and its support is required as a default deployment on such
  instances when deployed with Ubuntu.

  [UI Standards]
  Not applicable. This service is command-line only and has no configuration options.

  [Dependencies]
  There are no special dependencies to speak of.

  [Standards Compliance]
  This package has been thoroughly reviewed by a few Canonical engineers, there are no standards violations known.

  [Maintenance]
  This package is to be owned by the Ubuntu Foundations team.

  [Background Information]
  This is Amazon-specific, as previously mentioned.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ec2-instance-connect/+bug/1835114/+subscriptions