[chef] RE: Re: Re: Re: Distribute private ssh keys via users cookbook

[Kevin Keane Subscription < >]

Here is another thought – it may or may not work in your scenario. You could automatically start an SSH connection into your new instance, and then use agent forwarding to make the key available on the new instance without actually copying it over.

 

From: Cassiano Leal [mailto:
Sent: Monday, January 14, 2013 4:04 PM
To:
Subject: [chef] Re: Re: Re: Distribute private ssh keys via users cookbook

 

Thanks for the comments and the links, Mark. This precisely is the kind of information I was hoping to get from this thread.

 

I did think about the user-data solution, but I had already realised that it's not secure at all. The alternatives you provided seem more like something I would do. I'll have to study the feasibility of each solution for my scenario, but yours and lusis' e-mails have given me what I need to decide on a strategy.

 

- cassiano

 

On Monday, January 14, 2013 at 21:59, Mark Van De Vyver wrote:

On Tue, Jan 15, 2013 at 10:47 AM, Mark Van De Vyver < "> > wrote:

lusis and kevin keane have interesting suggestions/observations. I

tend to agree that this is outside the scope of Chef though some

elements of the following can be Chefified (Cooked?). Mike suggestions

are related but the data can be read by anyone in the instance with

access to http://169.254.169.254/latest/user-data

 

I think Shlomo Swidler has covered this issue best[0]. TLDR:

The Autoscaling requirement adds a wrinkle to the use of two signed

(short expiry) s3 urls (one the encrypted credentials and the other

containing the decryption key); the autoscaling launch config would

need to be constantly updated with current URLs, some may be happy to

script this periodic change but it has risks that an AMI fires up with

an expired URL in its userdata.

 

My suggestion would be to create an EBS with the encrypted private key

on it, and mount this at startup[1]. This is a superior approach to

burning the encrypted key into an AMI. There are several options re

handling the decryption key:

- passed in via signed URL (AS launch config needs to be updated, see

above caveat re expiry)

- create a public s3 url that is unguessable (cf signed url with no

expiry) and periodically update the AS launch config (now no instance

should AS launch without access to the decryption key), deleting the

old decryption key 'url' once AS launch config is changed. This URL

should also only be readable by an aws user whose sole permission+role

is to authenticate(via url)+decrypt private key. This user should be

able to be managed by IAM , previously you'd create an AWS user with

no permissions to incur charges (i.e. no credit card provided)

 

Of course your instance start-up script must clean up after itself to

remove traces of what it has done. The only remnant will be the

unguessable/signed (changed/expired) decrypt key URL in the user data.

 

HTH?

 

PS I should have added that ideally the decrypted ssh key is stored in

some ramdisk area so that it does not survive an instance reboot.

 

 

Regards

Mark

 

[0] http://shlomoswidler.com/2009/08/how-to-keep-your-aws-credentials-on-ec2.html

[1] http://shlomoswidler.com/2010/07/storing-aws-credentials-on-an-ebs-snapshot-securely.html

 

On Tue, Jan 15, 2013 at 12:41 AM, Mike < "> > wrote:

Late to the game.

Have you explored using the instance launch user-data to write out your certs?

 

i.e.

 

#!/bin/bash -e

exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

echo BEGIN

date '+%Y-%m-%d %H:%M:%S'

 

mkdir -p /etc/chef

 

cat <<- _EOF_

<YOUR CLIENT.RB CONFIG>

_EOF_

/etc/chef/client.rb

 

cat <<- _EOF_

<YOUR VALIDATION PEM FILE>

_EOF_

/etc/chef/validation.pem

 

cat <<- _EOF_

<YOUR SSH KEY HERE>

_EOF_

> /path/to/private/ssh/key

 

(if your AMI doesn't have chef installed, install it via omnibus:)

curl -L https://www.opscode.com/chef/install.sh | bash

 

/etc/init.d/chef-client start

 

echo END

 

 

This is effectively the `knife bootstrap` process, executed by your

EC2 AutoScaler. This works with most "vetted" AMI images that do the

fun 'cloud-init' and all that jazz.

 

 

On Mon, Jan 14, 2013 at 5:02 AM, Cassiano Leal < "> > wrote:

Since I want to auto-scale these nodes, I'll have to keep the validation.pem

in the AMI that will be used to boot up. That's one of the solutions that I

though -- store the SSH key in the AMI.

 

I understand the security implications of copying SSH keys around or leaving

them in an image on the Amazon cloud, but I still fail to see a better

solution. The fact that no alternative covering my requirements have

appeared in this thread makes me all the more confident that I'm not doing

something too crazy.

 

Recapitulating requirements:

 

- Auto-scaling -- this is the key point

- New nodes automatically register themselves on the chef server and

converge

- Newly auto-scaled nodes are able to clone/update from git repos:

* on the first chef run

* no manual copy of SSH keys to the node

* no manual authorisation of a new key on bitbucket

 

I guess the better question would be: how are people auto-scaling chef

nodes?

 

On 11 January 2013 22:38, Dan Razzell < "> > wrote:

 

+1

 

As Kevin pointed out yesterday, there's something essentially upside-down

about your PKI if you're copying private keys around.

 

 

On 13-01-11 04:31 PM, Kevin Keane (subscriptions) wrote:

 

I see two options for that:

 

 

 

- Upload the SSH key at the same time you bootstrap the new instance. You

could even add that to a custom bootstrap script, to copy it along with the

validation.pem

 

- Regularly, say, every 30 minutes, run a knife search and upload the SSH

key to each server it finds.

 

 

-----Original message-----

From: Cassiano Leal < "> >

Sent: Fri 01-11-2013 04:18 am

Subject: [chef] Re: Re: RE: Distribute private ssh keys via users cookbook

To: "> ;

All perfectly good points. How do I deal with auto-scaling, though? I don't

really want to be woken up in the middle of the night just because site

traffic has peaked and more servers are spinning up so that I can SCP a

private key to each new server. :)

 

- cassiano

 

 

On Friday, January 11, 2013 at 02:18, Kevin Keane (subscriptions) wrote:

 

I see where you are coming from! In this type of scenario, I would probably

step outside the Chef setup.

 

 

 

The security problem in Chef ultimately arises because you allow the clients

to *pull* the private keys. As a result, a hacker can potentially simply use

HTTP to download your databag, or even sniff the data going over the wire.

Databag encryption, client.pem and validation.pem are all protections, but

they can't compete (and aren't designed to compete) with established

security mechanisms such as SSH, PGP, ...

 

 

 

So maybe Chef isn't the best tool for this particular job?

 

 

 

I would instead use SCP to copy your private key from a single secure server

to wherever they need to go. The big advantage is that because it is a

push-based system, you have full control over which machines do the SSH keys

get copied to.

 

 

 

-----Original message-----

From: Cassiano Leal < "> >

Sent: Thu 01-10-2013 04:23 am

Subject: [chef] Re: RE: Distribute private ssh keys via users cookbook

To: "> ;

Hi Kevin,

 

Thanks for sharing your thoughts. Let me describe my scenario and what I'm

trying to accomplish.

 

I have VMs that spin up automatically on AWS. These VMs host several

applications, most written in PHP and RoR. The source code for all the apps

resides on bitbucket Git repos.

 

I'm setting up application deployment via the deploy resource [0]. For that,

I have created a "deploy" user through the users cookbook, but I need this

user to have read-only access to the repos. For that, I have a bitbucket

user with read-only access and I need to configure this user with the public

keys, and that's done on bitbucket website.

 

If I create a new key pair on each VM, I'll have to manually authorise the

new key on the bitbucket website so that the application can be deployed.

That would completely defeat auto-scaling and would make my life a bit

worse, specially on peak times. :)

 

I understand the security implications of this approach, but I still find it

all that much better than having to manage potentially infinite key pairs on

a website. This key would have read only access to our repos, and if I find

out that it has been compromised all I need to do is revoke it on bitbucket,

create a new key pair and redistribute it.

 

I can do that on a recipe manually, but it would be just a bit easier and

more concise if it was possible via the users cookbook.

 

Also, it beats me why the cookbook allows me to distribute private keys via

an unencrypted data bag [1], and has no secure-ish solution for that.

 

Again, I'm open for hearing other solutions to this problem. If they're more

secure, all the better, as long as it doesn't make me go to bitbucket.org

and insert a new pubkey each time I spin up a new server. :)

 

[0] http://docs.opscode.com/resource_deploy.html

[1]

https://github.com/opscode-cookbooks/users/blob/master/providers/manage.rb#L112-L122

 

- cassiano

 

 

On Thursday, January 10, 2013 at 09:51, Kevin Keane (subscriptions) wrote:

 

It seems to me that there might be a problem with what you are trying to

accomplish in the first place. When you distribute private SSH keys through

*any* means, you have a security problem. Doesn't matter if you are using

encrypted databags, or even manually copy the keys over. Unless they are

your own personal keys, it defeats the very concept behind public key

encryption - and if they *are* your personal keys, you should only manually

copy them, not put them in an automated distribution system.

 

 

 

Once generated, private SSH keys should never leave that user's control. You

can use chef to distribute the corresponding public keys - and you don't

even need an encrypted databag. Public keys are not sensitive; you can leave

them unencrypted.

 

 

 

If your users are humans, have them generated their SSH key pairs, and

create a databag with all the public keys that you want to distribute. Hint:

the users cookbook already has a mechanism built in to do that.

 

 

 

If you need SSH key pairs for some automated tasks (say, a cron script doing

a nightly rsync), generate a *new* key pair, and store only the public key

in a node attribute. You can use an execute or script resource to generate

the key pair; just make sure to check if the key may already exist (you

don't want to clobber an already-existing key pair).

 

 

On the SSH client side, you'd extract all the relevant public keys from your

databag or node attributes, and add them to the authorized_keys file as

needed.

 

 

 

-----Original message-----

From: Cassiano Leal < "> >

Sent: Wed 01-09-2013 11:40 am

Subject: [chef] Distribute private ssh keys via users cookbook

To: "> ;

Hi!

 

Is there a way to securely distribute private ssh keys through the users

community cookbook?

 

In my setup I have a user deploy that will fetch from git, and I need that

user to have a SSH key that's authorized on the git repo. I saw that the

users cookbook will use "ssh_private_key" and "ssh_public_key" data bag

items, but those would be unencrypted, so not secure.

 

If people are using a different approach, I'd like to hear about that too.

 

Thanks!

--

Cassiano Leal