General discussion about Chef

[chef] Re: Re: Re: Distribute private ssh keys via users cookbook


Chronological Thread 
  • From: Cassiano Leal < >
  • To:
  • Subject: [chef] Re: Re: Re: Distribute private ssh keys via users cookbook
  • Date: Mon, 14 Jan 2013 22:04:12 -0200
Thanks for the comments and the links, Mark. This precisely is the kind of information I was hoping to get from this thread.

I did think about the user-data solution, but I had already realised that it's not secure at all. The alternatives you provided seem more like something I would do. I'll have to study the feasibility of each solution for my scenario, but yours and lusis' e-mails have given me what I need to decide on a strategy.

- cassiano

On Monday, January 14, 2013 at 21:59, Mark Van De Vyver wrote:

On Tue, Jan 15, 2013 at 10:47 AM, Mark Van De Vyver < "> > wrote:
lusis and kevin keane have interesting suggestions/observations. I
tend to agree that this is outside the scope of Chef though some
elements of the following can be Chefified (Cooked?). Mike suggestions
are related but the data can be read by anyone in the instance with

I think Shlomo Swidler has covered this issue best[0]. TLDR:
The Autoscaling requirement adds a wrinkle to the use of two signed
(short expiry) s3 urls (one the encrypted credentials and the other
containing the decryption key); the autoscaling launch config would
need to be constantly updated with current URLs, some may be happy to
script this periodic change but it has risks that an AMI fires up with
an expired URL in its userdata.

My suggestion would be to create an EBS with the encrypted private key
on it, and mount this at startup[1]. This is a superior approach to
burning the encrypted key into an AMI. There are several options re
handling the decryption key:
- passed in via signed URL (AS launch config needs to be updated, see
above caveat re expiry)
- create a public s3 url that is unguessable (cf signed url with no
expiry) and periodically update the AS launch config (now no instance
should AS launch without access to the decryption key), deleting the
old decryption key 'url' once AS launch config is changed. This URL
should also only be readable by an aws user whose sole permission+role
is to authenticate(via url)+decrypt private key. This user should be
able to be managed by IAM , previously you'd create an AWS user with
no permissions to incur charges (i.e. no credit card provided)

Of course your instance start-up script must clean up after itself to
remove traces of what it has done. The only remnant will be the
unguessable/signed (changed/expired) decrypt key URL in the user data.

HTH?

PS I should have added that ideally the decrypted ssh key is stored in
some ramdisk area so that it does not survive an instance reboot.


Regards
Mark


On Tue, Jan 15, 2013 at 12:41 AM, Mike < "> > wrote:
Late to the game.
Have you explored using the instance launch user-data to write out your certs?

i.e.

#!/bin/bash -e
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'

mkdir -p /etc/chef

cat <<- _EOF_
<YOUR CLIENT.RB CONFIG>
_EOF_
/etc/chef/client.rb

cat <<- _EOF_
<YOUR VALIDATION PEM FILE>
_EOF_
/etc/chef/validation.pem

cat <<- _EOF_
<YOUR SSH KEY HERE>
_EOF_
> /path/to/private/ssh/key

(if your AMI doesn't have chef installed, install it via omnibus:)

/etc/init.d/chef-client start

echo END


This is effectively the `knife bootstrap` process, executed by your
EC2 AutoScaler. This works with most "vetted" AMI images that do the
fun 'cloud-init' and all that jazz.


On Mon, Jan 14, 2013 at 5:02 AM, Cassiano Leal < "> > wrote:
Since I want to auto-scale these nodes, I'll have to keep the validation.pem
in the AMI that will be used to boot up. That's one of the solutions that I
though -- store the SSH key in the AMI.

I understand the security implications of copying SSH keys around or leaving
them in an image on the Amazon cloud, but I still fail to see a better
solution. The fact that no alternative covering my requirements have
appeared in this thread makes me all the more confident that I'm not doing
something too crazy.

Recapitulating requirements:

- Auto-scaling -- this is the key point
- New nodes automatically register themselves on the chef server and
converge
- Newly auto-scaled nodes are able to clone/update from git repos:
* on the first chef run
* no manual copy of SSH keys to the node
* no manual authorisation of a new key on bitbucket

I guess the better question would be: how are people auto-scaling chef
nodes?

On 11 January 2013 22:38, Dan Razzell < "> > wrote:

+1

As Kevin pointed out yesterday, there's something essentially upside-down
about your PKI if you're copying private keys around.


On 13-01-11 04:31 PM, Kevin Keane (subscriptions) wrote:

I see two options for that:



- Upload the SSH key at the same time you bootstrap the new instance. You
could even add that to a custom bootstrap script, to copy it along with the
validation.pem

- Regularly, say, every 30 minutes, run a knife search and upload the SSH
key to each server it finds.


-----Original message-----
Sent: Fri 01-11-2013 04:18 am
Subject: [chef] Re: Re: RE: Distribute private ssh keys via users cookbook
All perfectly good points. How do I deal with auto-scaling, though? I don't
really want to be woken up in the middle of the night just because site
traffic has peaked and more servers are spinning up so that I can SCP a
private key to each new server. :)

- cassiano


On Friday, January 11, 2013 at 02:18, Kevin Keane (subscriptions) wrote:

I see where you are coming from! In this type of scenario, I would probably
step outside the Chef setup.



The security problem in Chef ultimately arises because you allow the clients
to *pull* the private keys. As a result, a hacker can potentially simply use
HTTP to download your databag, or even sniff the data going over the wire.
Databag encryption, client.pem and validation.pem are all protections, but
they can't compete (and aren't designed to compete) with established
security mechanisms such as SSH, PGP, ...



So maybe Chef isn't the best tool for this particular job?



I would instead use SCP to copy your private key from a single secure server
to wherever they need to go. The big advantage is that because it is a
push-based system, you have full control over which machines do the SSH keys
get copied to.



-----Original message-----
Sent: Thu 01-10-2013 04:23 am
Subject: [chef] Re: RE: Distribute private ssh keys via users cookbook
Hi Kevin,

Thanks for sharing your thoughts. Let me describe my scenario and what I'm
trying to accomplish.

I have VMs that spin up automatically on AWS. These VMs host several
applications, most written in PHP and RoR. The source code for all the apps
resides on bitbucket Git repos.

I'm setting up application deployment via the deploy resource [0]. For that,
I have created a "deploy" user through the users cookbook, but I need this
user to have read-only access to the repos. For that, I have a bitbucket
user with read-only access and I need to configure this user with the public
keys, and that's done on bitbucket website.

If I create a new key pair on each VM, I'll have to manually authorise the
new key on the bitbucket website so that the application can be deployed.
That would completely defeat auto-scaling and would make my life a bit
worse, specially on peak times. :)

I understand the security implications of this approach, but I still find it
all that much better than having to manage potentially infinite key pairs on
a website. This key would have read only access to our repos, and if I find
out that it has been compromised all I need to do is revoke it on bitbucket,
create a new key pair and redistribute it.

I can do that on a recipe manually, but it would be just a bit easier and
more concise if it was possible via the users cookbook.

Also, it beats me why the cookbook allows me to distribute private keys via
an unencrypted data bag [1], and has no secure-ish solution for that.

Again, I'm open for hearing other solutions to this problem. If they're more
secure, all the better, as long as it doesn't make me go to bitbucket.org
and insert a new pubkey each time I spin up a new server. :)

[1]

- cassiano


On Thursday, January 10, 2013 at 09:51, Kevin Keane (subscriptions) wrote:

It seems to me that there might be a problem with what you are trying to
accomplish in the first place. When you distribute private SSH keys through
*any* means, you have a security problem. Doesn't matter if you are using
encrypted databags, or even manually copy the keys over. Unless they are
your own personal keys, it defeats the very concept behind public key
encryption - and if they *are* your personal keys, you should only manually
copy them, not put them in an automated distribution system.



Once generated, private SSH keys should never leave that user's control. You
can use chef to distribute the corresponding public keys - and you don't
even need an encrypted databag. Public keys are not sensitive; you can leave
them unencrypted.



If your users are humans, have them generated their SSH key pairs, and
create a databag with all the public keys that you want to distribute. Hint:
the users cookbook already has a mechanism built in to do that.



If you need SSH key pairs for some automated tasks (say, a cron script doing
a nightly rsync), generate a *new* key pair, and store only the public key
in a node attribute. You can use an execute or script resource to generate
the key pair; just make sure to check if the key may already exist (you
don't want to clobber an already-existing key pair).


On the SSH client side, you'd extract all the relevant public keys from your
databag or node attributes, and add them to the authorized_keys file as
needed.



-----Original message-----
Sent: Wed 01-09-2013 11:40 am
Subject: [chef] Distribute private ssh keys via users cookbook
Hi!

Is there a way to securely distribute private ssh keys through the users
community cookbook?

In my setup I have a user deploy that will fetch from git, and I need that
user to have a SSH key that's authorized on the git repo. I saw that the
users cookbook will use "ssh_private_key" and "ssh_public_key" data bag
items, but those would be unencrypted, so not secure.

If people are using a different approach, I'd like to hear about that too.

Thanks!
--
Cassiano Leal


Archive powered by MHonArc 2.6.16.

§