[chef] Re: Re: Distribute private ssh keys via users cookbook

[Mark Van De Vyver < >]

lusis and kevin keane have interesting suggestions/observations. I
tend to agree that this is outside the scope of Chef though some
elements of the following can be Chefified (Cooked?). Mike suggestions
are related but  the data can be read by anyone in the instance with
access to http://169.254.169.254/latest/user-data

I think Shlomo Swidler has covered this issue best[0]. TLDR:
The Autoscaling requirement adds a wrinkle to the use of two signed
(short expiry) s3 urls (one the encrypted credentials and the other
containing the decryption key); the autoscaling launch config would
need to be constantly updated with current URLs, some may be happy to
script this periodic change but it has risks that an AMI fires up with
an expired URL in its userdata.

My suggestion would be to create an EBS with the encrypted private key
on it, and mount this at startup[1]. This is a superior approach to
burning the encrypted key into an AMI. There are several options re
handling the decryption key:
 - passed in via signed URL (AS launch config needs to be updated, see
above caveat re expiry)
 - create a public s3 url that is unguessable (cf signed url with no
expiry) and periodically update the AS launch config (now no instance
should AS launch without access to the decryption key), deleting the
old decryption key 'url' once AS launch config is changed. This URL
should also only be readable by an aws user whose sole permission+role
is to authenticate(via url)+decrypt private key.  This user should be
able to be managed by IAM , previously you'd create an AWS user with
no permissions to incur charges (i.e. no credit card provided)

Of course your instance start-up script must clean up after itself to
remove traces of what it has done. The only remnant will be the
unguessable/signed (changed/expired) decrypt key URL in the user data.

HTH?

Regards
Mark

[0] 
http://shlomoswidler.com/2009/08/how-to-keep-your-aws-credentials-on-ec2.html
[1] 
http://shlomoswidler.com/2010/07/storing-aws-credentials-on-an-ebs-snapshot-securely.html

On Tue, Jan 15, 2013 at 12:41 AM, Mike 
<
 >
 wrote:
> Late to the game.
> Have you explored using the instance launch user-data to write out your 
> certs?
>
> i.e.
>
> #!/bin/bash -e
> exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 
> 2>&1
> echo BEGIN
> date '+%Y-%m-%d %H:%M:%S'
>
> mkdir -p /etc/chef
>
> cat <<- _EOF_
> <YOUR CLIENT.RB CONFIG>
>         _EOF_
>> /etc/chef/client.rb
>
> cat <<- _EOF_
> <YOUR VALIDATION PEM FILE>
>         _EOF_
>> /etc/chef/validation.pem
>
> cat <<- _EOF_
> <YOUR SSH KEY HERE>
>         _EOF_
>  > /path/to/private/ssh/key
>
> (if your AMI doesn't have chef installed, install it via omnibus:)
> curl -L https://www.opscode.com/chef/install.sh ;| bash
>
> /etc/init.d/chef-client start
>
> echo END
>
>
> This is effectively the `knife bootstrap` process, executed by your
> EC2 AutoScaler. This works with most "vetted" AMI images that do the
> fun 'cloud-init' and all that jazz.
>
>
> On Mon, Jan 14, 2013 at 5:02 AM, Cassiano Leal 
> <
 >
>  wrote:
>> Since I want to auto-scale these nodes, I'll have to keep the 
>> validation.pem
>> in the AMI that will be used to boot up. That's one of the solutions that I
>> though -- store the SSH key in the AMI.
>>
>> I understand the security implications of copying SSH keys around or 
>> leaving
>> them in an image on the Amazon cloud, but I still fail to see a better
>> solution. The fact that no alternative covering my requirements have
>> appeared in this thread makes me all the more confident that I'm not doing
>> something too crazy.
>>
>> Recapitulating requirements:
>>
>> - Auto-scaling -- this is the key point
>> - New nodes automatically register themselves on the chef server and
>> converge
>> - Newly auto-scaled nodes are able to clone/update from git repos:
>>   * on the first chef run
>>   * no manual copy of SSH keys to the node
>>   * no manual authorisation of a new key on bitbucket
>>
>> I guess the better question would be: how are people auto-scaling chef
>> nodes?
>>
>> On 11 January 2013 22:38, Dan Razzell 
>> <
 >
>>  wrote:
>>
>> +1
>>
>> As Kevin pointed out yesterday, there's something essentially upside-down
>> about your PKI if you're copying private keys around.
>>
>>
>> On 13-01-11 04:31 PM, Kevin Keane (subscriptions) wrote:
>>
>> I see two options for that:
>>
>>
>>
>> - Upload the SSH key at the same time you bootstrap the new instance. You
>> could even add that to a custom bootstrap script, to copy it along with the
>> validation.pem
>>
>> - Regularly, say, every 30 minutes, run a knife search and upload the SSH
>> key to each server it finds.
>>
>>
>> -----Original message-----
>> From: Cassiano Leal 
>> <
 >
>> Sent: Fri 01-11-2013 04:18 am
>> Subject: [chef] Re: Re: RE: Distribute private ssh keys via users cookbook
>> To: 
>> 
 ;
>> All perfectly good points. How do I deal with auto-scaling, though? I don't
>> really want to be woken up in the middle of the night just because site
>> traffic has peaked and more servers are spinning up so that I can SCP a
>> private key to each new server. :)
>>
>> - cassiano
>>
>>
>> On Friday, January 11, 2013 at 02:18, Kevin Keane (subscriptions) wrote:
>>
>> I see where you are coming from! In this type of scenario, I would probably
>> step outside the Chef setup.
>>
>>
>>
>> The security problem in Chef ultimately arises because you allow the 
>> clients
>> to *pull* the private keys. As a result, a hacker can potentially simply 
>> use
>> HTTP to download your databag, or even sniff the data going over the wire.
>> Databag encryption, client.pem and validation.pem are all protections, but
>> they can't compete (and aren't designed to compete) with established
>> security mechanisms such as SSH, PGP, ...
>>
>>
>>
>> So maybe Chef isn't the best tool for this particular job?
>>
>>
>>
>> I would instead use SCP to copy your private key from a single secure 
>> server
>> to wherever they need to go. The big advantage is that because it is a
>> push-based system, you have full control over which machines do the SSH 
>> keys
>> get copied to.
>>
>>
>>
>> -----Original message-----
>> From: Cassiano Leal 
>> <
 >
>> Sent: Thu 01-10-2013 04:23 am
>> Subject: [chef] Re: RE: Distribute private ssh keys via users cookbook
>> To: 
>> 
 ;
>> Hi Kevin,
>>
>> Thanks for sharing your thoughts. Let me describe my scenario and what I'm
>> trying to accomplish.
>>
>> I have VMs that spin up automatically on AWS. These VMs host several
>> applications, most written in PHP and RoR. The source code for all the apps
>> resides on bitbucket Git repos.
>>
>> I'm setting up application deployment via the deploy resource [0]. For 
>> that,
>> I have created a "deploy" user through the users cookbook, but I need this
>> user to have read-only access to the repos. For that, I have a bitbucket
>> user with read-only access and I need to configure this user with the 
>> public
>> keys, and that's done on bitbucket website.
>>
>> If I create a new key pair on each VM, I'll have to manually authorise the
>> new key on the bitbucket website so that the application can be deployed.
>> That would completely defeat auto-scaling and would make my life a bit
>> worse, specially on peak times. :)
>>
>> I understand the security implications of this approach, but I still find 
>> it
>> all that much better than having to manage potentially infinite key pairs 
>> on
>> a website. This key would have read only access to our repos, and if I find
>> out that it has been compromised all I need to do is revoke it on 
>> bitbucket,
>> create a new key pair and redistribute it.
>>
>> I can do that on a recipe manually, but it would be just a bit easier and
>> more concise if it was possible via the users cookbook.
>>
>> Also, it beats me why the cookbook allows me to distribute private keys via
>> an unencrypted data bag [1], and has no secure-ish solution for that.
>>
>> Again, I'm open for hearing other solutions to this problem. If they're 
>> more
>> secure, all the better, as long as it doesn't make me go to bitbucket.org
>> and insert a new pubkey each time I spin up a new server. :)
>>
>> [0] http://docs.opscode.com/resource_deploy.html
>> [1]
>> https://github.com/opscode-cookbooks/users/blob/master/providers/manage.rb#L112-L122
>>
>> - cassiano
>>
>>
>> On Thursday, January 10, 2013 at 09:51, Kevin Keane (subscriptions) wrote:
>>
>> It seems to me that there might be a problem with what you are trying to
>> accomplish in the first place. When you distribute private SSH keys through
>> *any* means, you have a security problem. Doesn't matter if you are using
>> encrypted databags, or even manually copy the keys over. Unless they are
>> your own personal keys, it defeats the very concept behind public key
>> encryption - and if they *are* your personal keys, you should only manually
>> copy them, not put them in an automated distribution system.
>>
>>
>>
>> Once generated, private SSH keys should never leave that user's control. 
>> You
>> can use chef to distribute the corresponding public keys - and you don't
>> even need an encrypted databag. Public keys are not sensitive; you can 
>> leave
>> them unencrypted.
>>
>>
>>
>> If your users are humans, have them generated their SSH key pairs, and
>> create a databag with all the public keys that you want to distribute. 
>> Hint:
>> the users cookbook already has a mechanism built in to do that.
>>
>>
>>
>> If you need SSH key pairs for some automated tasks (say, a cron script 
>> doing
>> a nightly rsync), generate a *new* key pair, and store only the public key
>> in a node attribute. You can use an execute or script resource to generate
>> the key pair; just make sure to check if the key may already exist (you
>> don't want to clobber an already-existing key pair).
>>
>>
>> On the SSH client side, you'd extract all the relevant public keys from 
>> your
>> databag or node attributes, and add them to the authorized_keys file as
>> needed.
>>
>>
>>
>> -----Original message-----
>> From: Cassiano Leal 
>> <
 >
>> Sent: Wed 01-09-2013 11:40 am
>> Subject: [chef] Distribute private ssh keys via users cookbook
>> To: 
>> 
 ;
>> Hi!
>>
>> Is there a way to securely distribute private ssh keys through the users
>> community cookbook?
>>
>> In my setup I have a user deploy that will fetch from git, and I need that
>> user to have a SSH key that's authorized on the git repo. I saw that the
>> users cookbook will use "ssh_private_key" and "ssh_public_key" data bag
>> items, but those would be unencrypted, so not secure.
>>
>> If people are using a different approach, I'd like to hear about that too.
>>
>> Thanks!
>> --
>> Cassiano Leal
>>
>>
>>
>>
>>
>>
>>