[chef] Re: Re: Distribute private ssh keys via users cookbook

[Mark Van De Vyver < >]

On Tue, Jan 15, 2013 at 10:47 AM, Mark Van De Vyver 
<
 >
 wrote:
> lusis and kevin keane have interesting suggestions/observations. I
> tend to agree that this is outside the scope of Chef though some
> elements of the following can be Chefified (Cooked?). Mike suggestions
> are related but  the data can be read by anyone in the instance with
> access to http://169.254.169.254/latest/user-data
>
> I think Shlomo Swidler has covered this issue best[0]. TLDR:
> The Autoscaling requirement adds a wrinkle to the use of two signed
> (short expiry) s3 urls (one the encrypted credentials and the other
> containing the decryption key); the autoscaling launch config would
> need to be constantly updated with current URLs, some may be happy to
> script this periodic change but it has risks that an AMI fires up with
> an expired URL in its userdata.
>
> My suggestion would be to create an EBS with the encrypted private key
> on it, and mount this at startup[1]. This is a superior approach to
> burning the encrypted key into an AMI. There are several options re
> handling the decryption key:
>  - passed in via signed URL (AS launch config needs to be updated, see
> above caveat re expiry)
>  - create a public s3 url that is unguessable (cf signed url with no
> expiry) and periodically update the AS launch config (now no instance
> should AS launch without access to the decryption key), deleting the
> old decryption key 'url' once AS launch config is changed. This URL
> should also only be readable by an aws user whose sole permission+role
> is to authenticate(via url)+decrypt private key.  This user should be
> able to be managed by IAM , previously you'd create an AWS user with
> no permissions to incur charges (i.e. no credit card provided)
>
> Of course your instance start-up script must clean up after itself to
> remove traces of what it has done. The only remnant will be the
> unguessable/signed (changed/expired) decrypt key URL in the user data.
>
> HTH?

PS I should have added that ideally the decrypted ssh key is stored in
some ramdisk area so that it does not survive an instance reboot.

>
> Regards
> Mark
>
> [0] 
> http://shlomoswidler.com/2009/08/how-to-keep-your-aws-credentials-on-ec2.html
> [1] 
> http://shlomoswidler.com/2010/07/storing-aws-credentials-on-an-ebs-snapshot-securely.html
>
> On Tue, Jan 15, 2013 at 12:41 AM, Mike 
> <
 >
>  wrote:
>> Late to the game.
>> Have you explored using the instance launch user-data to write out your 
>> certs?
>>
>> i.e.
>>
>> #!/bin/bash -e
>> exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 
>> 2>&1
>> echo BEGIN
>> date '+%Y-%m-%d %H:%M:%S'
>>
>> mkdir -p /etc/chef
>>
>> cat <<- _EOF_
>> <YOUR CLIENT.RB CONFIG>
>>         _EOF_
>>> /etc/chef/client.rb
>>
>> cat <<- _EOF_
>> <YOUR VALIDATION PEM FILE>
>>         _EOF_
>>> /etc/chef/validation.pem
>>
>> cat <<- _EOF_
>> <YOUR SSH KEY HERE>
>>         _EOF_
>>  > /path/to/private/ssh/key
>>
>> (if your AMI doesn't have chef installed, install it via omnibus:)
>> curl -L https://www.opscode.com/chef/install.sh ;| bash
>>
>> /etc/init.d/chef-client start
>>
>> echo END
>>
>>
>> This is effectively the `knife bootstrap` process, executed by your
>> EC2 AutoScaler. This works with most "vetted" AMI images that do the
>> fun 'cloud-init' and all that jazz.
>>
>>
>> On Mon, Jan 14, 2013 at 5:02 AM, Cassiano Leal 
>> <
 >
>>  wrote:
>>> Since I want to auto-scale these nodes, I'll have to keep the 
>>> validation.pem
>>> in the AMI that will be used to boot up. That's one of the solutions that 
>>> I
>>> though -- store the SSH key in the AMI.
>>>
>>> I understand the security implications of copying SSH keys around or 
>>> leaving
>>> them in an image on the Amazon cloud, but I still fail to see a better
>>> solution. The fact that no alternative covering my requirements have
>>> appeared in this thread makes me all the more confident that I'm not doing
>>> something too crazy.
>>>
>>> Recapitulating requirements:
>>>
>>> - Auto-scaling -- this is the key point
>>> - New nodes automatically register themselves on the chef server and
>>> converge
>>> - Newly auto-scaled nodes are able to clone/update from git repos:
>>>   * on the first chef run
>>>   * no manual copy of SSH keys to the node
>>>   * no manual authorisation of a new key on bitbucket
>>>
>>> I guess the better question would be: how are people auto-scaling chef
>>> nodes?
>>>
>>> On 11 January 2013 22:38, Dan Razzell 
>>> <
 >
>>>  wrote:
>>>
>>> +1
>>>
>>> As Kevin pointed out yesterday, there's something essentially upside-down
>>> about your PKI if you're copying private keys around.
>>>
>>>
>>> On 13-01-11 04:31 PM, Kevin Keane (subscriptions) wrote:
>>>
>>> I see two options for that:
>>>
>>>
>>>
>>> - Upload the SSH key at the same time you bootstrap the new instance. You
>>> could even add that to a custom bootstrap script, to copy it along with 
>>> the
>>> validation.pem
>>>
>>> - Regularly, say, every 30 minutes, run a knife search and upload the SSH
>>> key to each server it finds.
>>>
>>>
>>> -----Original message-----
>>> From: Cassiano Leal 
>>> <
 >
>>> Sent: Fri 01-11-2013 04:18 am
>>> Subject: [chef] Re: Re: RE: Distribute private ssh keys via users cookbook
>>> To: 
>>> 
 ;
>>> All perfectly good points. How do I deal with auto-scaling, though? I 
>>> don't
>>> really want to be woken up in the middle of the night just because site
>>> traffic has peaked and more servers are spinning up so that I can SCP a
>>> private key to each new server. :)
>>>
>>> - cassiano
>>>
>>>
>>> On Friday, January 11, 2013 at 02:18, Kevin Keane (subscriptions) wrote:
>>>
>>> I see where you are coming from! In this type of scenario, I would 
>>> probably
>>> step outside the Chef setup.
>>>
>>>
>>>
>>> The security problem in Chef ultimately arises because you allow the 
>>> clients
>>> to *pull* the private keys. As a result, a hacker can potentially simply 
>>> use
>>> HTTP to download your databag, or even sniff the data going over the wire.
>>> Databag encryption, client.pem and validation.pem are all protections, but
>>> they can't compete (and aren't designed to compete) with established
>>> security mechanisms such as SSH, PGP, ...
>>>
>>>
>>>
>>> So maybe Chef isn't the best tool for this particular job?
>>>
>>>
>>>
>>> I would instead use SCP to copy your private key from a single secure 
>>> server
>>> to wherever they need to go. The big advantage is that because it is a
>>> push-based system, you have full control over which machines do the SSH 
>>> keys
>>> get copied to.
>>>
>>>
>>>
>>> -----Original message-----
>>> From: Cassiano Leal 
>>> <
 >
>>> Sent: Thu 01-10-2013 04:23 am
>>> Subject: [chef] Re: RE: Distribute private ssh keys via users cookbook
>>> To: 
>>> 
 ;
>>> Hi Kevin,
>>>
>>> Thanks for sharing your thoughts. Let me describe my scenario and what I'm
>>> trying to accomplish.
>>>
>>> I have VMs that spin up automatically on AWS. These VMs host several
>>> applications, most written in PHP and RoR. The source code for all the 
>>> apps
>>> resides on bitbucket Git repos.
>>>
>>> I'm setting up application deployment via the deploy resource [0]. For 
>>> that,
>>> I have created a "deploy" user through the users cookbook, but I need this
>>> user to have read-only access to the repos. For that, I have a bitbucket
>>> user with read-only access and I need to configure this user with the 
>>> public
>>> keys, and that's done on bitbucket website.
>>>
>>> If I create a new key pair on each VM, I'll have to manually authorise the
>>> new key on the bitbucket website so that the application can be deployed.
>>> That would completely defeat auto-scaling and would make my life a bit
>>> worse, specially on peak times. :)
>>>
>>> I understand the security implications of this approach, but I still find 
>>> it
>>> all that much better than having to manage potentially infinite key pairs 
>>> on
>>> a website. This key would have read only access to our repos, and if I 
>>> find
>>> out that it has been compromised all I need to do is revoke it on 
>>> bitbucket,
>>> create a new key pair and redistribute it.
>>>
>>> I can do that on a recipe manually, but it would be just a bit easier and
>>> more concise if it was possible via the users cookbook.
>>>
>>> Also, it beats me why the cookbook allows me to distribute private keys 
>>> via
>>> an unencrypted data bag [1], and has no secure-ish solution for that.
>>>
>>> Again, I'm open for hearing other solutions to this problem. If they're 
>>> more
>>> secure, all the better, as long as it doesn't make me go to bitbucket.org
>>> and insert a new pubkey each time I spin up a new server. :)
>>>
>>> [0] http://docs.opscode.com/resource_deploy.html
>>> [1]
>>> https://github.com/opscode-cookbooks/users/blob/master/providers/manage.rb#L112-L122
>>>
>>> - cassiano
>>>
>>>
>>> On Thursday, January 10, 2013 at 09:51, Kevin Keane (subscriptions) wrote:
>>>
>>> It seems to me that there might be a problem with what you are trying to
>>> accomplish in the first place. When you distribute private SSH keys 
>>> through
>>> *any* means, you have a security problem. Doesn't matter if you are using
>>> encrypted databags, or even manually copy the keys over. Unless they are
>>> your own personal keys, it defeats the very concept behind public key
>>> encryption - and if they *are* your personal keys, you should only 
>>> manually
>>> copy them, not put them in an automated distribution system.
>>>
>>>
>>>
>>> Once generated, private SSH keys should never leave that user's control. 
>>> You
>>> can use chef to distribute the corresponding public keys - and you don't
>>> even need an encrypted databag. Public keys are not sensitive; you can 
>>> leave
>>> them unencrypted.
>>>
>>>
>>>
>>> If your users are humans, have them generated their SSH key pairs, and
>>> create a databag with all the public keys that you want to distribute. 
>>> Hint:
>>> the users cookbook already has a mechanism built in to do that.
>>>
>>>
>>>
>>> If you need SSH key pairs for some automated tasks (say, a cron script 
>>> doing
>>> a nightly rsync), generate a *new* key pair, and store only the public key
>>> in a node attribute. You can use an execute or script resource to generate
>>> the key pair; just make sure to check if the key may already exist (you
>>> don't want to clobber an already-existing key pair).
>>>
>>>
>>> On the SSH client side, you'd extract all the relevant public keys from 
>>> your
>>> databag or node attributes, and add them to the authorized_keys file as
>>> needed.
>>>
>>>
>>>
>>> -----Original message-----
>>> From: Cassiano Leal 
>>> <
 >
>>> Sent: Wed 01-09-2013 11:40 am
>>> Subject: [chef] Distribute private ssh keys via users cookbook
>>> To: 
>>> 
 ;
>>> Hi!
>>>
>>> Is there a way to securely distribute private ssh keys through the users
>>> community cookbook?
>>>
>>> In my setup I have a user deploy that will fetch from git, and I need that
>>> user to have a SSH key that's authorized on the git repo. I saw that the
>>> users cookbook will use "ssh_private_key" and "ssh_public_key" data bag
>>> items, but those would be unencrypted, so not secure.
>>>
>>> If people are using a different approach, I'd like to hear about that too.
>>>
>>> Thanks!
>>> --
>>> Cassiano Leal
>>>
>>>
>>>
>>>
>>>
>>>
>>>