General discussion about Chef

[chef] Re: AWS Security Groups


Chronological Thread 
  • From: Eric Herot < >
  • To: , Douglas Garstang < >
  • Subject: [chef] Re: AWS Security Groups
  • Date: Mon, 17 Nov 2014 15:44:31 -0500
Chef Vault does address one of the earlier concerns you raised about the scalability of mass-distributing keys to use for decrypting data bags.  Instead it uses the client keys on the individual nodes to encrypt separate copies of your data bag item (one for each host that’s allowed to access it).  That way you don’t have to worry (much) about key management, and access to the data is not universal (like it is with normal encrypted data bags, which were invented mainly as a way to not store sensitive data in source control).

-- 
Eric

On November 17, 2014 at 3:33:53 PM, Eric Herot ( "> ) wrote:

Doug,

chef-vault doesn’t quite have native support in the aws_security cookbook, but as that cookbook doesn’t do much with its default recipe, it would be rather trivial it make the two things work together.

All you would need is to replace the original include_recipe "aws_security::default" clause with a short recipe doing almost the same thing (but with Chef Vault code instead of EncryptedDataBagSecret. E.g.:

Copy the existing recipe and replace this:

databag_item = Chef::EncryptedDataBagItem.load( node['aws_security']['encrypted_data_bag'], 'aws_keys' )

With this:

databag_item = ChefVault::Item.load(
node['aws_security']['encrypted_data_bag'],
'aws_keys'
)

-- 
Eric

On November 17, 2014 at 3:19:23 PM, Douglas Garstang ( "> ) wrote:

Does it work seamlessly with the aws_security cookbook? If no, then there's not much point.
On Mon, Nov 17, 2014 at 12:11 PM, Fabien Delpierre < " target="_blank"> > wrote:
I don't know if it would be usable in your case, but have you looked at chef-vault for managing your secrets?
https://github.com/Nordstrom/chef-vault

On Mon, Nov 17, 2014 at 3:05 PM, Douglas Garstang < " target="_blank"> > wrote:
Seriously?

That doesn't scale. Time to stop shaving the yak here I think.

Sigh.

I'm trying to automate the creation of security groups. Chef-metal/provisioner doesn't do it. Cloud Formation has so many limitations imposed, and I just tried terraform, and it's failed to impress fast because I can't have programmatic resource names. I would imagine that most people, like us, would have multiple hosts with the same name, one per environment. Ie, a web01 in prod and a web01 in dev for example. It seems like the only way to do this with terraform would be to have one config file per environment, with everything except the resource names duplicated, and again, that does not scale. :(

Doug.


On Mon, Nov 17, 2014 at 11:41 AM, Fabien Delpierre < " target="_blank"> > wrote:
To your question on how a cookbook decrypts an encrypted data bag item during the Chef run, it simply needs to have the secret. You can pass the secret as an argument when bootstrapping the node, e.g.:
$ knife bootstrap my.node.local -N my.node.local -E myenvironment -r 'recipe[aws_security]' -x root --secret foo
The bootstrapping process will then set up the node so it can access the encrypted data bag seamlessly (which means keeping a local file containing the secret in plain text).
You can also add it manually by editing /etc/chef/client.rb on the node.

On Mon, Nov 17, 2014 at 1:21 PM, Douglas Garstang < " target="_blank"> > wrote:
Apparently I'm still doing something wrong.

Using https://github.com/SearchSpring/aws_security.

Getting:
"Data Bag Items must contain a Hash or Mash!"

at:
==> cfgserver:  26:  if node['aws_security']['encrypted_data_bag']
==> cfgserver:  27>>   databag_item = Chef::EncryptedDataBagItem.load(
==> cfgserver:  28:      node['aws_security']['encrypted_data_bag'],
==> cfgserver:  29:        'aws_keys'

I've set the name of the encrypted data bag in the attribute node['aws_security']['encrypted_data_bag']. That's the name of the data bag, but how do I set the name of the data bag item? What's it looking for?

Also, how does the cookbook decrypt the data bag on the client side?

Doug.


On Mon, Nov 17, 2014 at 10:06 AM, Douglas Garstang < " target="_blank"> > wrote:
Thanks Eric and Fabien.

I hadn't realised that i need to download the encrypted version from the chef server in order to overwrite the unencrypted local version.

Doug.

On Mon, Nov 17, 2014 at 10:00 AM, Eric Herot < " target="_blank"> > wrote:

cc’ing Jeff ‘cause he’s my cow-orker…

So our workflow for this is a little more complicated than I’d like, but it basically works like this:

You run this command to create the data bag on the Chef server:

knife data bag create \
  —-secret-file ~/.chef/encrypted_data_bag_secret \
  data_bag_name \
  data_bag_item_name

This will open your editor and let you start creating your data bag. When you’re finished, you save and quit. This will encrypt the contents and upload it to your Chef server.

Next, you will want to save the encrypted version of that data bag to your source control system. For that you will need to output what’s on the Chef server in its encrypted form:

knife data bag show \
  data_bag_name \
  data_bag_item_name \
  -f json \
  > ~/path/to/data_bags/data_bag_name/data_bag_item_name.json

The newly created file will look something like this:

{
  "id": "data_bag_item_name",
  "some_top_level_key": {
    "encrypted_data": "e5WAWnV6aTlXTz4ZpSPXqZ2Y2gEFLBrxC6Jroic+qvgk5fB9Ad587NZJf1xc\nz94vqdzOwdWHYPxfbNg24G9hfljgi63ustAKrRFFH5B2xVI=\n",
    "iv": "MBxrn6waf9ZOwZYZOePJMg==\n",
    "version": 1,
    "cipher": "aes-256-cbc"
  }
}

Note that the top level keys are in plain text but everything else is encrypted.

You can then check that file into git:

git add data_bag_item_name.json && git commit data_bag_item_name.json

Does that make sense?
-- 
Eric

On November 17, 2014 at 12:40:21 PM, Douglas Garstang ( " target="_blank"> ) wrote:

This ain't half confusing is it.

The local file is not encrypted. It seems that knife does not encrypt it. Are you saying that I need to use a mechanism besides the knife command to independently encrypt the file?

Doug.

On Mon, Nov 17, 2014 at 9:38 AM, Eric Herot < " target="_blank"> > wrote:
In most setups, where the local file is what’s being pushed to source control, the data in that file *should* be encrypted. If you mean that you can easily look at the “raw” contents by looking at the local file, however, that would be correct.
-- 
Eric

On November 17, 2014 at 12:35:44 PM, Douglas Garstang ( " target="_blank"> ) wrote:

Well, I can also see the raw content by looking at my local file, since that's not encrypted either...
On Mon, Nov 17, 2014 at 9:28 AM, Jeff Byrnes < " target="_blank"> > wrote:
That’s correct. Knife will use your encryption key to de/encrypt the data in an encrypted data bag.

If you’d like to see the raw content of a data bag item, you can run this command:

knife exec -E "puts JSON.pretty_generate(api.get('/data/data_bag/data_bag_item'))"

That will output the raw JSON, pretty-printed, from your Chef Server.

To add to the confusion, if you have the `knife[:secret_file]` setting in your .knife.rb set, using the `knife data bag show` command will always display a decrypted data bag item, which can be _very_ confusing.
-- 
Jeff Byrnes
@berkleebassist
Lead DevOps Engineer

On November 17, 2014 at 12:24:27 PM, Douglas Garstang ( " target="_blank"> ) wrote:

I'm not following. The file is unencrypted locally. Are you saying that the knife command will encrypt the local file?
On Mon, Nov 17, 2014 at 5:41 AM, Fabien Delpierre < " target="_blank"> > wrote:
Yes. You can see that for yourself, just upload your JSON file with the secret, then use the knife download command to download it back from the server, then compare your original to the one you just downloaded, you'll see how the encryption works. And then you can add the encrypted version to Git, it is safe (or at least, it's as safe as your encryption key is).

On Mon, Nov 17, 2014 at 7:53 AM, Jeff Byrnes < " target="_blank"> > wrote:
Doug,

While the file itself won’t be encrypted, it’s contents will be, so it IS safe to version the file in Git (we do in a repo that covers our data bags, environments, & roles).
-- 
Jeff Byrnes
Operations Engineer

On November 17, 2014 at 12:36:03 AM, Douglas Garstang ( " target="_blank"> ) wrote:

Oh. Thanks for that. And, that's... terrible. :( The point is not to have the unencrypted file in git.

Doug.
On Sun, Nov 16, 2014 at 6:48 PM, Fabien Delpierre < " target="_blank"> > wrote:
RE: data bags being encrypted on the Chef server and not on your local system, that's correct.
I should say that data bags themselves are not encrypted. There's no difference between a data bag and an encrypted data bag. It's the items within the data bag that are (or aren't) encrypted. Even though they're called "encrypted data bag".
If you find it a problem that what's in your local file system is not encrypted, then what you can do is upload your JSON file with the secret (as seen in your other thread I just replied to), then delete the file from your local file system, and if you want the encrypted file locally, then just download it from the Chef server, it'll be encrypted then.



On Sun, Nov 16, 2014 at 9:11 PM, Douglas Garstang < " target="_blank"> > wrote:
Is there a known public cookbook for creating AWS security groups? I don't know about the security implications, but I'd like to try having the recipes creating the necessary security groups. Otherwise, it's a major hassle to put them into a script. AFAIK chef-metal/provisioner doesn't do security groups yet.

This one https://github.com/SearchSpring/aws_security, wants to use encrypted data bags and, well, that's a pain. I'd prefer to use IAM roles were possible. I'd never noticed before, but it looks like encrypted data bags are only encrypted on the Chef server, not on the local file system? If so, what's the point?

Doug





--




--



--



--




--




--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: ">
Cell: +1-805-340-5627

Archive powered by MHonArc 2.6.16.

§