[chef] Re: Re: Re: Managing an ec2 instance and security group together with chef?

[Aaron Abramson < >]

> Thanks for the input, I think that clarifies that this is possible.
> One thing concerns me, though, and that is that this style of using
> EC2 does require ec2 api keys be installed on every host, and that
> means that if any host is compromised then the entire cloud can be
> removed.  If this could be managed centrally then that reduces the
> proliferation of keys.  Is there a method to proxy this sort of
> meta-server management on either a second host, or on the chef server?

I forgot to address this part...

Write your controller cookbook to run the ec2 commandline tools giving the 
api-credentials at the commandline, and not through an environment variable.  
Then store your api-keys in a data bag, and use them in the cookbook when 
running your script resource that executes the ec2 tools.

Use amazon's IAM to create a user that ONLY has access to what you need, and 
store THAT api key/secret id in the databag.




On Sep 16, 2011, at 3:22 PM, Peter Norton wrote:

> Questions in-line
> 
> On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson 
> <
 >
>  wrote:
>> I would do it this way:
>> 
>> 1) Launch EC2 servers with 3 security groups, default, chef, and unique 
>> (unique being unique for that server)
>> - Remember, your server has to allow ssh access from where you're working, 
>> and I prefer to keep the 'default' SG as amazon internal only, thus the 
>> 'chef' SG.
>> 
> 
> Good point.  I was using default as a shorthand for "the things
> everything needs", but it should be "default,common,sg-0xDEADBEEF"
> where "default" should be everything off, and "common" should include
> anything that all servers need (ssh being the most likely, or ssh on
> an alternate port for example).
> 
>> 2) Make sure server has ec2 tools installed, and can update security group 
>> settings.
>> 
> 
> Ec2 tools? Would knife (with modules to handle security group
> manipulations) be sufficient?  I'm pretty sure the AMIs we use include
> the tools, but they're slooooow.
> 
>> 3) Update all cookbooks which install/configure services to open necessary 
>> ports in the 'unique' security group.
>> 
> 
> So once we decide on our group manipulation tool, use that tool in the
> recipes?  OK.
> 
>> Now when you install your http server of choice, part of the server setup 
>> is to open up the ports used in the security group.   You can store the 
>> 'unique' security group name as a node attribute.
>> 
>> I would suggest creating an EC2-controls cookbook, and make "sg-add" and 
>> "sg-del" LWRP's for use by your other cookbooks.  That way you simply need 
>> to tell the apache cookbook:
>> 
>> sg-add "apache ports" do
>>        group node[:ec2][:unique_group]
>>        tcp_ports [ "80", "443" ]
>>        udp_ports nil
>> end
>> 
>> This keeps it idempotent, only the packages/services which are installed 
>> get added to your security group. And you don't have to dink around with 
>> SG templates or anything else.  Let the security group be built/configured 
>> 'on-the-fly'.
>> 
> 
> Thanks for the input, I think that clarifies that this is possible.
> One thing concerns me, though, and that is that this style of using
> EC2 does require ec2 api keys be installed on every host, and that
> means that if any host is compromised then the entire cloud can be
> removed.  If this could be managed centrally then that reduces the
> proliferation of keys.  Is there a method to proxy this sort of
> meta-server management on either a second host, or on the chef server?
> 
> -Peter
> 
>> On Sep 16, 2011, at 2:22 PM, 
>> <
 >
>>  wrote:
>> 
>>> Hello,
>>> 
>>> I'm new to chef, but I'm trying it out as a way to manage amazon ec2
>>> instances, and I've found some places that I think chef can help manage,
>>> but which it can't completely manage yet.
>>> 
>>> My first challenge comes from the fact that it seems like security groups
>>> are inflexible, and can be limiting.
>>> 
>>> Specifically a use case that would be nice is to be able to create a new
>>> system, and have it attached to a security group after it's been 
>>> allocated to
>>> a service.  Since services require ports to be opened, this would mean
>>> changing security groups. However, since ec2 doesn't allow for security
>>> groups to change after a reservation is created, one way to get this
>>> behavior would be to have a security group created per-host.
>>> 
>>> I think I see how to do this with knife, so I think I can add the 
>>> necessary
>>> knife ec2 command to create a group, and maybe tie the same function
>>> into creating an instance to make my use case easier (1 command instead
>>> of 2).
>>> 
>>> However, the second part that makes this useful is to have the host's
>>> security group be patterned off of an existing group, or off a 
>>> configuration
>>> that chef maintains.  Either way, the important point is that changes to
>>> that template reflected in all groups that are patterned off of it.
>>> 
>>> One issue with this is that I don't see clearly how chef could be used to
>>> manage a security group in the way I envision it should work.  Here's 
>>> what I
>>> imagine:
>>> 
>>> 1) Knife creates a new security group using a pseudo-random name - let's 
>>> say
>>> 'sg-0xDEADBEEF' for this example.
>>> 2) The instance's reservation is created with two security groups:
>>> default,sg-0xDEADBEEF
>>> 3) The instance is created, and chef is bootstrapped
>>> 4) The host is created with an role called "available".  The "available"
>>> role includes the default security group's settings, so these are cloned
>>> into sg-0xDEADBEEF's settings and now it looks the same as default.
>>> 5) The server is needed an hour later, so it's prepared for a deployment 
>>> by
>>> being assigned the roles "http,https,tomcat-2121" which each include the
>>> security groups needed as well as the packages.  The installation of 
>>> apache,
>>> along with laying out conf files for port 80, port 443, and port 2121, the
>>> tomcat instance, java, etc. get pumped out, but each role also depends on 
>>> an
>>> associated security group, let's say sg-http, sg-https, and 
>>> sg-tomcat-2121,
>>> which allow port 80, port 443, and port 2121 respectively.
>>> 
>>> If later on the tomcat-2121 also needs port 2122 for some reason then
>>> port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
>>> that port added as part of the regular chef operation.
>>> 
>>> So... is it possible to do this with chef?  If someone can point me to any
>>> examples of similar situations (e.g. a configuration that is related to a 
>>> host,
>>> managed along with that host, but is not part of the host) that I
>>> could use as a
>>> starting point, I'd be quite willing to point my zero knowledge of ruby 
>>> at this
>>> problem, and use fog to make this happen.
>>> 
>>> Thanks,
>>> 
>>> -Peter
>> 
>>