[chef] Re: Re: Re: Managing an ec2 instance and security group together with chef?

[Aaron Abramson < >]

> Ec2 tools? Would knife (with modules to handle security group
> manipulations) be sufficient?  I'm pretty sure the AMIs we use include
> the tools, but they're slooooow.

You would need the commandline tools to be installed on the instance itself.  
Knife is being run from your local machine, so your knife ec2 commands are 
run locally, and can't be called when chef-client is run on the remote 
instance.

When chef-client runs on the particular instance, the recipes need to execute 
the ec2 command line tools there.  Unless you're installing knife, and all of 
the requisite tools and key files on each instance, having a cookbook run a 
knife command is going to be inefficient.

If you build your "contoller" cookbook around the standard ec2 api java tools 
(or a particular set of tools like the python based ones), then simply have 
that cookbook set up the tools, set your api credentials, and be a dependency 
of every other cookbook, or part of your base role.




On Sep 16, 2011, at 3:22 PM, Peter Norton wrote:

> Questions in-line
> 
> On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson 
> <
 >
>  wrote:
>> I would do it this way:
>> 
>> 1) Launch EC2 servers with 3 security groups, default, chef, and unique 
>> (unique being unique for that server)
>> - Remember, your server has to allow ssh access from where you're working, 
>> and I prefer to keep the 'default' SG as amazon internal only, thus the 
>> 'chef' SG.
>> 
> 
> Good point.  I was using default as a shorthand for "the things
> everything needs", but it should be "default,common,sg-0xDEADBEEF"
> where "default" should be everything off, and "common" should include
> anything that all servers need (ssh being the most likely, or ssh on
> an alternate port for example).
> 
>> 2) Make sure server has ec2 tools installed, and can update security group 
>> settings.
>> 
> 
> Ec2 tools? Would knife (with modules to handle security group
> manipulations) be sufficient?  I'm pretty sure the AMIs we use include
> the tools, but they're slooooow.
> 
>> 3) Update all cookbooks which install/configure services to open necessary 
>> ports in the 'unique' security group.
>> 
> 
> So once we decide on our group manipulation tool, use that tool in the
> recipes?  OK.
> 
>> Now when you install your http server of choice, part of the server setup 
>> is to open up the ports used in the security group.   You can store the 
>> 'unique' security group name as a node attribute.
>> 
>> I would suggest creating an EC2-controls cookbook, and make "sg-add" and 
>> "sg-del" LWRP's for use by your other cookbooks.  That way you simply need 
>> to tell the apache cookbook:
>> 
>> sg-add "apache ports" do
>>        group node[:ec2][:unique_group]
>>        tcp_ports [ "80", "443" ]
>>        udp_ports nil
>> end
>> 
>> This keeps it idempotent, only the packages/services which are installed 
>> get added to your security group. And you don't have to dink around with 
>> SG templates or anything else.  Let the security group be built/configured 
>> 'on-the-fly'.
>> 
> 
> Thanks for the input, I think that clarifies that this is possible.
> One thing concerns me, though, and that is that this style of using
> EC2 does require ec2 api keys be installed on every host, and that
> means that if any host is compromised then the entire cloud can be
> removed.  If this could be managed centrally then that reduces the
> proliferation of keys.  Is there a method to proxy this sort of
> meta-server management on either a second host, or on the chef server?
> 
> -Peter
> 
>> On Sep 16, 2011, at 2:22 PM, 
>> <
 >
>>  wrote:
>> 
>>> Hello,
>>> 
>>> I'm new to chef, but I'm trying it out as a way to manage amazon ec2
>>> instances, and I've found some places that I think chef can help manage,
>>> but which it can't completely manage yet.
>>> 
>>> My first challenge comes from the fact that it seems like security groups
>>> are inflexible, and can be limiting.
>>> 
>>> Specifically a use case that would be nice is to be able to create a new
>>> system, and have it attached to a security group after it's been 
>>> allocated to
>>> a service.  Since services require ports to be opened, this would mean
>>> changing security groups. However, since ec2 doesn't allow for security
>>> groups to change after a reservation is created, one way to get this
>>> behavior would be to have a security group created per-host.
>>> 
>>> I think I see how to do this with knife, so I think I can add the 
>>> necessary
>>> knife ec2 command to create a group, and maybe tie the same function
>>> into creating an instance to make my use case easier (1 command instead
>>> of 2).
>>> 
>>> However, the second part that makes this useful is to have the host's
>>> security group be patterned off of an existing group, or off a 
>>> configuration
>>> that chef maintains.  Either way, the important point is that changes to
>>> that template reflected in all groups that are patterned off of it.
>>> 
>>> One issue with this is that I don't see clearly how chef could be used to
>>> manage a security group in the way I envision it should work.  Here's 
>>> what I
>>> imagine:
>>> 
>>> 1) Knife creates a new security group using a pseudo-random name - let's 
>>> say
>>> 'sg-0xDEADBEEF' for this example.
>>> 2) The instance's reservation is created with two security groups:
>>> default,sg-0xDEADBEEF
>>> 3) The instance is created, and chef is bootstrapped
>>> 4) The host is created with an role called "available".  The "available"
>>> role includes the default security group's settings, so these are cloned
>>> into sg-0xDEADBEEF's settings and now it looks the same as default.
>>> 5) The server is needed an hour later, so it's prepared for a deployment 
>>> by
>>> being assigned the roles "http,https,tomcat-2121" which each include the
>>> security groups needed as well as the packages.  The installation of 
>>> apache,
>>> along with laying out conf files for port 80, port 443, and port 2121, the
>>> tomcat instance, java, etc. get pumped out, but each role also depends on 
>>> an
>>> associated security group, let's say sg-http, sg-https, and 
>>> sg-tomcat-2121,
>>> which allow port 80, port 443, and port 2121 respectively.
>>> 
>>> If later on the tomcat-2121 also needs port 2122 for some reason then
>>> port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
>>> that port added as part of the regular chef operation.
>>> 
>>> So... is it possible to do this with chef?  If someone can point me to any
>>> examples of similar situations (e.g. a configuration that is related to a 
>>> host,
>>> managed along with that host, but is not part of the host) that I
>>> could use as a
>>> starting point, I'd be quite willing to point my zero knowledge of ruby 
>>> at this
>>> problem, and use fog to make this happen.
>>> 
>>> Thanks,
>>> 
>>> -Peter
>> 
>>