[chef] Re: Re: Chef on Amazon EC2 with auto-scaling

[Oliver Beattie < >]

Hi John,

Thanks, that's very helpful, and solidifies some of the things I'd read but couldn't quite see how they fit together.

So I guess I'm left with a couple other questions:

* As I originally mentioned, what is the procedure for managing these servers? Would I just be able to run commands via knife to all my servers? How does it keep track of nodes joining (or more importantly leaving) my "cluster"?

* Another (somewhat unrelated question) I had is how does Chef manage OS upgrades? Does it manage them at all? For instance, how would I say "go run aptitude upgrade on all my production servers"?

—Oliver

On 18 July 2011 11:17, John E. Vincent (lusis) <lusis.org+ "> > wrote:

On Mon, Jul 18, 2011 at 5:53 AM, Oliver Beattie < "> > wrote:
> Hi there,
> I'm very interested in using Chef for deployment of our web services on
> Amazon EC2, and was looking if someone could help me out with some advice.
> I've been trying my best to read through the existing information that's out
> there, but I'm just coming up very confused. Basically, the desired setup I
> want to achieve is as so:-
> * Use Amazon auto-scaling to automatically bring up new instances as the
> load requires
> * The servers automatically configure themselves with services via Chef
> Is this something that's actually possible with using Chef? From what I see,
> it is possible, but I don't really see any information on the real mechanics
> of this, especially with regard to how one manages these servers once they
> have been brought up. Are they manageable using Knife (even though they
> weren't launched using it?)
> If anyone could point me in the direction of some more information, I'd be
> most grateful.
> Thanks,
> Oliver Beattie

How you do this is pretty flexible and depends on how much logic you
want to bake into your AMIs.

I've only glanced askew at the Amazon autoscaling but the gist is this:

- You bake in ruby and chef-client
- You don't have to bake in the validation.pem but you could (This
would mean respinning your AMIs if you ever had to revoke it. Also
consider how that impacts things if running multiple chef-servers)
- Using userdata, you would essentially create a "first-boot.json" and
run chef-client against it.

The node will autoregister itself. Remember that userdata has a fixed
size (that I can't recall off hand). If there's anything more complex,
you'd probably want to pull everything from a private S3 bucket.

To get an idea of how this would work, look at the bootstrap templates
that knife uses. They're just ERB templates that knife executes via
ssh. knife-ec2 just adds in the ec2 api calls to spin up the instance.

You can see (if you've never looked) what the bootstrap templates look
like here: https://gist.github.com/2c049eb9df73ed0e01ea

The biggest thing is how do you want to get your validation.pem onto
the server? S3 private bucket or bake it into the AMI. You'll need
some "custom" logic for parsing the run_list out of userdata as well.
All first-boot.json needs is this:

{"run_list":["role[tracker]"]}