[chef] Re: Re: Chef on Amazon EC2 with auto-scaling

["John E. Vincent (lusis)" < >]

On Mon, Jul 18, 2011 at 6:31 AM, Oliver Beattie 
<
 >
 wrote:
> Hi John,
> Thanks, that's very helpful, and solidifies some of the things I'd read but
> couldn't quite see how they fit together.
> So I guess I'm left with a couple other questions:
> * As I originally mentioned, what is the procedure for managing these
> servers? Would I just be able to run commands via knife to all my servers?
> How does it keep track of nodes joining (or more importantly leaving) my
> "cluster"?

The servers can still be managed just fine via knife in the future.
They register with the chef-server like any node boostrapped with
knife.

As for leaving the cluster, that's entirely on you to decide how to
handle it. I'm not sure, off-hand, if the autoscaling issues a
shutdown command or has hooks for calling custom logic.
My thought would be to add something to the shutdown that does a
'knife node delete nodename -y' and 'knife client delete nodename -y'.
However you'll have to have a different set of credentials on the
server somewhere that has permissions to do that. Another option is to
have your servers send some sort of message on shutdown to something
that does the cleanup for you. I'm considering a Noah callback that
does some of the ec2 api interaction but I've just not gotten around
to it. Another option is to use RunDeck or even Jenkins and have the
hook call those guys when it shuts down. It only needs to be a small
message (something like {"node":"nodename","action":"shutdown"}) to a
small service somewhere that can then do the cleanup on the backend
(including removing it from monitoring if need be).

> * Another (somewhat unrelated question) I had is how does Chef manage OS
> upgrades? Does it manage them at all? For instance, how would I say "go run
> aptitude upgrade on all my production servers"?

That is also up to you. If you leave chef-client running in daemon
mode, then if you have a cookbook/recipe that does that (upgrades all
packages), it will run. Personally, I have a cookbook that upgrades a
specific set of packages if they're available on each run. Since
you're running in EC2 and being elastic, it's probably not worth the
effort. The instances won't be long lived enough (generally speaking)
to matter. If you're concerned about package drift from any long-lived
instances and autoscaling ones, you could always have your bootstrap
update to the latest packages but that prolongs the time it takes the
instance to come up.

I'm still debating how much logic to bake into my AMIs myself. I'm a
big fan of "just enough operating system" but over time my bootstraps
start to take a bit longer catching up the AMI to where it needs to
be. I personally treat this as an impetus to not install so much crap
on my servers.

> —Oliver
> On 18 July 2011 11:17, John E. Vincent (lusis)
> <
 >
>  wrote:
>>
>> On Mon, Jul 18, 2011 at 5:53 AM, Oliver Beattie 
>> <
 >
>> wrote:
>> > Hi there,
>> > I'm very interested in using Chef for deployment of our web services on
>> > Amazon EC2, and was looking if someone could help me out with some
>> > advice.
>> > I've been trying my best to read through the existing information that's
>> > out
>> > there, but I'm just coming up very confused. Basically, the desired
>> > setup I
>> > want to achieve is as so:-
>> > * Use Amazon auto-scaling to automatically bring up new instances as the
>> > load requires
>> > * The servers automatically configure themselves with services via Chef
>> > Is this something that's actually possible with using Chef? From what I
>> > see,
>> > it is possible, but I don't really see any information on the real
>> > mechanics
>> > of this, especially with regard to how one manages these servers once
>> > they
>> > have been brought up. Are they manageable using Knife (even though they
>> > weren't launched using it?)
>> > If anyone could point me in the direction of some more information, I'd
>> > be
>> > most grateful.
>> > Thanks,
>> > Oliver Beattie
>>
>> How you do this is pretty flexible and depends on how much logic you
>> want to bake into your AMIs.
>>
>> I've only glanced askew at the Amazon autoscaling but the gist is this:
>>
>> - You bake in ruby and chef-client
>> - You don't have to bake in the validation.pem but you could (This
>> would mean respinning your AMIs if you ever had to revoke it. Also
>> consider how that impacts things if running multiple chef-servers)
>> - Using userdata, you would essentially create a "first-boot.json" and
>> run chef-client against it.
>>
>> The node will autoregister itself. Remember that userdata has a fixed
>> size (that I can't recall off hand). If there's anything more complex,
>> you'd probably want to pull everything from a private S3 bucket.
>>
>> To get an idea of how this would work, look at the bootstrap templates
>> that knife uses. They're just ERB templates that knife executes via
>> ssh. knife-ec2 just adds in the ec2 api calls to spin up the instance.
>>
>> You can see (if you've never looked) what the bootstrap templates look
>> like here: https://gist.github.com/2c049eb9df73ed0e01ea
>>
>> The biggest thing is how do you want to get your validation.pem onto
>> the server? S3 private bucket or bake it into the AMI. You'll need
>> some "custom" logic for parsing the run_list out of userdata as well.
>> All first-boot.json needs is this:
>>
>> {"run_list":["role[tracker]"]}
>
>