[chef] Re: Re: Re: Using Chef to Configue Autoscaled EC2 Instances

[Bryan Brandau < >]

We manage several environments that use lots of auto-scaling.  We've iterated on this a little.

At first, we were using user-data and this just ends up being bad.  There is no way of changing this unless you have the ability to stop the instance.  To me, this is unacceptable.  We ended up getting trapped when we wanted to change some of the user-data.

We then ran JEOS with chef-client and put the role in client-config.json.  When chef would run it would configure the server for the role that it is supposed to be.  This can be easily changed or managed with chef.  

The problem with all of this bootstrapping and configuration on the fly is that when you want to auto-scale you're triggering off of an event.  You want the machine to become available as soon  as it can boot.  This lead us to baking/bundling AMI's with the full chef run.  We still assign the role with client-config.json.  Chef was used to build/configure and will then maintain state.  If it's a minor change, we don't care about we won't bundle/bake a new AMI.  If we care about it, we bake/bundle.

We'll never be impacted from scaling by waiting for a machine to configure or if the chef server is unavailable.

With user-data or managing the role on the machine you can always just change what roles/recipes are assigned to the role.

On Thu, Nov 10, 2011 at 12:08 PM, Aaron Abramson < "> > wrote:

I've thought about this, but haven't put much work into implementing it yet.  So this is more of a concept than anything.

Set up a script listening on a server.  It could be a simple php script, IE http://someurl/script.php

Set up the user-data with a bash script that hits the listener.  (wget -O /dev/null http://someurl/script.php?role=myrole)

The script receives the request, and it knows 2 things, the public hostname of the server making the request, and the "role" it is requesting.  The script makes certain assumptions (username, ssh-key, etc) and then executes a system command, calling "knife bootstrap *server* -r "role[myrole]"

So, if you auto-scale official cannonical ubuntu AMI's, simply have the user-data "ping" your knife listener, and knife then bootstraps the server.

Perhaps such a bootstrap request feature could be built into chef-server?


Aaron

On Nov 10, 2011, at 11:55 AM, John Martinez wrote:

>
> Hello,
>
> I agree with you, using knife to provision EC2 instances is not the best way to do this in autoscaled environments. As a temporary solution, I've done exactly what you suggest. I pass along a lot of things into user-data, and specifically, a node's role is one of those. I read user-data and tell Chef on the node what its role is (amongst other things). It's a kludge. I'm going to be looking into integrating Chef with CloudFormation, which plays nice(r) with ASGs.
>
> http://aws.amazon.com/cloudformation/aws-cloudformation-articles-and-tutorials/ (second link)
> https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithOpscodeChef.pdf
>
> -john
>
> On Nov 10, 2011, at 9:44 AM, < "> > < "> > wrote:
>
>> Hi all, I'd like to get some feedback on how people are doing this now. It
>> seems that since .10, the preferred way of starting Amazon EC2 instances is via
>> the 'knife ec2 server create' command, while the 'knife ec2 instance data' and
>> chef preinstalled AMIs are deprecated. That's all fine and good for manually
>> launching instances, but what about autoscaling? When a node comes up for
>> autoscalling, I want it to update and configure itself according to its role(s)
>> and environment. With the instance data method of doing thing deprecated,
>> what's the alternative (aside from having a custom baked AMI that I need to
>> manually update as is the normal way of doing this without chef)? If there is
>> no alternative, and I have to use the deprecated method which would mean using
>> 'knife ec2 instance data' to wire into the autoscaling group as UserData. When
>> using this method how would I specify the environment that the server should be
>> run as too? I don't see anyway to do this using the instance data. Running some
>> command outside the server after the server it is started to set the
>> environment is not an option as we won't know when a new server has been spun
>> up (controlled by Amazon).
>>
>> I'm interested in what other people have found to be best practice for this
>> scenario and any advice, tips or tricks that have been learned while trying to
>> work with chef and autoscaled EC2 instances.
>>
>> Thanks for your thoughts!
>