[chef] Re: Re: Re: Re: Using Chef to Configue Autoscaled EC2 Instances

[Hedge Hog < >]

On Fri, Nov 11, 2011 at 5:27 AM, Bryan Brandau 
<
 >
 wrote:
> We manage several environments that use lots of auto-scaling.
>  We've iterated on this a little.
> At first, we were using user-data and this just ends up being bad.  There is
> no way of changing this unless you have the ability to stop the instance.
>  To me, this is unacceptable.  We ended up getting trapped when we wanted to
> change some of the user-data.
> We then ran JEOS with chef-client and put the role in client-config.json.
>  When chef would run it would configure the server for the role that it is
> supposed to be.  This can be easily changed or managed with chef.
> The problem with all of this bootstrapping and configuration on the fly is
> that when you want to auto-scale you're triggering off of an event.  You
> want the machine to become available as soon  as it can boot.  This lead us
> to baking/bundling AMI's with the full chef run.  We still assign the role
> with client-config.json.  Chef was used to build/configure and will then
> maintain state.  If it's a minor change, we don't care about we won't
> bundle/bake a new AMI.  If we care about it, we bake/bundle.
> We'll never be impacted from scaling by waiting for a machine to configure
> or if the chef server is unavailable.
> With user-data or managing the role on the machine you can always just
> change what roles/recipes are assigned to the role.
>

This arises if you use continuous deployment and AutoScaling
instances, i.e. Bryan's bake in your Chef run won't suffice on its own
(we do bake an AMI with the most time consuming installation steps
completed).
Essentially you need to move from pushing your code to machines to
having your instances pulling your code.
Apart from employing userdata, another way is to:
 - place the Chef role and environment names in the security group
name of an instance.
 - userdata is a simple script that:
   - git clones, then, checks out master (master branch has a ruby
script that queries (using Ohai) the security group name and parses
the role and env names).
   - runs a script from the master branch to parse the environment and
role details into shell env variables
   - Checks out the $AWS_ENV branch (which contains the cookbooks)
   - runs chef-solo, where chef-solo.rb uses ENV['AWS_ENV'],
ENV['AWS_ROLE'] to configure the instance.

Actually the last step is done by running a bluepill `chef-solo` process.
Of course you still cannot change the role of an instance when it is
running, since security groups are bound to an instance.

HTH?

> On Thu, Nov 10, 2011 at 12:08 PM, Aaron Abramson 
> <
 >
> wrote:
>>
>> I've thought about this, but haven't put much work into implementing it
>> yet.  So this is more of a concept than anything.
>>
>> Set up a script listening on a server.  It could be a simple php script,
>> IE http://someurl/script.php
>>
>> Set up the user-data with a bash script that hits the listener.  (wget -O
>> /dev/null http://someurl/script.php?role=myrole)
>>
>> The script receives the request, and it knows 2 things, the public
>> hostname of the server making the request, and the "role" it is requesting.
>>  The script makes certain assumptions (username, ssh-key, etc) and then
>> executes a system command, calling "knife bootstrap *server* -r
>> "role[myrole]"
>>
>> So, if you auto-scale official cannonical ubuntu AMI's, simply have the
>> user-data "ping" your knife listener, and knife then bootstraps the server.
>>
>> Perhaps such a bootstrap request feature could be built into chef-server?
>>
>>
>> Aaron
>>
>> On Nov 10, 2011, at 11:55 AM, John Martinez wrote:
>>
>> >
>> > Hello,
>> >
>> > I agree with you, using knife to provision EC2 instances is not the best
>> > way to do this in autoscaled environments. As a temporary solution, I've
>> > done exactly what you suggest. I pass along a lot of things into 
>> > user-data,
>> > and specifically, a node's role is one of those. I read user-data and 
>> > tell
>> > Chef on the node what its role is (amongst other things). It's a kludge. 
>> > I'm
>> > going to be looking into integrating Chef with CloudFormation, which 
>> > plays
>> > nice(r) with ASGs.
>> >
>> >
>> > http://aws.amazon.com/cloudformation/aws-cloudformation-articles-and-tutorials/
>> > (second link)
>> >
>> > https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithOpscodeChef.pdf
>> >
>> > -john
>> >
>> > On Nov 10, 2011, at 9:44 AM, 
>> > <
 >
>> >  
>> > <
 >
>> > wrote:
>> >
>> >> Hi all, I'd like to get some feedback on how people are doing this now.
>> >> It
>> >> seems that since .10, the preferred way of starting Amazon EC2
>> >> instances is via
>> >> the 'knife ec2 server create' command, while the 'knife ec2 instance
>> >> data' and
>> >> chef preinstalled AMIs are deprecated. That's all fine and good for
>> >> manually
>> >> launching instances, but what about autoscaling? When a node comes up
>> >> for
>> >> autoscalling, I want it to update and configure itself according to its
>> >> role(s)
>> >> and environment. With the instance data method of doing thing
>> >> deprecated,
>> >> what's the alternative (aside from having a custom baked AMI that I
>> >> need to
>> >> manually update as is the normal way of doing this without chef)? If
>> >> there is
>> >> no alternative, and I have to use the deprecated method which would
>> >> mean using
>> >> 'knife ec2 instance data' to wire into the autoscaling group as
>> >> UserData. When
>> >> using this method how would I specify the environment that the server
>> >> should be
>> >> run as too? I don't see anyway to do this using the instance data.
>> >> Running some
>> >> command outside the server after the server it is started to set the
>> >> environment is not an option as we won't know when a new server has
>> >> been spun
>> >> up (controlled by Amazon).
>> >>
>> >> I'm interested in what other people have found to be best practice for
>> >> this
>> >> scenario and any advice, tips or tricks that have been learned while
>> >> trying to
>> >> work with chef and autoscaled EC2 instances.
>> >>
>> >> Thanks for your thoughts!
>> >
>>
>
>



-- 
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com