[chef] Re: Re: Re: Re: Re: Re: Re: Chef on Amazon EC2 with auto-scaling

[Avishai Ish-Shalom < >]

You are almost correct. User data can be modified after the instance is up using ec2-modify-instance-attribute, however this is cumbersome and requires the instance to be stopped first (and naturally only works with EBS based amis). sigh.

The secure, convenient option would be a signed s3 link set to expire in 15 minutes, this however forces you to generate user-data using templates (i use erubies) and doesn't work with autoscaling.

In short, if you want a method that works with autoscaling and doesn't require bundling an ami, you're screwed security-wise. Unless someone in this list figures how to do it of course...  i've considered ip bound one-time tokens but decided against implementing yet another security layer. I bake ami's with validation.pem when i can and take my chances when i don't have the time.

Regards,
Avishai

On 23/07/11 03:38, Bryan Brandau wrote:

" type="cite">Avishai,

Passing in the validation.pem with the user-data becomes a security concern.  It will always be present as the instance runs.  Once a client gets it's client.pem you should be removing the validation.pem.  You won't be able to do this when it's passed in.

On Mon, Jul 18, 2011 at 8:25 AM, Avishai Ish-Shalom < "> > wrote:

I've been using chef with autoscaling quite often in the last two years.
I've found the most versatile approach is to have a minimal user-data
script take care of bootstrapping chef and let chef do the rest. the
validation.pem is included in the user-data.

Another, somewhat unrelated script is a cleanup daemon, basically it
scans the list of EC2 servers periodically and updates chef nodes with
the status of the related ec2 instance (e.g. node[:ec2][:status] =
"running", matched on instance-id). This allows filtering search results
for servers that are dead/stopped/etc. the daemon also removes nodes and
clients after a they are dead for a while.

pros of my method:

* no ami maintenance, you can use any community ami

* works very well with high rates of recycling nodes

* simple, easy to extend and modify cluster functionality

* cleanup doesn't depend on proper node shutdown


Cons (those i thought of at least)

* servers take longer to get to "production ready" status

* chef server and recipes become major points of failure for autoscaling

* a little more load on chef server

Regards,
Avishai

On 18/07/11 15:46, Edward Sargisson wrote:

> Hi,
> Please forgive me for directing you to my own blog but here is my post
> on how I did it [1] (which Opscode kindly link to). This method
> (provided to me on this list) uses Ubuntu's cloud-init to bootstrap
> Chef onto the image and then gets Chef to do the rest.
>
> Re: OS upgrades. If you mean package upgrades then write a cookbook
> that does it. There is an apt cookbook for ubuntu that updates the
> package list but doesn't run the upgrade for you.
>
> If you want to actually upgrade the OS (i.e. Ubuntu Maverick to Natty)
> then Chef doesn't do this directly. In EC2 these images are pre-baked
> so, with Chef, instead of starting with the Maverick image you start
> with the Natty image. Chef will then install everything else you need
> and you just need to test to make sure it worked..
>
> [1] http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/
>
> On Mon, Jul 18, 2011 at 5:16 AM, Bryan McLellan < "> > wrote:
>> On Jul 18, 2011 6:32 AM, "Oliver Beattie" < "> > wrote:
>>> * As I originally mentioned, what is the procedure for managing these
>>> servers? Would I just be able to run commands via knife to all my servers?
>>> How does it keep track of nodes joining (or more importantly leaving) my
>>> "cluster"?
>> Knife uses the Chef server API to talk to the server. Since all nodes
>> register with the server (both a node object for the data and a client
>> object for authentication) knife node list produces a list of all nodes
>> registered with the server. Knife doesn't know about nodes itself. When you
>> use knife to create a new system, via ec2 server create or bootstrap, the
>> node still registers itself with the chef server, not knife.
>>
>>> * Another (somewhat unrelated question) I had is how does Chef manage OS
>>> upgrades? Does it manage them at all? For instance, how would I say "go run
>>> aptitude upgrade on all my production servers"?
>> knife ssh name:* "sudo aptitude upgrade -y"
>>
>> Or you can create a cookbook to do this if you trust upstream to produce
>> non-breaking changes.
>>
>> Chef itself doesn't manage OS upgrades, but it certainly can.Remember that
>> Chef is a tool designed to help you automate your systems. A hammer doesn't
>> pound nails alone.
>>
>> Bryan
>>

begin:vcard
fn:Avishai Ish-Shalom
n:Ish-Shalom;Avishai
org:FewBytes technologies
adr:;;7 Jabotinsky St.;Ramat-Gan;;52520;Israel
email;internet:
 
title:IT Consultant
tel;work:+972-3-5444-230 
tel;fax:+972-3-5444-720 
tel;cell:+972-54-4466499
note:skype: avishaiish
x-mozilla-html:TRUE
url:http://www.fewbytes.com/
version:2.1
end:vcard

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature