[chef] Re: aws autoscaling and chef cleanup

[Morgan Blackthorne < >]

This is what I use: https://gist.github.com/stormerider/5600427 -- bear in mind you're still using a node with knife access, but it doesn't have to be the Chef server. Any workstation can suffice.

This was based on a script that floated around the mailing list a while back. I modified it so that a) it checks more than one AWS account (at one point we had three, now we're down to 2 and soon just one), and b) it checks more than one AWS region. For Chef 11 I also had to modify it so that it uses the embedded Chef ruby (and I also had to install the aws gem inside Chef as well).

This doesn't necessarily catch everything. I do have some tool scripts to find nodes that exist in EC2 that don't exist in Chef (and vice versa...). If for some reason the ec2.instance_id field gets nulled out, the node can get stuck in limbo. This also applies to nodes that are in a stopped (vs. terminated) state, because in that case the instance_id field is still valid, but everything else is bogus.

--

~*~ StormeRider ~*~

"Every world needs its heroes [...] They inspire us to be better than we are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, May 17, 2013 at 9:43 AM, Sam Darwin < " target="_blank"> > wrote:

If using AWS auto-scaling + Chef, the final step of instance cleanup seems to
be slightly unclear.

One solution is to run a script in /etc/rc0.d which is called on shutdown.
"knife node delete".    This requires knife to be configured and working on the
instance, which is a (minor) pain.    This method will also fail for an abrupt
machine crash.

Another solution is to have a script which queries chef server for instances
that haven't checked in for a while, and removes those.   That would require
having chef-client running very often or as a daemon.

I wonder what the security implications would be of adding functionality into
chef-client:

chef-client --remove-self-from-server

Some people have posted about a script which checks for terminated instances
and removes them.   this sounds like the best way.    Perhaps they mean to
query AWS first, and then make changes to chef-server.    now to figure out
how...