[chef] Re: aws autoscaling and chef cleanup

[Cassiano Leal < >]

I have one machine profile that runs on auto-scale. It's an ephemeral worker that listens on a sidekiq queue and performs tasks.

I took three steps to ensure that those don't keep polluting the Chef Server.

1 - During the first Chef run, it adds to its own run list a recipe that actually deletes itself (node and client) from the server. This way, on the second run it gets deleted. -- Note that this machine gets configured once and stays that way until its life ends, which is perfectly fine for my use case, but might not be for yours;

2 - Another recipe adds a rc0 script to delete it when it shuts down (this is in case the first chef run never completes -- ideally it would be executed at compile time. Alternatively, this script could be baked in the AMI or created in the user-data script;

3 - I have a cron job that searches for stray nodes and deletes them -- how to do that will depend on your setup, but you seem to have a pretty good grasp of what you'll need.

As for setting up knife, this is a non-issue. Just point it to your client.rb:

knife node delete <%= node.name %> -y -c /etc/chef/client.rb

knife client delete <%= node.name %> -y -c /etc/chef/client.rb

It's been working quite well for me for about 4 or 5 months already, but my setup isn't anything very fancy. :)

Hope this helps a bit!

- cassiano

On Friday, May 17, 2013 at 13:43, Sam Darwin wrote:

If using AWS auto-scaling + Chef, the final step of instance cleanup seems to

be slightly unclear.

One solution is to run a script in /etc/rc0.d which is called on shutdown.

"knife node delete". This requires knife to be configured and working on the

instance, which is a (minor) pain. This method will also fail for an abrupt

machine crash.

Another solution is to have a script which queries chef server for instances

that haven't checked in for a while, and removes those. That would require

having chef-client running very often or as a daemon.

I wonder what the security implications would be of adding functionality into

chef-client:

chef-client --remove-self-from-server

Some people have posted about a script which checks for terminated instances

and removes them. this sounds like the best way. Perhaps they mean to

query AWS first, and then make changes to chef-server. now to figure out

how...