[chef] Re: Newbie on autoscaling with Chef

[Daniel Condomitti < >]

All of this is absolutely possible. It sounds complicated at first but when it’s broken down into the three different tasks it’s really not that bad:

Automatic scaling with ASGs is most effectively accomplished using custom metrics[1]. You can do this based off of the built in metrics for EC2 nodes for memory or CPU usage but normally queue depth is a more accurate measurement of when you need to bring up more workers. If you have a fixed number of workers running jobs they normally won’t dramatically increase CPU load on their own.

Registering the workers can be handled by either including your chef validator and client.rb (with the chef client preinstalled) in a custom AMI[2] or by using instance metadata to handle installing it[3]. Creating an AMI with your validator and chef-client already preinstalled would be the easiest to accomplish as the only metadata that needs to be passed in is a role or runlist.

Cleanup can be handled with shutdown hooks depending on the permissions on your chef server. At my org we use knife delete_[client,node] in orchestration scripts for cleaning up production/staging nodes but allow vagrant nodes to clean themselves up. There’s a good example for self-cleanup of nodes on cotap’s engineering blog [4] though if you’re using Ubutntu (we’re not) but a similar process can be built for any other distro.

[1] http://www.thatsgeeky.com/2012/01/autoscaling-with-custom-metrics/

[2] http://marksdevserver.com/2013/06/19/chef-bootstrap-autoscaled-ec2-instance/

[3] http://awsadvent.tumblr.com/post/37773106407/bootstrap-cfg-mgmt-aws

[4] http://engineering.cotap.com/post/66396370532/instance-deregistration-with-chef-and-sensu

On Tuesday, November 26, 2013 at 2:25 AM, James Crosswell wrote:

I'd like to use Chef to manage an application hosted on Amazon's EC2 platform. There are three "components" to the appliction:

1. Web application
2. Database server
3. Background worker/application

I'd like to have the 3rd component (background worker) automatically scale based on CPU usage. So there might be one permanent background worker, but if CPU levels on that worker remained above x% for over y seconds then a couple of extra background workers would automatically be powered up. Similarly, if CPU levels dropped below z% for y seconds, a couple of background workers would automatically be terminated.

I think a major piece of the puzzle that is missing for me is how to register new instances as nodes with the Chef server along with the necessary cookbooks. All of the stuff I've read on using chef so far appears to assume a static list of nodes that are being managed manually from a workstation. What I'd like to do is have new nodes (not necessarily workstations) register themselves to be configured by Chef... and also drop off the list of managed nodes when they are terminated.

Is this possible using Chef?

Kind Regards,

James Crosswell

Founder | Tea Boy

www.mentaldesk.com