[chef] Re: Re: Re: Re: Ad-hoc tasks and ad-hoc sources

[Alon Rohter < >]

My recipes/roles are fairly efficient, and most of my chef-client runs complete in 15sec or less, so having to execute the full run_list isn't terribly bad today, although the engineer in me cries a little to see many dozens of resources called when only one resource execution is actually needed.

My real need is the ability to execute recipes/resources not in the normal run_list, as ad-hoc tasks (that I dont want run every 30min by the daemon).  Things like triggering the deployment of a new application build, creating and submitting a debug dump, exporting or importing a dataset, etc.  Today I handle this by incorporating all these tasks into the run_list recipes, and then use file or databag guards/flags to prevent them from running until I want them to.  It works, but is inelegant and becomes awkward to scale.

I'm looking for more direct interaction and control of chef executions.  This is an area where RightScale's RightLink has been particularly interesting to me, as I believe it's trying to achieve similar goals in terms of elegant chef recipe runs; which boils down to direct recipe executions triggered by pushed (amqp type) messages.  Basically I'm looking to move away from static & monolithic ssh'd ./chef-client command line runs, towards something more directly programmatic.  

This may not persuade you, but for me Chef gets me 90% of the awesomeness I need, so finding a solution for the other 10% is my goal.

-a

On Fri, Mar 23, 2012 at 4:56 PM, Brad Knowles < " target="_blank"> > wrote:

On Mar 21, 2012, at 6:49 PM, Alon Rohter wrote:

> I'm actually quite interested in ad hoc resource execution as well, particularly as applied to more advanced orchestration uses.  For command and control, full chef-client runs are expensive and awkward, particularly when I need to run one-off resources/recipes.  I love having all my code in one place, and utilizing all the existing knowledge/state that the chef-server keeps about my infrastructure, so having to use some external system (like mcollective) to handle this kind of thing isn't appealing.

If your full chef-client runs are too expensive and awkward, then my view is that they have not been written correctly.  Or, at the very least, you're going to have to do a lot of work to convince me that they have been written as well as they can be, and that there are some operations that just take too long.

Even the default Zenoss cookbook will take nearly 30 minutes to run when it does a full re-model, but it's smart and keeps some internal timers so that only happens every six hours or so.  And even that could be potentially moved outside of chef, so that chef is just responsible for installation and configuration of which nodes need to be monitored, and all the rest is done asynchronously to the chef-client run.


If you need some orchestration hooks, then you can use heavier tools like zookeeper, or lighter ones like noah.  That would at least get you the near real-time callback mechanism, which could then be hooked into a chef run.


I remain extremely skeptical that there is a real need for running ad-hoc individual recipes outside of a proper chef run.  In my book, this falls into the category of extraordinary claims.  Feel free to provide extraordinary proof in order to convince me.

--
Brad Knowles < " target="_blank"> >
SAGE Level IV, Chef Level 0.0.1