[chef] Re: Re: Re: Request for input on orchestration

[Avishai Ish-Shalom < >]

Going over people cookbooks and this list, I've seen quite a few scenarios where people built makeshift orchestration scripts. A few example would be my own ec2-chef bridge to drive actions based on data exchange between ec2 api calls and chef data (a node wen down, so update an ec2_status attribute a force a chef run on other nodes, a chef run failed so mark that node unhealthy in the autoscaling group), the Chef cluster cookbooks using "cluster[:service][:timestamp]" attributes to orchestrate service discovery, complex application deployments scripts using chef attributes and so on.

So, I think many people need orchestration features so much they build them from scratch, over an over again. Having a standard interface or engine for building these solutions will definitely be a step forward, and I would like to make some suggestions for chef features that may promote such an engine and/or integration with existing orchestration tools:

• Chef server plugin/trigger framework - this will make writing external attribute processors, tools using chef's node status database, monitoring tools and most importantly - responding to events like chef run success/failure, attribute change etc.
• Provider status - I share the view that mis-configured node should be thrown away and built from scratch, however, as chef scripts become more complex and more platforms are supported, we are seeing more and more failure conditions that can be easily overcome by some generic brute error handler. Also, it's important to note that while Chef's philosophy tend towards idempotency, pure idempotency is seldom possible in practice as the "not_if" conditions well prove. I'm already making a lot of assumptions about OS status when writing cookbooks (a package can be installed, a user can be created) that cannot yet be described by chef.
• Resource status - most providers already check the status of resource, mainly to compare new_resource with current_resource. Why not expose the low level status of a resource to recipes?
• Resource status triggers - currently we can only trigger other resources if a resource has been updated. But what about triggering a resource on resource action failure? or perhaps even some other complex condition based on the provider status?

These are just my 2 cents, feel free to bash me for them.

Regards,
Avishai

On 01/30/2011 09:27 AM, Hedge Hog wrote:

" type="cite">

On Sat, Jan 29, 2011 at 8:19 AM, Dan Nemec 
 
 "><
 > wrote:

Mandatory disclosure. We've been using Control Tier for over a year.
I don't know that I am interested in Chef being the orchestration engine. I
would like to see Chef more friendly with other existing orchestration
tools. Control Tier is great if you want to code much of your customized
commands into the tool. The spinoff RunDeck is great if you have your own
scripts but just need them orchestrated.
We envision a scenario where Chef is mainly configuration and Control Tier
is orchestration. We would go so far as having Chef configure Control Tier
which could then call back to Chef to initiate Chef actions.
Orchestration is a difficult problem and Control Tier/RunDeck is already
fairly mature. Now that they support arbitrary workflows in Jobcenter
Control Tier is't missing any feature we require for orchestrating our very
complex deployments.
I don't know that I have much to add to your questions above. We're fairly
new to Chef and it being Friday afternoon is making my head hurt thinking of
how to incorporate orchestration into Chef. All I can imagine is how we're
doing it today with Control Tier.
1) You hit the high points in your overview of scope.
2/3) All I can say is to read Control Tier's documentation. In Control Tier
you define a "command" that performs a piece of work. You string a bunch of
commands together into a workflow. Then you can also string different
workflows into larger events. You add to that the error-handling and gating
so that the next step only executes after the first one completed
successfully. Threading concurrently across servers is mandatory to make it
go fast.
Happy Friday,
Dan

On Fri, Jan 28, 2011 at 3:26 PM, Chris Walters 
 
 "><
 > wrote:

Ohai Chefs!
We're in the preliminary stages of designing possible solutions for
orchestration and would like to understand the community's
requirements.
I'm going to write down my thoughts and questions. Nothing is gospel,
so please feel free to comment on everything, including the framing.
Background:
Chef, as currently conceived, does a great job of exposing a model for
how to get a system from either an embryonic state or a slightly
misconfigured state to the desired state, mainly via the mechanism of
resource idempotence.
What I think is not yet well-modeled is how to go from one
well-configured state to a completely different well-configued
state. It also doesn't yet model synchronization of actions across
multiple boxes in that there isn't a first-class way to gate actions
that are dependent on the completion of steps on other servers. For
example, a complex migration or deployment might require bringing
boxes up or down, copying data, cleanly removing artifacts or services
installed by previous chef runs, not restarting load balancers until
some quorum of webservers have re-started, etc.
We'd like to collect the use cases, requirements, and thoughts that
best serve the community.
1) What do you think the scope of orchestration is and is not?
2) What are the use cases that you would like to see an orchestration
system/DSL accommodate? The more specific and granular the steps of
the orchestration, the better. (If you would not like your use case
made public but would nonetheless like it considered during design,
validation, and testing, please send it to me directly at

 
 ">
 .)
3) What generic primitives do you think would be useful in such a
system?
Thanks!
Chris Walters

Orchestration is something I have yet to deal with.  It will loom and
I had in mind to try and use Ruote[1].
RunDeck sounds like it might be a specialized (aka limited?) workflow engine.
Given ruote-kit (RESTful)[2], ruote-jig[3] decision tables[4],
persistence, ampq, etc. etc. I'd be surprised if solving Chef
orchestration didn't involve adopting/employing a subset of ruotes
existing functionality.
As mentioned any chef specifics could be placed in a ruote-chef project?
I think a route-chef project might give biggest bang-for-the-buck.

[1] http://openwferu.rubyforge.org/
[2] https://github.com/kennethkalmer/ruote-kit
[3] https://github.com/jmettraux/rufus-jig
[4] http://jmettraux.wordpress.com/?s=decision+tables

HTH