[chef] Re: Request for input on orchestration

["John E. Vincent (lusis)" < >]

On Fri, Jan 28, 2011 at 3:26 PM, Chris Walters 
<
 >
 wrote:
> Ohai Chefs!
>
> We're in the preliminary stages of designing possible solutions for
> orchestration and would like to understand the community's
> requirements.
>
> I'm going to write down my thoughts and questions. Nothing is gospel,
> so please feel free to comment on everything, including the framing.
>
> Background:
>
> Chef, as currently conceived, does a great job of exposing a model for
> how to get a system from either an embryonic state or a slightly
> misconfigured state to the desired state, mainly via the mechanism of
> resource idempotence.
>
> What I think is not yet well-modeled is how to go from one
> well-configured state to a completely different well-configued
> state. It also doesn't yet model synchronization of actions across
> multiple boxes in that there isn't a first-class way to gate actions
> that are dependent on the completion of steps on other servers. For
> example, a complex migration or deployment might require bringing
> boxes up or down, copying data, cleanly removing artifacts or services
> installed by previous chef runs, not restarting load balancers until
> some quorum of webservers have re-started, etc.
>
> We'd like to collect the use cases, requirements, and thoughts that
> best serve the community.
>
> 1) What do you think the scope of orchestration is and is not?
>

Orchestration is a pretty loaded term. In my VERY opinionated mind it
mainly revolves around the coordination of constituent parts of the
system as a whole. I think conductor when I think orchestration. The
brass section is my web farm, the woodwinds are the apis and the
percussion brings up data.

> 2) What are the use cases that you would like to see an orchestration
> system/DSL accommodate? The more specific and granular the steps of
> the orchestration, the better. (If you would not like your use case
> made public but would nonetheless like it considered during design,
> validation, and testing, please send it to me directly at
> 
 )
>

Having taken my own swing at this problem space, there are a few key
things you need to be able to express. Some of these chef does already
in an abstract sense.

- relationships/deps between nodes
- relationships/deps between things running on the node
- gating and phasing
- unmanaged resources

As for a DSL it could be as simple as:

depends_external "foo"
  type "role"
end

where foo is the result of a role search and the run will block until
that condition is met

or

triggers "haproxy" do
  role "loadbalancer"
  action :restart
end

to trigger a restart of the haproxy service on all nodes of type loadbalancer.

Obviously if you can express these in a more chef-appropriate
data-driven approach all's the better. I'll be rewritting the Noah
LWRP to do that before ChefConf.

Unmanaged resources, however, is the stickler.

This is the reason I wrote Noah. The Chef API is dead simple but it
doesn't do call backs (nor should it probably) and it's not as
friendly for application developers to interact with (you need a cert
and whatnot).

The Noah approach was to be a bridge between those two worlds -
unmanaged and managed - by providing a simple API with callbacks.

> 3) What generic primitives do you think would be useful in such a
> system?

http://lusislog.blogspot.com/2011/04/operational-primitives.html

As a side note, if you want to look at the Noah LWRP in its current
state, feel free. I'm not implying that it has the answers but it's an
example of how I've taken to solve these problems:

https://github.com/lusis-cookbooks/noah

I've successfully used this to do demos of most of the stack-based
chef cookbooks (wordpress, django) fully gated and phased (see the
BLOCKING section). I also was brainstorming with Seth Chisamore about
a pokeable chef-client daemon that would get callbacks from Noah. The
idea being that you push something to Noah from the cookbook at the
end of the run. There's a registered watcher in Noah that pokes the
chef-client on the node.

>
> Thanks!
> Chris Walters