[chef] Re: Why does chef have a two-pass execution flow?

[Sean OMeara < >]

Hi Ian.

This is one of the biggest conceptual stumbling blocks with new Chef users.

Chef is very much not "just imperative programming" (despite what
you'll read at certain places on the internet). Chef is about building
a set of convergent resource statements called a Resource Collection.
By doing this, it allows the you to create subscribe/notify relations
between resources, so you can restart a service when you fix a
configuration file, for example. Having a compilation phase allows you
to make decisions about what to put on the collection.

It allows for things like this:

1.upto(99) { |i|
  beer_bottle "#{i}" do
    action [:take_down, :pass_around]
  end
}

That would place 99 uniquely named convergent beer_bottles on the
resource collection.

More usefully, it lets you take advantage of ruby libraries (sql,
chef, anything) to drive data about what to declare.

mah_nodes = search(:node, "role:webserver")
template "lb_config" do
  variables( :members => mah_nodes)
end

Chef was designed with a larger infrastructure in mind, not just
single nodes. Spinning up a new node can automatically be integrated
without having to manually track topology information, which is very
hard, if not impossible to do on IaaS providers.

As to your concerns:

- "It is very easy to execute tasks out of order by accidentally
defining work outside of a resource"

With great power comes great responsibility. There's also nothing
stopping you from dd'ing /dev/urandom into your boot sector. After
some practice with Chef and an understanding of the resource
collection, spotting instances of "doing work in the compile phase"
stand out very sorely at a glance.

Repeat: "Chef is Not Just Ruby"

- "LWRPs that declare inline resource execute in the compilation phase"

Actually, no, LWRPs (or "custom types", as I've starting calling them)
are evaluated in the compilation phase, and added to the resource
collection just like any core resource type. Currently, custom_type
with nested_resource1 and nested_resource2 will appear as three
resources on the collection. In Chef11, there will be the option to
have it appear as a single resource, through the magic of
run_contexts.

- "It is not very obvious to newcomers"

I could not agree more. We make a point to spend a bit of time
explaining this and making it as clear as possible in our training
sessions, but the docs could definitely be more clear about it. Making
this more widely understood is a bit of a personal mission of mine.
Unfortunately, the "Chef is just pure Ruby" myth remains widespread.

Hope that helps,

-s

On Mon, Jan 28, 2013 at 9:16 PM, Ian MacLeod 
<
 >
 wrote:
> Sorry if this is (most likely) a repost; I wasn't able to find any dupes in
> the list archive, though :(
>
> Chef executes recipes with a two pass model: First it compiles each
> resource/provider by walking through the run list.  Then it converges each
> of the compiled resource/providers to get the host to the desired state.
>
> While I (finally) understand how it works; I don't understand the why.  It
> seems like there are quite a few pitfalls w/ this model:
>
> * It is very easy to execute tasks out of order by accidentally defining
> work outside of a resource (forgetting to use execute/ruby_block providers,
> dropping to lower level Ruby, etc)
>
> * LWRPs that declare inline resources have a confusingly different execution
> order (their actions occur in the convergence phase, but any resources they
> declare get run after the current provider)
>
> * It is not very obvious to newcomers; they seem to expect that recipes are
> executed immediately.
>
> What are the benefits of this two pass architecture?  Why was Chef built
> this way?  (I'm not finding obvious docs on it)
>
> Thanks,
> -Ian
>