[chef] Re: New user feedback and questions

[Adam Jacob < >]

On Mon, Sep 7, 2009 at 3:57 AM, Brian 
Candler<
 >
 wrote:
> Hello,
>
> I've been dabbling a bit with Chef, and I thought I would give a bit of
> new-user feedback, in case you're interested :-)

Hell yes we're interested! :)  Thank you so much for taking the time, Brian.

> (1) Something which confused me at first was the use of code blocks. In Ruby
> these are usually callbacks to be executed zero or more times at some point
> in the future (look at how Rake uses code blocks, for example).

Or they are just closures/lambda's, which is what we're using them for.

> But in Chef, it took me a while to realise that these are just executed
> immediately as syntactic sugar for setting parameters. Take this example:
>
>    service "apache" do
>      action :enable
>    end
>
>    directory "/tmp/something" do
>      recursive true
>      action :delete
>    end
>
> It becomes much clearer (to me) if I translate this mentally to:
>
>    service "apache", :action => :enable
>
>    directory "/tmp/something",
>      :recursive => true,
>      :action => :delete
>
> Now I see these as just a series of actions with parameters which are
> executed one after the other, not tasks to be deferred. Therefore the issues
> about ordering of actions vanish: if you want action A to be executed before
> action B, you just write A before B :-)

The tricky part is, they *are* being executed later.  When you write:

service "apache" do
  action :enable
end

You are creating a new resource, and adding it to the
ResourceCollection.  It's more accurate to think of it this way:

r = service "apache"
r.action :enable
collection << r

The block that follows is just being evaluated in the context of the
resource created by 'service "apache"'.  It then pushes the new
resource onto the ResourceCollection.  Chef then walks the
ResourceCollection and executes every action, sends notifications,
etc.  This two-phase action (compiling the ResourceCollection, then
executing it) allows you a crazy high degree of flexibility (you can
go back and inspect the resource collection, for example, or modify a
resource later, etc.)

Also, when you are in the block you are executing code in the context
of an instance of the resource class - which means you can insert any
arbitrary code you might want in there. (And people do - for example,
using a case statement to set an attribute to two different values.)

> <aside>
> The second syntax I've shown above doesn't work right now, but perhaps
> it could if the optional resource parameters 'collection' and 'node' were
> made into hash options, i.e.
>
>  directory "/tmp/foo", :collection=>xxx, :node=>yyy
> </aside>

I took that approach early on in the design of Chef's recipe syntax,
and I found I much preferred the block syntax for a few reasons:

1) It's fewer characters
2) It doesn't require comma's after every attribute
3) It lets me use arbitrary ruby to set attributes

We're always open to changing things up, but I for one strongly prefer
the current syntax.

> It's now clear to me (but wasn't at first) that there are no dependencies
> apart from those which you set up with notifies/subscribes. For example: if
> you define a task using 'execute', it will be run *every* time your recipe
> is run, unless you set up a precondition like 'creates' or 'only_if'

Right - Chef does what you tell it to do, in the order you tell it to
do it.  Idempotency (whether a thing should/should not be done)
happens at the Resource level.  For most resources, we take care of
this for you (we detect whether a package is installed and at the
correct version, say).  For some it's impossible - things like execute
or script resources, or you may know better what the conditions are.
In that case, you use not_if/only_if.

> I also found the concept of 'resource' a bit unclear at first. For example,
> I may want to create a file using a script. But in that case, the 'resource'
> is not a 'file' (the thing I want), it is an 'execute' (the way to create
> it).

Right - the resource abstraction provides a way to easily declare that
you want a particular thing to be in a particular state at a given
point in the Chef run.

> (2) Based on the above, I think there's a gap in the documentation between
> installing/getting started (which installs your first cookbook without
> really understanding it) and the detail of the individual components. But
> maybe I was just being dumb.
>
> I found the detailed documentation on resources/providers etc was fine, and
> in any case it's easy enough to dig into the code once you get down to that
> level.

Have any suggestions on where in the documentation we should insert
some more knowledge for folks?

> I found another gap at
> http://wiki.opscode.com/display/chef/Anatomy+of+a+Chef+Run
> where it talks about "saving the state of a node". What exactly is saved?
> For example, does every 'file' resource keep a backup of the given file?
> Or is it just a dump of the node state from Ohai?

The attributes and run list of the node.  That means data from Ohai,
but also any other attributes you set on the node.  This lets you do
some pretty fun things - like have a recipe that upgrades your kernel,
sets an attribute, reboots, and takes specific action after the
reboot.

> (Right now I'm just using chef solo rather than a server)

So the above does not apply. :)

> (3) In the environment where I work, the most likely need for chef is in
> distributing config files [*].
>
> So I'm looking as to ways we can make this as safe and simple as possible
> for the sysadmins to use.
>
> Have you any pointers to best practices on *testing* new recipes? I don't
> really want to set up a whole dev/test environment just for developing and
> testing each recipe change - it would be too cumbersome. But equally I don't
> want invalid recipes being pushed out, or worse, ones which destroy data.

Testing recipes without taking the actions described is basically an
impossible task to get 100% correct.  You can do dry runs, but any
time you check the run-time state of the system and that state could
have been modified previously in the run, you're going to get a false
positive.

We've been working on ways to let you write real integration tests,
which would go a long way towards allowing you to do real testing in a
VM.  As it stands, the best practice is to have a staging environment
that gets your new changes first, and only prop changes to production
when you are comfortable.

> One possibility would be to distribute new versions of cookbooks to
> machines, but only activate them in a dry-run way (like make -n). Only once
> the expected actions are checked would I then enable the new cookbook to be
> run live.

This is on the road-map, but see what I said above for the fact that
it might not actually tell you the whole truth.

> Anyway, that's about it. This looks like a cool framework for machine
> management, so thanks for releasing it!

You are most welcome.  Thank you for taking the time to write such
detailed feedback, and I look forward to hearing more about your
experiences with Chef!

> However I can also see a need for a resource to fetch a particular svn/git
> tag of source code, compile it, and install it. This would make it much
> easier to deploy a head or near-head version of a rapidly developing
> application such as CouchDB, without having to keep re-packaging it.

A source package provider is on the way, and an SCM resource with SVN
and Git providers are also in the works.

Best,
Adam

-- 
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: