[chef-dev] Re: Re: Re: Re: Re: Re: Fwd: How do I know if my application has really been "provisioned"? a suggestion

[Erik Hollensbe < >]

On Dec 9, 2012, at 3:07 PM, Mark Van De Vyver 
<
 >
 wrote:

> Hi,
> Interesting thread...
> 
> On Mon, Dec 10, 2012 at 7:59 AM, Peter Donald 
> <
 >
>  wrote:
>> It would be nice if chef allowed to to converge recipes/resources in
>> parallel where possible and even the possibility of futures to join against
>> when you wanted to wait between resources however that would significantly
>> increase the complexity of chef.
>> 
> LTDR:
> +1 Erik Hollensbe's wait/ensure{}
> -{\infty} parallelism in Chef
> Of course +2 My Observer suggestion.
> 
> Anyway:
> 
> Erik Hollensbe  also mentioned a desire for parallelism in Chef runs.
> 
> Parallel runs are very useful.
> 
> Without raining on the Unicorn parade can I suggest this be addressed
> in other ways?
> My experiences with MPI and PETSc make me more than a little leary of
> casual talk about parallelism.  Essentially, outside of embarrassingly
> parallel tasks, you'll get your self into a whole lot of pain very
> quickly.  Also, I think the ZeroMQ guys made a compelling case against
> multithreading, again just from experience these are tough problems to
> execute reliably, never mind rigorously - the early MPI list has some
> interesting debates about whether some parts of the spec is
> theoretically possible, and how can people be claiming to have
> implemented it fully.
> 
> I don't think the Chef community will get a great deal of benefit by
> trying to bake parallelism into Chef.  I'm pretty sure they will get a
> whole lot of pain - Does ant one really want to reach the point where
> we start saying we need a Chef plugin for TotalView or Eclipse/PTP in
> order to find
> out why some Chef run behaves the way it does?

Well, I should explain some things. The scheduler I wrote isn't really built 
for chef -- it's a generic scheduler that works in kernel style and is used 
for a testing system I'm working on that needs to do a lot of expensive, 
blocking things. It just turns out that after I built it, that I realized it 
really wouldn't take much to bolt this right over chef's resource runs 
because they already do exactly what I'm doing, so it may turn out that this 
is a worthwhile effort.

I also agree with you on multithreading and parallel runs in general -- this 
isn't for everyone or every use case, including the use cases that the 
scheduler was built for -- it has both parallel and serial modes, which work 
interchangably because there is no shared state between threads and the 
coordinator is always a single thread -- whether that's ruby's "main" thread 
or a secondary coordination thread is arguably unimportant to how it operates.

Personally I don't care about coordination coming from chef -- chef doesn't 
need to be a be-everything-to-everyone tool, that's the path to madness, and 
there's plenty of work in this area that could be hooked into chef (things 
like mcollective come to mind) without being so invasive that chef needs to 
change.

The scheduler is just a coordination thread that pops a queue. Other calls 
are allowed to write the queue but the scheduler itself is only expected to 
read from it. The collection of resources (server groups in my case) has an 
associated list of dependencies, currently with a restriction that the 
dependencies have already been declared, and it also has a group of code to 
execute (exposed as objects similar to the actor model). Items come through 
the queue and are lined up with these resources -- if there are no 
dependencies to satisfy, execution happens in a child thread and when it 
comes back its state is inspected -- if it succeeded, it gets put in a 
"satisfied" set, on failure the scheduler barfs (it is built for a testing 
system, after all). If there are any outstanding dependencies, it gets pushed 
back on to the queue and will be re-evaluated until its dependencies are 
satisifed.

In serial mode, the child threads just execute in the main thread and the 
queue coordinator also runs in the main thread. This 100% translates to a 
single threaded scenario -- all this business about shared state (or the lack 
thereof) is also what the ZeroMQ guys mean when they say don't use threading 
(I have a bit of history with ZeroMQ and similar tech when I was still doing 
application programming). :)

I think if you think about it some, you'll see the parallels (heh!) to chef 
runs here. During compilation phase the resources are arranged into a queue 
based on what order they need to execute in, and then that queue is acted 
upon. Resources are independent, just like my provisioner classes -- there 
are a few cases where you can abuse rubyisms to modify state outside of the 
resource (local variables and calls like String#replace come to mind) but 
they are hard to do and firmly in the "I know what I'm doing" department. The 
big difference is that by the time a chef resource converge starts almost all 
of the details are sorted out, and this scheduler is a bit more dynamic in 
the "I've just been given something to do" department.

Anyhow, I want to doubly express that it wasn't built for chef, that chef 
itself wasn't the reason I wrote it, and I have no interest in throwing 
myself at a wall over and over again to get it merged or even tried out. It 
just turns out that it's a disturbingly good fit for how chef already works, 
and I think it was important to mention as I know some people @ opscode have 
been thinking about how to solve this.

Anyhow, probably gonna drop off this thread -- more code to write. :P

-Erik