[chef-dev] Re: Re: Re: Re: Re: Fwd: How do I know if my application has really been "provisioned"? a suggestion

[Mark Van De Vyver < >]

Hi,
Interesting thread...

On Mon, Dec 10, 2012 at 7:59 AM, Peter Donald 
<
 >
 wrote:
> It would be nice if chef allowed to to converge recipes/resources in
> parallel where possible and even the possibility of futures to join against
> when you wanted to wait between resources however that would significantly
> increase the complexity of chef.
>
LTDR:
+1 Erik Hollensbe's wait/ensure{}
-{\infty} parallelism in Chef
Of course +2 My Observer suggestion.

Anyway:

Erik Hollensbe  also mentioned a desire for parallelism in Chef runs.

Parallel runs are very useful.

Without raining on the Unicorn parade can I suggest this be addressed
in other ways?
My experiences with MPI and PETSc make me more than a little leary of
casual talk about parallelism.  Essentially, outside of embarrassingly
parallel tasks, you'll get your self into a whole lot of pain very
quickly.  Also, I think the ZeroMQ guys made a compelling case against
multithreading, again just from experience these are tough problems to
execute reliably, never mind rigorously - the early MPI list has some
interesting debates about whether some parts of the spec is
theoretically possible, and how can people be claiming to have
implemented it fully.

I don't think the Chef community will get a great deal of benefit by
trying to bake parallelism into Chef.  I'm pretty sure they will get a
whole lot of pain - Does ant one really want to reach the point where
we start saying we need a Chef plugin for TotalView or Eclipse/PTP in
order to find
out why some Chef run behaves the way it does?

However, we *do* all want things to run in parallel.
I'm writing a lib to be released shortly that does use the parallelism
provided by
another library, so I'm not against this on principle.

A chef run can get complex, so I think the way to address needs for
parallelism is to suggest multiple chef-solo/client runs, and leave it
up to the recipes in each process/run to wait as they need to.  Some
best practice suggestions/patterns will emerge. One simple one is
'share nothing' between these processes, where share refers to
roles/recipes/cookbooks, not databags etc. which should be shared.
Another is don't change any data that another run uses, without it
being communicated via the wait/ensure{} block.  Even this opens up
a Pandora's box for the unaware. But hopefully we can make an effort to push
back and keep things simple and not have this list descend
into communication topology discussions around parallel Chef runs.

This suggests Erik Hollensbe wait LWRP should suffice for immediate
needs. Longer term I agree that something like the `ensure{}`
suggestion might work to allow one chef-solo/client run to coordinate
with another.  Irrespective of whether that other process/Chef-Solo run is on
another machine or the same machine.

The benefit of this is that:
 - Inexperienced users are encouraged to adopt embarrassingly parallel
setups (run x instances of chef-solo each with their 'role'/'cookbook'
as waiting/ensuring appropriate)
 - Attention/emphasis switches to making/keeping chef-solo/client runs
as lean as possible - which benefits everyone.

For users who are adamant that their problem requires Chef be
"parallelized": perhaps the can be gently nudged in the direction of
Ruote, or Condor.  If they are really adamant that directed acyclic
graphs just won't suffice for them, then more firmly point them to MPI
and hope they write a ffi-MPI + chef extension for some kind of
virtual topology?

Hope that helps

Best wishes