[chef] Chef stability?

[< >]

Hi,

I've been working the past few days on tweaking my chef scripts to go into
production on EC2 and struggling to get anything I feel good about trusting.
Chef looks like a great tool with a strong community. I'm hoping that there's
some Chef way of looking at the world I haven't been exposed to that you can
all enlighten me on.

I'm running Ubuntu 10.10 on EC2 with the version of chef from the Opscode 
Lucid
repo (0.9.8).

A few things going on:

I can't seem to keep chef-solr or chef-solr-indexer from crashing. I keep
having to restart them for some reason. I'm using It makes everything feel
really flakey, but I'm not convinced that's the only thing I'm running into.

Sometimes the webui (and knife) show the status of all the nodes and sometimes
it refuses saying that I have no nodes (even though the node list shows there
are some there). The error in the logs is only the same 500 internal server
error: connection refused that I see for lots of things.

Running chef-client by hand on a machine causes a different result than 
letting
the timer driven version work. Like it forces the client to reevaluate all the
data bags and search results and actually apply them.

Sometimes the clients get new data/nodes and update everything fine, sometimes
they don't.

Yesterday I started 8 boxes to bring a whole cluster up. On a few of them, 
Chef
just randomly stopped working. Running chef-client by hand finished building
the box correctly. One of them built part of a configuration file using data
from a node that I had deleted off the Chef server a few hours earlier and 
then
could never get out of that state. Deleting the configuration file and
rerunning client fixed it.

Anyway, all of these small, but annoying, little glitches give me a really bad
feeling about trusting Chef to manage my production infrastructure. Of the
tools I've looked at, it's the most promising. 

I'd really like to given the promise of such powerful ability when it works,
the time that I've put into it, and the time it will save. Is anyone using 
Chef
at a large scale? Does it take handholding and massaging along the way, and
that's just the price for cutting-edge technology that will be solved as the
code matures?

Thanks,
Allan