[chef] Re: Chef stability?

[Adam Jacob < >]

On Wed, Nov 17, 2010 at 10:09 AM,  
<
 >
 wrote:
> I've been working the past few days on tweaking my chef scripts to go into
> production on EC2 and struggling to get anything I feel good about trusting.
> Chef looks like a great tool with a strong community. I'm hoping that 
> there's
> some Chef way of looking at the world I haven't been exposed to that you can
> all enlighten me on.
>
> I'm running Ubuntu 10.10 on EC2 with the version of chef from the Opscode 
> Lucid
> repo (0.9.8).
>
> A few things going on:
>
> I can't seem to keep chef-solr or chef-solr-indexer from crashing. I keep
> having to restart them for some reason. I'm using It makes everything feel
> really flakey, but I'm not convinced that's the only thing I'm running into.

This is going to be the source of several problems - can you send us a
gist of what you get in the logs when these crash?

> Sometimes the webui (and knife) show the status of all the nodes and 
> sometimes
> it refuses saying that I have no nodes (even though the node list shows 
> there
> are some there). The error in the logs is only the same 500 internal server
> error: connection refused that I see for lots of things.

Those pages both use search - if you are seeing consistent failures of
Solr, thats the source of these issues.

> Running chef-client by hand on a machine causes a different result than 
> letting
> the timer driven version work. Like it forces the client to reevaluate all 
> the
> data bags and search results and actually apply them.

In what way?  The code paths here are identical for the most part.  If
you're using data bags and search in the recipes, and you are seeing
failures of Solr, I would wager that these differences are actually
just a representation of the search service not being stable for you.

> Sometimes the clients get new data/nodes and update everything fine, 
> sometimes
> they don't.

Again, if it's data that comes from search, that's your issue.

> Yesterday I started 8 boxes to bring a whole cluster up. On a few of them, 
> Chef
> just randomly stopped working. Running chef-client by hand finished building
> the box correctly. One of them built part of a configuration file using data
> from a node that I had deleted off the Chef server a few hours earlier and 
> then
> could never get out of that state. Deleting the configuration file and
> rerunning client fixed it.

All the symptoms you talk about sound search related - so we should
focus there. :)

> Anyway, all of these small, but annoying, little glitches give me a really 
> bad
> feeling about trusting Chef to manage my production infrastructure. Of the
> tools I've looked at, it's the most promising.

Sorry to hear that, but it's been quite stable for us (and for lots of
other folks).  We'll get you fixed up.

> I'd really like to given the promise of such powerful ability when it works,
> the time that I've put into it, and the time it will save. Is anyone using 
> Chef
> at a large scale? Does it take handholding and massaging along the way, and
> that's just the price for cutting-edge technology that will be solved as the
> code matures?

There are people using Chef at the scale of many thousands of systems,
and Opscode manages a production multi-tenant infrastructure that is
also quite significant, using many of the same components that are in
the open source Chef.

Happy to help - hook us up with the logs.

Best,
Adam

-- 
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: