[chef] Re: Getting off of couchdb?

[Noah Kantrowitz < >]

On Feb 11, 2012, at 8:48 AM, Ian Marlier wrote:

> Hey, Chefs --
> 
> As we've rolled out Chef to larger and larger sites, I've discovered that 
> CouchDB is a significant scaling problem.  The time to compact the database 
> and views has risen (and risen, and risen), and search performance has 
> declined.  I've heard rumblings that Hosted Chef is no longer using Couch 
> as it's backend, which suggests that Opscode might have hit this pain as 
> well.
> 
> There's obviously some tuning that we could do to Couch which would make it 
> behave better, but I need to scale it to approx. 40x what it's currently 
> doing in order to roll out Chef to the remaining sites in my environment, 
> and I'm extremely skeptical that anything -- even the most aggressive 
> tuning -- will make that work.
> 
> I'm wondering a few things:
>   - Are there existing branches of Chef (within Opscode or otherwise) that 
> aim to replace CouchDB with another database (be it a document-based store 
> or something more relational)?
>   - Are any of those branches public, or are there plans to roll those 
> changes into the open source version?
>   - If those branches don't exist, how much interest is there on the list 
> in removing the dependency on CouchDB?  Would other people be interested in 
> participating in a project like that?

The short version; yes we are moving the backend from Sinatra (Ruby) + 
CouchDB to WebMachine (Erlang) + SQL (MySQL and Postgres specifically). No, 
this code isn't public yet because the migration is ongoing and it isn't 
really the kind of thing that we want to support others running right now. We 
are switch over API endpoints as they are written and swapping data into SQL 
usually as the endpoint is rewritten in Erlang, so right now it requires a 
lot of magic in the load balancers to keep everything flowing smoothly. The 
general goal is to try and open-source it once it is fully converted and in a 
situation where other people could run it without going insane, but there 
aren't specific plans or timelines yet. Wish I could give you something 
firmer, but hopefully that sheds some light on the situation.

--Noah

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail