Re: storing easily queried metadata - was Re: some questions

[snacktime <snacktime@gmail.com>]

On Thu, Apr 23, 2009 at 9:46 PM, David Lee <david.lee@kanji.com.au> wrote:

Actually, before I go offering any more opinions, can I please ask what we're actually doing? In reading the responses, I realised I don't know enough about what's going on yet to offer very constructive input.

*Please* correct me if i'm wrong, but it seems like we're basically wanting to:

1) store node data records, which are a fairly large, arbitrarily structured nested hash (ruby / json), with string key/value pairs

2) find node records which match some very simple criteria, and return the entire node data structure for matches

3) store some additional metadata about nodes themselves, recipes, and other Chef classes / objects; these bits of metadata would be lightweight and possibly act like polymorphically associated ActiveRecord objects.

Is this a reasonable summary?

I think that eventually you will want to attach attributes to a variety of chef objects, such as cookbooks, recipes, etc..  

If it is, would a decent native Ruby object database be a pretty reasonable thing to use as a backend? Does such a thing exist?

Not that I know of.

The problem with multiple backend stores is the lowest common denominator.  If you are going to mix sql/non sql backends, you need a reference implementation to go by.  It wouldn't be much work to drop in activerecord, but the minute I start using activerecord functionality that doesn't exist in the other backends, I've just broken chef for everyone not using activerecord.

Off the top of my head, I would probably pick one of the full featured orm's as a reference, and then define a subset of that orm's functionality that can be used in chef core.  The goal being to use a reference implementation that has enough functionality to carry you forward.  I suspect activerecord, datamapper, or couchrest would be good candidates.

The only other approach I can think of is to use couchdb OR sql, but not both, and then just pick the best orm and stick with it.

Since chef already uses couchdb, I think just using the best couchdb orm is the saner approach.  Multiple backends of completely different types will lead to chaos.

If others care to chime in and come to some consensus, I'd be happy to start coding it up. 

Chris