[chef] Thinking out loud

[Graeme Mathieson <mathie@woss.name>]

Hi,

I've been messing with Chef for the past week and this morning I wrote  
out some notes on our private wiki about things I'd like to enhance/ 
do. This is totally off my head and some of it is almost certainly as  
a result of missing out key bits of understanding.  After writing it,  
I figured, why not share it, so here goes:

* Some sort of rspec/test-driven mechanism. At the very least, this  
should properly sanity check all the cookbooks, looking for typo'd  
attribute hashes, missing recipes and logic errors. I'm thinking that  
this could be built fairly simply just by stubbing parts of the  
framework and running each of the recipes. Second goal: get autospec  
working for it. Third goal: rspec matchers suitable for chef.

* Remove the needless verbosity of the attribute definitions, like  
creating new Mashes, and only defining attributes unless they already  
exist. It's just messy.

* Remove the duplication in the metadata.  While enhancing the  
attribute definitions, tweak them so that they include enough  
information to allow the metadata to be automatically generated.  
Dependencies can be calculated instead of specifying them manually.

* Allow cookbooks to be distributed as gems with gem dependencies on  
other cookbooks.

* Allow vendoring of these gems, because I don't like relying on  
system gems, even in this sort of environment. Bootstrapping is key.

* Get rubigen to have generators for an empty repo, individual  
cookbooks, that kind of thing.  In general, make the repo  
infrastructure feel more like a Rails app, just because that's a good  
example of an app development environment.  Individual recipes can be  
generated with: {{script/generate recipe ruby/gems}} which would  
automatically create the 'ruby' cookbook too, if it doesn't already  
exist.

* A thin bootstrap tarball of chef solo that doesn't rely on anything  
other than ruby being installed on the client system. No rubygems,  
nowt. That way, we can handle everything else through chef and have  
real bare-bones system images.

* Some sort of external, persistent information about nodes as well as  
roles in the repo. I kinda want the chef server to be easily  
replaceable, and everything about our infrastructure to be in version  
control. I can't quite see how to do that with nodes yet, but perhaps  
I'm just missing something.

* Bootstrapping a node from a pristine image, pulling all its  
information from an external system. I'm (so far) not liking the idea  
of creating a node by passing in some JSON when it boots. It should  
boot then get that information from a persistent store so that, next  
time it's imaged from afresh, it can start back where it left off  
without me having to separately document the JSON is was passed in on  
bootup. I think this is a repetition of the previous point. :-)

* The equivalent of dpkg's "Provides" for providing alternative  
implementations of packages. Say, for example, I have a  
ruby_enterprise_edition cookbook, which {{provides "ruby"}}. If  
another cookbook says {{include_recipe "ruby"}} and we've already seen  
ruby_enterprise_edition, it will satisfy that dependency.

h3. Cookbooks I want

* Percona's build of MySQL [http://www.percona.com/mysql/5.0.82-b16/ 
deb/]
* RabbitMQ
* ActiveMQ (until I make sure I can port our apps that use ActiveMQ  
across to a single message bus...)
* Confluence
* Bamboo
* Ruby Enterprise Edition with dpkg-diverts to neuter the existing  
ruby package, while leaving it around to satisfy dependencies...
* Variants of all the existing recipes that don't use runit! (What can  
I say? I like consistency, and having a mixture of init.d and runit  
services isn't consistent. And out of band monitoring with God is  
easier than shifting *everything* across to runit...)
* Nanite infrastructure.

h3. Stuff I want

This is stuff that I want to implement, but I'm not yet sure whether  
it's best done with chef, nanite or something else entirely.

* Rolling out software updates (in particular, package security  
updates) in a timely manner across our infrastructure, after having  
tested on our staging infrastructure.
* Upgrading an EC2 instance to a bigger instance type. Automagically.
* An overarching DSL for describing our infrastructure, from which all  
the node stuff is generated and deployed. What I'd *like* to do is  
"draw" our infrastructure in Graphviz dot notation and for that to  
turn into physical, working, connected infrastructure. That's my  
nearly-end-goal.

The end goal is to be able to sketch out these diagrams while relaxing  
on a beach in Malibu and hand them to a minion who transcribes them to  
dot notation and for that to turn into physical, working, connected  
infrastructure. :-)

Cheers,

Graeme.