[chef-dev] Re: [chef] Re: Re: The future of the database and application cookbooks

[Noah Kantrowitz < >]

Can you point out anything specific that is incompatible with what I've said? 
The general structure of having callbacks for "create server", "attach 
slave", "create database", "create user", etc seems suitably generic to me as 
no assumption about the internals of those is made and most of the database 
DSL is just collection the params into hashes to send to the backend. If the 
end point of this is that it can only work for MySQL, Postgres, and SQL 
Server that still seems pretty useful to me at least.

--Noah

On Aug 30, 2011, at 10:12 PM, Jason J. W. Williams wrote:

> I would agree with Alex. Trying to force MySQL, Redis and Riak (as 
> examples) into the same generic DB cookbook abstraction will turn into a 
> cookbook that in theory works universally, but in practice works well for 
> no one, it seems to me. ...for the same reasons that trying to force a 
> MySQL data 
> model onto Riak will fail. These are fundamentally different systems, with 
> only superficial similarities. 
> 
> -J
> 
> Sent via iPhone
> 
> Is your email Premiere?
> 
> On Aug 30, 2011, at 22:26, Alex Soto 
> <
 >
>  wrote:
> 
>> Thanks for the opportunity to discuss, always a good thing to hear 
>> different opinions.
>> 
>> As an engineer I totally understand the desire to DRY up these common 
>> patterns, but IMO these multi-stack/generic cookbooks add complexity to 
>> the cookbook and pull in lots of dependencies I don't use, cluttering up 
>> my cookbook repo.
>> 
>> I see this being similar to the 'rails' database agnosticism of the 
>> migrations dsl.  I've only used it once, and that was due to engineering 
>> curiosity of how well it worked using sqlite locally and a 'real' db in 
>> production.  In practice, there has always been something that forced me 
>> to go outside the dsl and run db specific DDL.  So in my experience you 
>> really don't utilize this particular db agnosticism feature once you pick 
>> a stack, but it adds lot's of engineering effort for the feature 
>> implementor to cover all the variations in platforms.
>> 
>> With databases' in particular, each vendor has their peculiarities, so I 
>> see lot's of rat holes in a single cookbook that covers different 
>> databases and deployment patterns.  I understand that's the reason for 
>> providers, but I think the engineering effort is better spent getting a 
>> kick-ass db specific cookbook versus a superficial cookbook that 
>> kinda/sorta works across db's.
>> 
>> Alex
>> 
>> 
>> On Aug 30, 2011, at 7:39 PM, Noah Kantrowitz wrote:
>> 
>>> As some people have noticed from my musings on IRC and elsewhere, I have 
>>> embarked on a (probably long overdue) overhaul of the application and 
>>> database cookbooks. This is largely orthogonal (for now) with the recent 
>>> LWRPs added to the database cookbook, though in the end they would be 
>>> removed. What follows is a set of syntax ideas for where I think this 
>>> should head. Not all of this is implemented, but you can see what is in 
>>> my cookbooks branch https://github.com/coderanger/cookbooks/tree/COOK-634.
>>> 
>>> Database clusters
>>> =================
>>> 
>>> In the new LWRP hierarchy the top level resource is a database_cluster. 
>>> This defines an abstract model of a single-master-multi-slave cluster 
>>> setup. Within a cluster is one or more database_servers, each of which 
>>> defines a single type of database service (MySQL, Postgres, Redis, 
>>> Cassandra, etc) mapped on to the containing cluster. In general I would 
>>> expect most clusters to only contain a single database_server block, but 
>>> this isn't required and if you have a beefy primary box for, say, both 
>>> Postgres and Redis and wanted to have both use the same backup machine 
>>> you could put those both in the same cluster definition. Within a 
>>> database_server you can define database and users, though this isn't 
>>> needed for all types of servers and the exact implementation of what 
>>> those two constructs do is left up the backend plugin implementing the 
>>> specified server type.
>>> 
>>> With all that laying down the model, lets look at some syntax:
>>> 
>>> database_cluster "prod"
>>> master_role "prod_database_master"
>>> slave_role "prod_database_slave"
>>> 
>>> database_server do
>>> type :mysql
>>> database "prod" do
>>>   engine "innodb"
>>> end
>>> user "myuser" do
>>>   password "xxx"
>>>   grant do
>>>     select "prod"
>>>     insert "prod"
>>>     update "prod"
>>>   end
>>> end
>>> end
>>> 
>>> database_server do
>>> type :redis
>>> end
>>> end
>>> 
>>> This will create a cluster running a MySQL server with one database named 
>>> prod and a Redis server (in the case of Redis there are no specific 
>>> tuning params we need to worry about so far). This example shows a very 
>>> verbose form of the desired syntax, below is an equivalent version using 
>>> some more sugar and allowing for sane defaults:
>>> 
>>> database_cluster "prod"
>>> mysql do
>>> database do
>>>   engine "innodb"
>>> end
>>> user "myuser" do
>>>   password "xxx"
>>>   grant "prod" do
>>>     select
>>>     insert
>>>     update
>>>   end
>>> end
>>> end
>>> 
>>> redis
>>> end
>>> 
>>> You can also pass Hashes to database and user instead of blocks if you 
>>> want to feed them from an external data source (such as a data bag):
>>> 
>>> database_cluster "prod"
>>> mysql do
>>> database "prod", {:engine => "innodb"}
>>> user "myuser", {:password => "xxx", :grant => {:select => "prod, ...}}
>>> end
>>> 
>>> redis
>>> end
>>> 
>>> As mentioned before, the actual implementation can choose how to handle 
>>> databases and users. In the above examples, if you added any to the redis 
>>> section it would simply be a no-op (or maybe a runtime error?). Also as a 
>>> structural piece, database and user definitions will only ever be 
>>> processed on the database master (are there databases where this isn't 
>>> the case and slaves should create users and such too?). The individual 
>>> backends for this would live in the cookbook for that program, so the 
>>> database cookbook would only hold the core infrastructure and plumbing to 
>>> run things. The idea is that this resource block would be placed 
>>> somewhere central and assigned to all servers so they can use it for 
>>> reference (see below in the app cookbook section). Only nodes with the 
>>> master or slave role would actually do anything. As a special case if the 
>>> system detects that the master role actually doesn't exist it will run in 
>>> a degraded mode assuming a single-node cluster (so obviously the current 
>>> node is the master).
>>> 
>>> The database cookbook would also grow an associated knife-database plugin 
>>> to handle replication setup/teardown as for some databases you have to 
>>> perform an out-of-band data sync before attaching the slave.
>>> 
>>> Seen in isolation, does this seem suitable flexible to cover most needs? 
>>> I am mostly familiar with SQL databases and a few smaller NoSQL products, 
>>> so for the greater NoSQL world is this still a viable model?
>>> 
>>> 
>>> Application deployment
>>> ======================
>>> 
>>> The model for the application LWRPs is generally simpler in terms of 
>>> data, but is also more callback-driven. The top-level application 
>>> resource just contains the central data used for all applications, and 
>>> then any framework or platform specific bits are contained in 
>>> sub-resources similar to the database_server arrangement. Below is an 
>>> example of deploying the Radiant CMS:
>>> 
>>> application "radiant" do
>>> path "/srv/radiant"
>>> owner "nobody"
>>> group "nogroup"
>>> repository "git://github.com/radiant/radiant.git"
>>> revision "master"
>>> packages ["libxml2-dev", "libxslt1-dev", "libsqlite3-dev"]
>>> migrate true
>>> 
>>> rails do 
>>> gems "bundler" => nil, "RedCloth" => "4.2.3"
>>> database :mysql => "radiant_production" do
>>>   user "radiant"
>>>   reconnect true
>>>   encoding "utf8"
>>> end
>>> migration_command "bundle exec rake db:migrate"
>>> end
>>> 
>>> unicorn do
>>> port 8000
>>> end
>>> end
>>> 
>>> The global data maps very directly to fields in the existing application 
>>> data bags, except that they are not Hashes indexed by environment name. 
>>> In the rails sub-resource you see that again things mostly map to the 
>>> data bag. The biggest exception is the database section which doesn't 
>>> actually state any information about the database. Instead things are 
>>> looked up by reference to the information in the database resource. It 
>>> would still be possible to specify all information manually if you aren't 
>>> using the database cookbook.
>>> 
>>> Application backends define actions as callbacks more or less, to provide 
>>> code to execute at different points during the deploy process. The 
>>> current callbacks are :before_compile, :before_deploy, :before_migrate, 
>>> :before_symlink, :before_restart, and :after_restart. The last 4 just map 
>>> to the existing callbacks in the deploy resource system. before_compile 
>>> takes place as the first thing when the application resource executes (so 
>>> when the provider is compiled), while before_deploy takes place during 
>>> the execute phase in the provider, but after the main application folder 
>>> structure is created.
>>> 
>>> The plan is to convert all the existing application cookbook recipes into 
>>> a small wrapper just piping data bags into this LWRP structure, so they 
>>> should continue to work as they always have.
>>> 
>>> So, what do people think? Is this a step in the right direction for 
>>> application deployment? Do you like the idea but what a different syntax 
>>> for it? Are there big, glaring use cases I've missed?
>>> 
>>> --Noah Kantrowitz
>>> 
>>> 
>>> 
>>> tl;dr Application deployment with Chef is going to be awesome!
>>> 
>>

Attachment: smime.p7s
Description: S/MIME cryptographic signature