[chef] Re: Re: Re: Re: Re: Re: Re: Re: More on Cookbook Design Patterns

[Jamie Winsor < >]

I still completely suggest to set constraints in metadata, it's just not easy to set equality constraints in every single cookbook and their dependencies. You'll need to ensure that all of your dependencies and their dependencies (and so on) are listed in the metadata.

That's what the Berksfile.lock is for. It will do that for you.

Jamie Winsor

@resetexistence

On Jun 1, 2014, at 12:59 PM, Torben Knerr < "> > wrote:

Hey Jamie,

agree that reeducating with a different name might cause confusion.

I think what it would gain is that people who prefer locking their dependencies in metadata.rb rather than environments can use the same name, because its 90% the same pattern (10% being locking via environments).

But maybe there are not so many people using that approach anyway, who knows...

If nobody jumps in, I'm probably the only one ;-)

Cheers,

Torben

On Sun, Jun 1, 2014 at 7:20 PM, Jamie Winsor < " target="_blank"> > wrote:

Hey Torben,

The name isn't really all that important as much as the consistency of use. I wrote that blog post to attempt to solve that very problem.

I chose the name Environment Cookbook because it is "applied" to a Chef Environment and makes a sort of "container" for your nodes to live in. While the word *environment* can be rather overloaded in software development and operations, I still think that this is the appropriate name for the pattern.

What would using a more generic name gain for us? I think at this point it would cause more confusion to try and reeducate with a different name. There would need to be substantial and clear advantages for this to be a good choice. 

Jamie Winsor

@resetexistence

On Jun 1, 2014, at 7:43 AM, Torben Knerr < " target="_blank"> > wrote:

Hi Marco,

yes, that was great work by Jamie summing everything up so nicely!

The "environment cookbook" pattern is indeed exactly what I call "top-level cookbook", and I'm not a fan of re-inventing the wheel or new names :-) 

@Jamie: what do you think? Would renaming the "environment cookbook" pattern be desireable? Just a more generic name that does not imply _how_ the dependencies are locked (environments vs. metadata.rb), but something that better describes the true nature of the pattern. Any ideas?

Cheers,

Torben

On Wed, May 28, 2014 at 5:59 PM, Marco Betti < " target="_blank"> > wrote:

Hi Torben,
we are following Jamie's pattern evolution and, by the way, we like it very much.
I just wanted to point out that in the "environment cookbook pattern" myface cookbook is the real environment cookbook and not an application cookbook, even if "An Environment Cookbook is nearly identical to an Application Cookbook in structure with one seemingly small, but crucial difference: The Environment Cookbook is the only cookbook which should have it's Berksfile.lock committed into version control."
So while the structure is "nearly identical" the meaning is very different and is very like to what you call "top-level cookbook".
I personally agree that naming switch from "application cookbook pattern" (The Berkshelf way) to the new "environment cookbook pattern" has been somehow confusing at first, so was it for us at least...

It would be useful if everybody converged on the same naming, but probably often many of us use different names to refer to the same pattern, probably because everybody sees the same thing from a slightly different point of view, that is his own personal point of view.

Greetings
Marco

On Wed, May 28, 2014 at 10:58 AM, Torben Knerr < " target="_blank"> > wrote:

Hi Jay,

just be careful, the names of the cookbook patterns are heavily overloaded, especially regarding the term "application cookbook".

I mostly agree with Tom [0] and your notions of the "application cookbook", because that's what the original "myface" example [1] suggested. Eric Reeves described this notion of "application cookbook" [2] also quite well some time ago.

However, Jamie recently blogged about the different kinds of cookbooks as he meant it [3], which puts a slightly different view on "application cookbooks". In this view, cookbooks-mysql or cookbooks-apache2 are application cookbooks as well (what you call infrastructure cookbooks). But also, myface would be an application cookbook as well in this view (it's just composed of other application cookbooks).

Acknowledging that Jamie probably coined the term and so he could define or refine it as he wants, I accepted that the mysql-cookbook now is an "application cookbook" to many people and that we probably don't share the same meaning.

In order to not further drive the confusion I refrained from calling them "application cookbooks" and now call them "top-level cookbooks". For me the essential distinction of a "top-level cookbook" is:

* it's the only cookbook on your run-list (thus "top-level")

* it's a versionable replacement for roles

* it includes all kinds of other cookbooks (library, wrapper, application, base)

* it's so unique to your project that it's not a reusable artifact

* it should lock all it's dependencies (including transitive ones!) when released to ensure repeatable deployments

* it's the interface to the people who want to install it so it should document all relevant stuff in the README (e.g. "exposed" attributes vs. "internal" attributes)

Damn I finally need to blog about this...

Cheers,

Torben

[0] https://tomduffield.com/everything-as-a-cookbook-devops-minneapolis-meetup/

[1] http://vialstudios.com/guide-authoring-cookbooks.html

[2] http://alluvium.com/blog/2013/05/03/the-application-cookbook-pattern-berkshelf-and-team-chef-workflow/

[3] http://blog.vialstudios.com/the-environment-cookbook-pattern/

On Mon, Apr 7, 2014 at 5:54 PM, Lamont Granquist < " target="_blank"> > wrote:

I'd start with no version constraints in cookbooks and then pin with equality in environments.  You might use ~> 1.0 semver-assuming constraints for external cookbooks, but if you're testing your cookbooks you should be able to pull in major version bumps of upstream cookbooks and if the tests pass, then you weren't affected by the major version bump so you should be able to ship it, so why be fussy and create yourself more work managing the major version bump manually?  The more you go towards continuous deployment/delivery, the less you should worry about version numbers and the more you should lean on your tests -- of course without good testing, then the argument runs the other direction and you probably should be more defensive.

On 4/4/14, 12:43 PM, Jay Perry wrote:

Tom/Lamont,

Got another question regarding version dependencies with the infrastructure, platform, application cookbook pattern.  I plan to lock application cookbooks at the environment file level and then the cookbooks locking like so in their metadata file:

Application Cookbook

- use the optimistic operator for all cookbooks it depends on, for example ~> 1.0.0

Platform Cookbook

- this could either use the optimistic operator or no version constraint at all

Infrastructure Cookbook

- no version constraint

My question is do you see this falling apart?  If I had a wrapper cookbook say "acme-java" should that lock to a specific version of the "java" cookbook?  Avoiding version constraint conflicts can be hard and I want things to be loosly locked but if they need to be locked only at the bug fix level.  I feel the application cookbook must be locked at the environment file level.  Maybe this is where I struggle when the base role cookbook isn't tested with the application cookbook since if the base cookbook depends on specific versions then things conflict.  Thoughts on this?

Thanks,

Jay

On Thu, Apr 3, 2014 at 7:48 PM, Jay Perry < " target="_blank"> > wrote:

Thanks Lamont, this discussion has been enlightening to me.  I appreciate what you and Tom have provided for information, it has been very helpful.  I hope at a later time I can write something up about our experience transitioning from what we have to a more modular design capturing the challenges and decisions we made along the way.  Again thanks for the help.  Hopefully we can have more discussions in the future.

-- Jay

> On Apr 3, 2014, at 5:38 PM, Lamont Granquist < " target="_blank"> > wrote:
>
>> On Thu Apr  3 14:17:56 2014, Jay Perry wrote:
>> Okay, so is the acme_tomcat_server cookbook app specific or a platform cookbook that can be shared?  If myapp need special configuration or extra setup very specific to myapp can that go in myapp::default?  I guess in your example is the myapp cookbook serving a dual role.  It manages what the role would have done and it handles app specific settings and any specialized setup?  For example say "myapp" required a special mongo setup.
>>
>> -- Jay
>>
>>>> On Apr 3, 2014, at 5:03 PM, Lamont Granquist < " target="_blank"> > wrote:
>>>>
>>>> On Thu Apr  3 13:38:12 2014, Jason Perry wrote:
>>>> What I'm also following is that it's probably not a good idea to combine the application cookbook with a role cookbook.  Currently our application cookbooks have replaced the roles runlist setting where previously the role would have a run list that was something like "recipe[acme_java],recipe[tomcat],recipe[groovy],recipe[myapp]" with the role being called "myapp".  Now the myapp default recipe has a bunch of include_recipe calls and the role now just has "recipe[myapp]".  When the node is bootstrapped the run list ends up looking like so "role[datacenterA],role[base],role[myapp]".  In this model I would have to move whatever the base role is doing down to the myapp::default recipe level or parts of it for testing.  I think this is probably the most tricky part of  it all and how to avoid too many dependencies with still being able to test the cookbooks purpose.
>>>>
>>>>   -- Jay
>>>
>>> i'd suggest that all roles do is map 1-1 with role cookbooks.  then i'd suggest that recipe[myapp] be pretty thin and it would just setup some attributes to point at the instance of the software to deploy and then include_recipe something like "acme_tomcat_server".  the acme_tomcat_server recipe would be more complicated and would express all the dependencies including pulling in the your java cookbook, and the actual tomcat cookbook along with monitoring and whatnot.  you might want to have this export an LWRP that your myapp cookbook could use rather than have it be attribute-driven.
>>>
>>> then if installing java requires setting up an internal corporate yum repo mirror then you would want to break that out into its own recipe and include_recipe it there in the java cookbook.  you might also include_recipe this from the base role cookbook as well, but that's fine.  chef will figure that out and the yum repo will get setup before anything that depends on it.   if you're doing your testing correctly and expressing your dependencies correctly then you shouldn't have any issues with this.  myapp will pick up the tomcat_server cookbook which will pickup the java cookbook which will pickup your yum repo cookbook (even if your base role comes after your role cookbook it should still work out fine so long as you've tested your myapp cookbook in isolation and it has passed -- although generally i'd suggest that base comes first so that your user accounts get setup early so you can login to servers and troubleshoot -- but the further down the test-driven-infrastructure road yo
> u get, hopefully the less likely this is a real concern).
>
> as long as the special configuration is really special to myapp then i'd put it directly in myapp::default.  once you repeat yourself then you should probably push that into your acme_tomcat_server cookbook.  ideally all your tomcat servers use acme_tomcat_server::default.  as you create slightly different common sub-flavors of tomcat servers you'll create additional complexity in that cookbook.  you might switch behavior based on node attributes, based on LWRP attributes, you might have different versioned 'branches' (1.x vs 2.x) of the cookbook, you might have different flavors of recipes, and you might even bust out two acme_tomcat_server_v1 and acme_tomcat_server_v2 cookbooks.  all of those different approaches to handling conditional logic have their strengths and weaknesses but you probably want to prefer solutions closer to the beginning of that list than the end.
>
> i would say that the myapp cookbook is a role cookbook, but i don't really care about the name you give the thing.  it is a cookbook that expresses what is different about myapp from your other apps (database credentials, github repo, artifacts to install, special magic sauce that only gets applied to myapp).  then you want another cookbook that expresses what is the same between all your different tomcat apps (which probably exports some LWRPs and may have a default recipe for common setup).  you write the latter to look more like a library cookbook pattern where its intended to be reused.  the myapp cookbook isn't intended to be reused at all.
>

--
Ing. Marco Betti
RHCE RHEL4 id 804006512121056