[chef] Re: Re: deployment and rollback of cookbooks, roles, environments and data bags

[Jens Skott < >]

We have a setup like this:

Core cookbooks - core applications like ntp, monitoring, rsyslog and such (maintained by chef operators)

Library cookbooks - base application cookbooks like httpd, tomcat, java and such (maintained by chef operators)

Application cookbooks - tweaks and config for the library cookbooks (maintained by application owners) 

Then we use roles based on application cookbook + library cookbook that is first branched out into a "feature branch" of our git dev branch. There we test the role in a vagrant environment and see that everything is ok, the one developing into the cookbook then submits a pull request to the chef ops and he + 1 reviewer merges it into dev where its can be further tested on a real staging server with the help of a chef solo run. If all checks out ok its merged into master and pushed out to prod. 

With the help of bamboo and the branching feature along with tags and scripts the stage and prod parts are automated with the exception of a jira status change =) 

And so far we have had no bad code submitted into prod, with about 30 application cookbook commiters and 5 library and core cookbook comitters internaly. 

Dont know if its the best way but keeps our production servers running at without any faults. 

Jens Skott 

Tel: +46-8-5142 4396

Schibsted Centralen IT

On Mon, May 20, 2013 at 8:09 PM, Morgan Blackthorne < " target="_blank"> > wrote:

It seems to me that that's likely where environment based versioning might come in handy. Assuming the last run succeeded, you would want to freeze an environment (let's call it testing-old) at the last set of cookbooks uploaded before you start uploading more (you could determine if the previous run succeeded by checking whether or not a second environment exists, testing). You then upload more. If any of them fail, you abort out and throw an error (making sure the testing environment is gone). If you succeed in uploading them all, then you create the testing environment, freezing all versions at the new current. You then try these cookbooks out in your cluster on testing nodes, and assuming things go fine, you can then bless that version as dev/prod/whathaveyou.

Just because the cookbook uploaded doesn't mean that everything's copacetic. It just means that the syntax passed. There's a certain amount of automatic testing that will help, but beyond that point, you simply need to work with real world data to see whether or not the code is working right. Automated testing rarely catches edge-case scenarios because the people writing the testing usually don't imagine them-- or they would have been thought about when the code was being developed in the first place.

--

~*~ StormeRider ~*~

"Every world needs its heroes [...] They inspire us to be better than we are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Mon, May 20, 2013 at 10:29 AM, Kirill Timofeev < " target="_blank"> > wrote:

Folks,

currently we have all chef stuff (cookbooks) in git repository. Post-commit hook triggers jenkins job, which updates chef using slightly modified version of chef-jenkins synchronization tool. Unfortunately this approach allows submission of broken code, which would be revealed only while pushing changes to chef-server. I want to modify this approach by utilizing update hook and accepting changes only if they would be successfully uploaded to chef server. But in order to do this I also need some rollback mechanism on chef-server side to discard update in case of any issues.

So my question is: do we have in chef server 11.08 any built in mechanism for transactional updates or I need to implement some custom solution?

Thanks,
Kirill.