[chef] Re: Re: Re: Re: deployment and rollback of cookbooks, roles, environments and data bags

[Ranjib Dey < >]

i agree that an ability to treat the entire cookbooks upload (along with databags and roles) will be awesome, and simplify lot of tooling, i am not sure how much effort it will take to do something like that ..

we currently follow pretty much the same workflow as you have described, but we do use a staging chef server. we also do a cleanse before the upload (remove all cookbooks, databags etc), this ensures we are not carrying over any artifacts, and that we can completely restore an older state (including the clients). This is pretty much like implementing the transaction logic at your end.

also we use jenkins , along with the ghprb plugin (github pull request builder), which test  individual PRs . We dont test all the feature branches, only the PRs. the ghprb plugin lets you retest, bypass PRs using comments to.

till now this is working, and solving the problem we wanted to address, but its time consuming (chef zero might be helpful on this), but it also opened up a can of other issues related to jenkins automation.. but thats different

ranjib

On Mon, May 20, 2013 at 12:35 PM, Kirill Timofeev < " target="_blank"> > wrote:

Thanks a lot for sharing your thoughts Jens and Morgan!

I tried to implement approach with testing and master branches in git where all pushes to testing are automatically uploaded to testing chef server and all pushes to master are automatically uploaded to production chef server. This approach appeared to be inconvenient since it required frequent merges between branches. Also if 2 people worked simultaneously in testing and one of them was integrating his changes into master he could accidentally merge changes of other person, which may not be ready for production yet.

So I decided to switch to approach where chef development can be done in feature branches, updates are delivered to testing chef server via knife and when everything works as expected feature branch is being integrated into master branch and changes automatically propagate to production chef server. But I want to introduce protection against changes that were not tested on testing chef server and to reject pushes that failed to upload to production chef server. I understand, that this doesn't guarantee that successfully uploaded cook books have flawless logic. Also I can't rely on everybody using cookbook version locking, some people just use latest versions.

Considering all above I need to implement rollback mechanism if upload would fail. I can preserve old versions of roles and environments and upload them again if something went wrong. For cookbooks this is even easier. All updated cookbooks should have bumped versions so for rollback I just need to delete specific cookbook version.

But it would be really great if I can update chef server in transactional manner. By this I mean that I somehow tell chef server "update started", upload changes and after that send command "update ok" and those changes take effect simultaneously, or in case of "update failed" command all changes are discarded. Do I understand correctly, that currently chef server doesn't support such use case?

Thanks,
Kirill.

On 05/20/2013 11:23 AM, Jens Skott wrote:

We have a setup like this:

Core cookbooks - core applications like ntp, monitoring, rsyslog and such (maintained by chef operators)

Library cookbooks - base application cookbooks like httpd, tomcat, java and such (maintained by chef operators)

Application cookbooks - tweaks and config for the library cookbooks (maintained by application owners) 

Then we use roles based on application cookbook + library cookbook that is first branched out into a "feature branch" of our git dev branch. There we test the role in a vagrant environment and see that everything is ok, the one developing into the cookbook then submits a pull request to the chef ops and he + 1 reviewer merges it into dev where its can be further tested on a real staging server with the help of a chef solo run. If all checks out ok its merged into master and pushed out to prod. 

With the help of bamboo and the branching feature along with tags and scripts the stage and prod parts are automated with the exception of a jira status change =) 

And so far we have had no bad code submitted into prod, with about 30 application cookbook commiters and 5 library and core cookbook comitters internaly. 

Dont know if its the best way but keeps our production servers running at without any faults. 

Jens Skott 

Tel: +46-8-5142 4396

Schibsted Centralen IT

On Mon, May 20, 2013 at 8:09 PM, Morgan Blackthorne < " target="_blank"> > wrote:

It seems to me that that's likely where environment based versioning might come in handy. Assuming the last run succeeded, you would want to freeze an environment (let's call it testing-old) at the last set of cookbooks uploaded before you start uploading more (you could determine if the previous run succeeded by checking whether or not a second environment exists, testing). You then upload more. If any of them fail, you abort out and throw an error (making sure the testing environment is gone). If you succeed in uploading them all, then you create the testing environment, freezing all versions at the new current. You then try these cookbooks out in your cluster on testing nodes, and assuming things go fine, you can then bless that version as dev/prod/whathaveyou.

Just because the cookbook uploaded doesn't mean that everything's copacetic. It just means that the syntax passed. There's a certain amount of automatic testing that will help, but beyond that point, you simply need to work with real world data to see whether or not the code is working right. Automated testing rarely catches edge-case scenarios because the people writing the testing usually don't imagine them-- or they would have been thought about when the code was being developed in the first place.

--

~*~ StormeRider ~*~

"Every world needs its heroes [...] They inspire us to be better than we are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Mon, May 20, 2013 at 10:29 AM, Kirill Timofeev < " target="_blank"> > wrote:

Folks,

currently we have all chef stuff (cookbooks) in git repository. Post-commit hook triggers jenkins job, which updates chef using slightly modified version of chef-jenkins synchronization tool. Unfortunately this approach allows submission of broken code, which would be revealed only while pushing changes to chef-server. I want to modify this approach by utilizing update hook and accepting changes only if they would be successfully uploaded to chef server. But in order to do this I also need some rollback mechanism on chef-server side to discard update in case of any issues.

So my question is: do we have in chef server 11.08 any built in mechanism for transactional updates or I need to implement some custom solution?

Thanks,
Kirill.