[chef] Re: Re: Re: Re: Re: Monitoring chef runs

[KC Braunschweig < >]

> On 9/6/12 7:52 AM, Tetsu Soh wrote:
> well, it really depends on how you manage operations carried out by chef.

In the end you have to sit down and come up with a workflow and risk
model that fits your business and they're all somewhat different. A
few other things to consider besides the info above:

- Just because you run chef all the time across your whole
infrastructure doesn't mean you have to push changes to your whole
infrastructure all the time. Consider designing your change control
process around the the environments and/or chef servers you're pushing
to. In the past I've seen things like: dev/qa always gets new
cookbooks all the time, prod used a separate server/organization so a
separate push was required to migrate into prod which was done in a
scheduled window but those windows were easy to come by as long as
your change was targeted at a specific (usually out of service)
cluster (modeled with environments). It was a somewhat complex flow
but it was clear, let people move fast and kept chef running
everywhere every 30 minutes. This is good because chef could still
remediate any out-of-band changes it finds very quickly while new
changes go out in a slower and more controlled way.

- Another reason to keep chef running all the time is that you're less
likely to fight with it. If it takes minimum 24 hours to push a change
with chef (example above) and you need to do something faster (OMG
prod is on fire!), you're not going to use chef, you're going to do
something else (ssh for loop?). Maybe you'll remember to backport your
change into chef. Maybe when you backport it into chef it'll actually
come out compatible rather than just slightly not the same. Maybe not,
and every time that fails everyone who doesn't know better blames chef
for undoing their productive work. Now they're working against the
tool instead of the tool making their lives better.

KC