[chef] Re: dry-run / no-op mode

[jon stuart < >]

Adam Jacob wrote:

[ snip ]

> 
> All of this means that, while a dry-run mode is possible, it is also
> likely to tell you lies about what might really happen.
> 

[ snip ]

> What other use-cases are there here, that aren't under-cut by the very
> real potential for lies?  Would it be enough to enable the visibility
> into how the system is really behaving that would allow you to gain
> the level of trust you need, rather than a full-on dry run mode?
> 

Hi,

Thanks for explaining your reservations about dry-run.

I'd suggest however that the caveats of dry-runs not perfectly
reflecting what happens in real runs are understood by the users of most
tools that provide such a facility, especially in the case of an
intermediate step's failure changing or halting execution.

For example, make's -n dry-run assumes every compile succeeds, and walks
the dependency graph accordingly. And, probably less so than Chef,
intermediate steps can alter the graph too beyond just success or
failure. However it's making no promises that a real build will behave
like that, but it's providing a massively useful facility: "Roughly, in
a perfect world, what would you do to build this target?"

Concerning use-cases: of particular interest to me is being able to diff
the the outputs of template generation. Not just to see how a template
change might look, but more importantly what the outcome of changing the
attributes that govern their population.

A contrived but hopefully valid example: the address of an internal web
service (a key-value store, perhaps) is being managed by Chef. It uses
this attribute to populate templates that tell other applications on the
platform where to connect for this service.

One day I decide to pair up two hosts to provide this service and expose
them via a VIP managed by some appliance, and update the attribute in
Chef accordingly. Great, the apps can survive the failure of one of the
service's hosts.

What I've forgotten is that I've also got Chef to use this attribute to
indicate where to poll SNMP data from and where overnight backups should
rsync from. These templates now point at the VIP-managing appliance
rather than the real host. Things go weird, things break.

Yes, I should've thought about that. Yes, I can discover this by looking
at what Chef did after it runs. Yes, I can fix by making a clearer
distinction in my Chef configuration between hosts and services. But
finding out how feeble my brain is *after* the change has broken stuff
isn't great.

It's possible such a situation might arise even when testing changes in
environments prior to production. I might not backup the key-value
service in staging so there was no change to notice there. (As I said,
contrived example!)

True, having staging not reflecting production is always going to hurt.
But I don't want my configuration manager to increase that hurt by
making me drink the "*everything* that works on staging works in
production" koolaid. IME, there's always some variance somewhere, even
just in the dimensions.

I'd be happy with the potential for a dry-run mode to be optimistic and
sometimes inaccurate if it helps save me from myself. Given the
precedent of dry-run modes in other complex tools I'd like to believe
it's not just my capacity for forgetfulness and oversight that makes
them useful, other people think so too :-)

Regards, jon.