[chef] Re: Re: Re: Delayed actions don't run if a recipe fails

[Peter Struijk < >]

On Mon, Nov 22, 2010 at 11:25 PM, Noah Kantrowitz <
 
 ">
 > wrote:

On Nov 22, 2010, at 7:13 PM, Daniel DeLeo wrote:

On Mon, Nov 22, 2010 at 7:25 PM, Michael Rowe < " target="_blank"> > wrote:

Hi folks,

I am currently bumping up against the issue described here:
http://tickets.opscode.com/browse/CHEF-1067

In a nutshell:

 * a recipe sets up some file and notifies a service it needs to restart
 * the restart is delayed
 * further things happen...
 * a recipe fails
 * the chef run ends, without running the delayed actions

On the next chef-client run, the original change is already in place,
so the service isn't notified to restart again. But it never actually
got restarted because the previous run ended prematurely. This is a
Bad Thing.

The last update to that issue was in July, saying "maybe discuss on
the list"... but no one seemed to at the time, so here I am! :)

Thoughts?

It's a tough situation, services are unique in that they have state
in-memory that's difficult/impossible for chef to query (unless you
somehow instrument your services).

The specific problem with delayed notifications is that you may have
depended on some later resource in your resource collection to modify
some other bit of configuration relevant to that service. So in the
case of a failure, you'd prefer to fix everything by hand rather than
restarting your service in a broken state.

To get around this issue, you might consider keeping some state in a
file on disk that marks the service as "dirty" and add some mechanism
to the service resource to trigger a restart if the service is marked
dirty. The problem with this is that you might erroneously restart the
service if an administrator has manually resolved the issue in between
the failed chef run and the next one. Even so, it may be possible on
some systems to work back to the last time the service was restarted
using information from something like `ps -o etime` and detect if the
service has been manually restarted.

That all said, it's definitely not a general solution, so delayed
notifications to execute resources would still be a problem. The best
you could do there as far as I can figure is print the notifications
that did not execute.

I haven't thought all of this through completely, so if there's some
problem with the solution I described, I'd like to hear it.

Chef could internally do something similar. The client could note in a state file that there is a pending notification and wait until the end of the next successful cycle to execute it. There isn't a whole lot of state attached to notifications, so a simple JSON file would probably suffice. This still isn't quite bullet-proof if the client itself crashes, but it would cover the 99% case of bad recipes.

--Noah

My two cents..

Chef-client should probably execute all pending delayed notifications by default, even if there was an exception during the run. To me this behavior would be preferable and most of the time it would be the correct thing to do anyway. If it breaks a service because of inconsistent state, I can live with that. You could even make this behavior optional and be able to explicitly enable/disable it per notification. e.g. add a :persistent option to notifies.  
This also has the benefit of attempting to resolve it within a single chef run. I do not like the idea of trying to preserve state between two (or more) runs.

--Peter