[chef] Re: Delayed actions don't run if a recipe fails

[Daniel DeLeo < >]

On Mon, Nov 22, 2010 at 7:25 PM, Michael Rowe 
<
 >
 wrote:
> Hi folks,
>
> I am currently bumping up against the issue described here:
> http://tickets.opscode.com/browse/CHEF-1067
>
> In a nutshell:
>
>  * a recipe sets up some file and notifies a service it needs to restart
>  * the restart is delayed
>  * further things happen...
>  * a recipe fails
>  * the chef run ends, without running the delayed actions
>
> On the next chef-client run, the original change is already in place,
> so the service isn't notified to restart again. But it never actually
> got restarted because the previous run ended prematurely. This is a
> Bad Thing.
>
> The last update to that issue was in July, saying "maybe discuss on
> the list"... but no one seemed to at the time, so here I am! :)
>
> Thoughts?

It's a tough situation, services are unique in that they have state
in-memory that's difficult/impossible for chef to query (unless you
somehow instrument your services).

The specific problem with delayed notifications is that you may have
depended on some later resource in your resource collection to modify
some other bit of configuration relevant to that service. So in the
case of a failure, you'd prefer to fix everything by hand rather than
restarting your service in a broken state.

To get around this issue, you might consider keeping some state in a
file on disk that marks the service as "dirty" and add some mechanism
to the service resource to trigger a restart if the service is marked
dirty. The problem with this is that you might erroneously restart the
service if an administrator has manually resolved the issue in between
the failed chef run and the next one. Even so, it may be possible on
some systems to work back to the last time the service was restarted
using information from something like `ps -o etime` and detect if the
service has been manually restarted.

That all said, it's definitely not a general solution, so delayed
notifications to execute resources would still be a problem. The best
you could do there as far as I can figure is print the notifications
that did not execute.

I haven't thought all of this through completely, so if there's some
problem with the solution I described, I'd like to hear it.

Dan DeLeo

>
> --
> Michael Rowe
>