[chef] Re: Re: Delayed actions don't run if a recipe fails


Chronological Thread 
  • From: Noah Kantrowitz < >
  • To:
  • Subject: [chef] Re: Re: Delayed actions don't run if a recipe fails
  • Date: Mon, 22 Nov 2010 23:25:38 -0800


On Nov 22, 2010, at 7:13 PM, Daniel DeLeo wrote:

On Mon, Nov 22, 2010 at 7:25 PM, Michael Rowe < > wrote:
Hi folks,

I am currently bumping up against the issue described here:
http://tickets.opscode.com/browse/CHEF-1067

In a nutshell:

* a recipe sets up some file and notifies a service it needs to restart
 * the restart is delayed
 * further things happen...
 * a recipe fails
 * the chef run ends, without running the delayed actions

On the next chef-client run, the original change is already in place,
so the service isn't notified to restart again. But it never actually
got restarted because the previous run ended prematurely. This is a
Bad Thing.

The last update to that issue was in July, saying "maybe discuss on
the list"... but no one seemed to at the time, so here I am! :)

Thoughts?

It's a tough situation, services are unique in that they have state
in-memory that's difficult/impossible for chef to query (unless you
somehow instrument your services).

The specific problem with delayed notifications is that you may have
depended on some later resource in your resource collection to modify
some other bit of configuration relevant to that service. So in the
case of a failure, you'd prefer to fix everything by hand rather than
restarting your service in a broken state.

To get around this issue, you might consider keeping some state in a
file on disk that marks the service as "dirty" and add some mechanism
to the service resource to trigger a restart if the service is marked
dirty. The problem with this is that you might erroneously restart the
service if an administrator has manually resolved the issue in between
the failed chef run and the next one. Even so, it may be possible on
some systems to work back to the last time the service was restarted
using information from something like `ps -o etime` and detect if the
service has been manually restarted.

That all said, it's definitely not a general solution, so delayed
notifications to execute resources would still be a problem. The best
you could do there as far as I can figure is print the notifications
that did not execute.

I haven't thought all of this through completely, so if there's some
problem with the solution I described, I'd like to hear it.

Chef could internally do something similar. The client could note in a state file that there is a pending notification and wait until the end of the next successful cycle to execute it. There isn't a whole lot of state attached to notifications, so a simple JSON file would probably suffice. This still isn't quite bullet-proof if the client itself crashes, but it would cover the 99% case of bad recipes.

--Noah



Archive powered by MHonArc 2.6.16.

§