[chef] Re: Re: Delayed actions don't run if a recipe fails

[Noah Kantrowitz < >]

On Nov 22, 2010, at 7:13 PM, Daniel DeLeo wrote:

> On Mon, Nov 22, 2010 at 7:25 PM, Michael Rowe <
>  >  
> wrote:
> > Hi folks,
> >
> > I am currently bumping up against the issue described here:
> > http://tickets.opscode.com/browse/CHEF-1067
> >
> > In a nutshell:
> >
> >  * a recipe sets up some file and notifies a service it needs to  
> > restart
> >  * the restart is delayed
> >  * further things happen...
> >  * a recipe fails
> >  * the chef run ends, without running the delayed actions
> >
> > On the next chef-client run, the original change is already in place,
> > so the service isn't notified to restart again. But it never actually
> > got restarted because the previous run ended prematurely. This is a
> > Bad Thing.
> >
> > The last update to that issue was in July, saying "maybe discuss on
> > the list"... but no one seemed to at the time, so here I am! :)
> >
> > Thoughts?
>
> It's a tough situation, services are unique in that they have state
> in-memory that's difficult/impossible for chef to query (unless you
> somehow instrument your services).
>
> The specific problem with delayed notifications is that you may have
> depended on some later resource in your resource collection to modify
> some other bit of configuration relevant to that service. So in the
> case of a failure, you'd prefer to fix everything by hand rather than
> restarting your service in a broken state.
>
> To get around this issue, you might consider keeping some state in a
> file on disk that marks the service as "dirty" and add some mechanism
> to the service resource to trigger a restart if the service is marked
> dirty. The problem with this is that you might erroneously restart the
> service if an administrator has manually resolved the issue in between
> the failed chef run and the next one. Even so, it may be possible on
> some systems to work back to the last time the service was restarted
> using information from something like `ps -o etime` and detect if the
> service has been manually restarted.
>
> That all said, it's definitely not a general solution, so delayed
> notifications to execute resources would still be a problem. The best
> you could do there as far as I can figure is print the notifications
> that did not execute.
>
> I haven't thought all of this through completely, so if there's some
> problem with the solution I described, I'd like to hear it.

Chef could internally do something similar. The client could note in a  
state file that there is a pending notification and wait until the end  
of the next successful cycle to execute it. There isn't a whole lot of  
state attached to notifications, so a simple JSON file would probably  
suffice. This still isn't quite bullet-proof if the client itself  
crashes, but it would cover the 99% case of bad recipes.

--Noah