[chef-dev] Re: CHEF-3747 - Compile Time Packages

[Chris Roberts < >]

Hi! I have some work in progress to update this pull request to be more  
generic rather than package specific. The discussion we had in the code  
review made me rethink this a bit about the impact it has on resources at  
large, not just the specific resources where I was running into it. That  
said, I feel the current ChefPackage cookbook (which the pull request  
originated from) provides a good example of the problem as well as one  
approach to the solution.

So, to break the problem down, lets say I have the postgresql::ruby recipe  
in my run list because I need the pg gem for some of my other recipes. The  
postgresql::ruby recipe will then install the packages it requires during  
the compile phase so the pg gem can be properly built. Up to this point,  
everybody is happy.

Now, lets say in some custom recipe, I have something like this:

package 'postgresql-client' do
 notifies :create, 'ruby_block[log pg install]', :immediately
end

ruby_block 'log pg install' do
 block do
   Chef::Log.warn "*** Postgresql Client has been installed!"
 end
 action :nothing
end

What happens here is that postgresql::ruby breaks the expected behavior.  
Since the postgresql-client package was installed during compile time, no  
notifications will be processed, and the ruby_block resource will never be  
executed. I was experiencing this issue explicitly with package resources,  
which is why I whipped up the chef_package cookbook. However, as was pointed  
out in the code review, and as I have been starting to run into in some  
recent work, this issue affects more than just package resources. It affects  
all resources and their notifications.

In the cookbook, and the current pull request, I solved this for packages by  
creating ChefPackage resources for all Package resources. When a ChefPackage is  
discovered in the compile phase, a corresponding Package resource would be  
created based on the ChefPackage resource (ensuring notifications were in  
tact) and added to the resource collection. These resources would be  
processed regularly by the runner, but instead of attempting to run the action,  
it would simply check the resource if it had been updated by the last action  
and if so, apply notifications.

I am currently updating the pull request to provide this behavior for all  
resources by using a flag to mark it as having been updated during the  
compile phase, so the runner will know its status and if the notifications should  
be processed. One of the big questions is is that the expected behavior. In the  
code review, it was mentioned that immediate notifications might imply that  
the notified resource be executed during the compile phase. I don't feel  
that would be the expected behavior and don't think it _should_ be the  
behavior. However, with regards to expectations, I think having it defined  
in the documentation would be enough to make it clear.

The change that I am proposing is flagging resources that are run during  
compile time. This allows the updated_by_last_action value to be propogated  
to the execution phase. It also allows the runner to know the resource's  
action should not be attempted, only the notifications processed if its  
updated_by_last_action? is returning true.

While I would be really happy with built in tools for adding resources to  
arbitrary points in the resource collection, I don't see it providing a  
solution for cases like postgresql::ruby where recipes will be depending on  
the pg gem during the compile phase rather than the execution phase.

So, this change is targeting a very edge case, since compile time resource  
execution doesn't (and shouldn't) happen that often. But I feel that it's  
important for things to behave as expected when that edge case is  
encountered.

Thanks!

- Chris

Daniel DeLeo writes:

> « HTML content follows »
>
> Hi Chefs,
>
>
> Recently we've been reviewing CHEF-3747 and wanted to get outside input.
>
>
> http://tickets.opscode.com/browse/CHEF-3747
>
>
> In short, this adds a shortcut for installing a package at compile time, but  
> it has an interesting twist: It creates a corresponding package resource that  
> will trigger notifications at converge time where the resource "would have  
> been" if it had been a converge-time resource.
>
>
> We wanted to hear more from Chris (the patch's author) about what exactly  
> he's using this behavior to do. As I understand it, there are two  
> constraints:
>
>
> 1. These packages are prerequisites for gems that are installed and loaded  
> via chef_gem;
> 2. When installed, additional resources need to run via notifications.
>
>
> We also want to hear from you. This behavior seems quite complex; is it  
> something you think you would use? Is the complexity worth the benefit?
>
>
> I've also come across some discussions arguing that moving resources to  
> compile time is an anti pattern. Personally I think its fine for one-off  
> hacks, but I'm wary of baking it deeper into core Chef. Considering that the  
> compile/converge distinction is pretty much required to implement  
> notifications, those obviously act weirdly when everything gets moved to  
> compile time. I personally also think Chef's strict ordering of run list  
> items and the resource collection is the best way to get changes made in the  
> desired order. On the other hand, this may necessitate splitting cookbooks  
> into smaller pieces and sprinkling them across the run_list, which can be  
> awkward.
>
>
> One alternative we've been discussing here at Opscode is a refactor of the  
> way that the recipe DSL is implemented such that you'd be able to prepend  
> resources to the resource collection, or insert them at an arbitrary point,  
> but this isn't something that would be implemented soon.
>
>
> Thoughts?
>
>
>
> -- 
> Daniel DeLeo