[chef-dev] Re: CHEF-3747 - Compile Time Packages

Chronological Thread 
  • From: Chris Roberts < >
  • To: Daniel DeLeo < >
  • Cc: Chef Dev < >
  • Subject: [chef-dev] Re: CHEF-3747 - Compile Time Packages
  • Date: Mon, 28 Jan 2013 17:11:14 -0800

Hi! I have some work in progress to update this pull request to be more generic rather than package specific. The discussion we had in the code review made me rethink this a bit about the impact it has on resources at large, not just the specific resources where I was running into it. That said, I feel the current ChefPackage cookbook (which the pull request originated from) provides a good example of the problem as well as one approach to the solution.

So, to break the problem down, lets say I have the postgresql::ruby recipe in my run list because I need the pg gem for some of my other recipes. The postgresql::ruby recipe will then install the packages it requires during the compile phase so the pg gem can be properly built. Up to this point, everybody is happy.

Now, lets say in some custom recipe, I have something like this:

package 'postgresql-client' do
 notifies :create, 'ruby_block[log pg install]', :immediately

ruby_block 'log pg install' do
 block do
   Chef::Log.warn "*** Postgresql Client has been installed!"
 action :nothing

What happens here is that postgresql::ruby breaks the expected behavior. Since the postgresql-client package was installed during compile time, no notifications will be processed, and the ruby_block resource will never be executed. I was experiencing this issue explicitly with package resources, which is why I whipped up the chef_package cookbook. However, as was pointed out in the code review, and as I have been starting to run into in some recent work, this issue affects more than just package resources. It affects all resources and their notifications.

In the cookbook, and the current pull request, I solved this for packages by creating ChefPackage resources for all Package resources. When a ChefPackage is discovered in the compile phase, a corresponding Package resource would be created based on the ChefPackage resource (ensuring notifications were in tact) and added to the resource collection. These resources would be processed regularly by the runner, but instead of attempting to run the action, it would simply check the resource if it had been updated by the last action and if so, apply notifications.

I am currently updating the pull request to provide this behavior for all resources by using a flag to mark it as having been updated during the compile phase, so the runner will know its status and if the notifications should be processed. One of the big questions is is that the expected behavior. In the code review, it was mentioned that immediate notifications might imply that the notified resource be executed during the compile phase. I don't feel that would be the expected behavior and don't think it _should_ be the behavior. However, with regards to expectations, I think having it defined in the documentation would be enough to make it clear.

The change that I am proposing is flagging resources that are run during compile time. This allows the updated_by_last_action value to be propogated to the execution phase. It also allows the runner to know the resource's action should not be attempted, only the notifications processed if its updated_by_last_action? is returning true.

While I would be really happy with built in tools for adding resources to arbitrary points in the resource collection, I don't see it providing a solution for cases like postgresql::ruby where recipes will be depending on the pg gem during the compile phase rather than the execution phase.

So, this change is targeting a very edge case, since compile time resource execution doesn't (and shouldn't) happen that often. But I feel that it's important for things to behave as expected when that edge case is encountered.


- Chris

Daniel DeLeo writes:

« HTML content follows »

Hi Chefs,

Recently we've been reviewing CHEF-3747 and wanted to get outside input.


In short, this adds a shortcut for installing a package at compile time, but it has an interesting twist: It creates a corresponding package resource that will trigger notifications at converge time where the resource "would have been" if it had been a converge-time resource.

We wanted to hear more from Chris (the patch's author) about what exactly he's using this behavior to do. As I understand it, there are two constraints:

1. These packages are prerequisites for gems that are installed and loaded via chef_gem;
2. When installed, additional resources need to run via notifications.

We also want to hear from you. This behavior seems quite complex; is it something you think you would use? Is the complexity worth the benefit?

I've also come across some discussions arguing that moving resources to compile time is an anti pattern. Personally I think its fine for one-off hacks, but I'm wary of baking it deeper into core Chef. Considering that the compile/converge distinction is pretty much required to implement notifications, those obviously act weirdly when everything gets moved to compile time. I personally also think Chef's strict ordering of run list items and the resource collection is the best way to get changes made in the desired order. On the other hand, this may necessitate splitting cookbooks into smaller pieces and sprinkling them across the run_list, which can be awkward.

One alternative we've been discussing here at Opscode is a refactor of the way that the recipe DSL is implemented such that you'd be able to prepend resources to the resource collection, or insert them at an arbitrary point, but this isn't something that would be implemented soon.


Daniel DeLeo

Archive powered by MHonArc 2.6.16.