[chef] Re: Re: Re: Re: Re: Re: Re: Re: Policyfiles and chef provision

[Chris Sibbitt < >]

Thank you very much Christine and Daniel for your helpful discussion. I think this line is really what answers my question: 

So, for the policyfile focused stuff, I opted to make it work more like `knife bootstrap` and `knife cloud create` 

That means you specifically were looking at the case where a provisioning recipe handles just one machine. I don't think I will bother with the "chef provision" tool in that case, since I don't want to have to enforce that "one machine per recipe" rule in our code, and the allure of chef-provisioning for us is being able to use it to bring up whole "Stacks" of related machines (what Christine called 'topologies' I think).

I did think about clustering scenarios when I was writing `chef provision`, but it turns out that it can be complicated depending on what the exact use case is. Do you want a “throwaway” cluster to integration-test your cookbooks as a whole? How do you keep different developers’ throwaway clusters from conflicting with each other?

So far my solution to this (though we haven't worked with it for long enough to say if it's a good one long term) is to require ENV['DEPLOYMENTID'] exist and use it in the name of every chef_provisioning resource. That way, each deployment of a stack has it's own machine names and can be managed as a distinct entity of related machines. 

 

The workflow I've settled on is similar to how Daniel describes it:

1. Set up the policyfile

2. Install/Update/Push the policyfile

3. Write chef-provisioning code to set up machines to use a named policyfile

4. Run the provisioning recipes with something like `chef-client -z -o my-provisioning::mystack`

This thread was about whether I should be using 'chef provision' in step #4 because running chef-zero or chef-solo on my provisioning node feels weird.

For step 3, I've done a more cookbook-heavy version of Daniel's suggestion:

From there, you need to set the policy_group and policy_name on the nodes via the client.rb (not sure how well this is documented, but you can pass configuration as a string with { chef_config: “your client.rb content” } as convergence_options)

Here's what I do:  

default['my-provisioning']['node_attrs'] = {

  :chef_client => {

    :config => {

      :use_policyfile => true,

      :policy_document_native_api => true,

      :policy_group => 'mypolicygroup',

      :policy_name => 'mypolicy'

} } }

machine "mymachine-#{ENV['DEPLOYMENTID']}" do

  run_list ['recipe[chef-client]', 'recipe[chef-client::config]']

  attributes node['my-provisioning']['node_attrs'].merge(

    :mycookbooks => {

      :config => {

        :stuff => 'values'

    } } )

end

This uses the chef-client cookbook to set up the policyfile mode for the node. Actual control of the run_list is in the policyfile.

There are some downsides to this:

* You need to run a second converge on the node in order to get it running the right run_list; so that can either happen automatically via chef-client service config, or you can add the second converge right in your chef-provisioning recipe.

* IIRC, subsequent executions of the the provisioning recipe don't actually run the real run_list, they just ensure that it's still set up for the right policy

* You need to serve a version of the chef-client cookbook without the benefit of policyfile locking

* Your run_list is separate from your machine definition, which is a bit awkward, leaving all your machines with the same uninformative run_list specification in your provisioning recipes

I think I will experiment with using 'chef_config: “your client.rb content”' instead of the chef-client cookbook, it might cut down a bit on complexity.

Thanks again, this is exactly what I was looking for from this thread. I would be interested in hearing any additional thoughts about my workflow and especially any insight to how you see the co-ordination between these tools evolving in the next year or so.

On Thu, Jul 23, 2015 at 7:02 PM, Daniel DeLeo < " target="_blank"> > wrote:

On Thursday, July 23, 2015 at 1:19 PM, Christine Draper wrote:
> Dan,
>
> I didn't mean 'all nodes being used for testing need the same cookbook versions'. I wanted to communicate:
> * A particular topology (dbserver + appserver) should have the same cookbook versions

Are you using community cookbooks at all? Would it violate a policy if the dbserver and app server used different versions of a cookbook like apt or yum or build-essentials? Just want to be sure I’m understanding your constraints.

> * Different topologies may have different cookbook versions - depending on what the tester is working on
>
> That's actually part of why I'm interested in policyfiles - trying to achieve this with environments would be painful.
>
> The mechanics of the process sounds plausibly workable. However, I thought policyfiles set the runlist and attributes on the node as well as constraining cookbooks, so wouldn't that conflict with setting the runlist and attributes in the provisioning recipe?
Node-specific attributes are not affected by policyfiles, they work the same as they have. Policyfiles allow you to set attributes in the same way as roles (defaults and overrides, which have the same precedence as role attributes). You can pick and choose between setting the attributes on the node via provisioning and default and override attributes in the policyfile as you like.

When you enable policyfile mode on a chef-client, it will wipe the run list from the node and replace it with the one from the policyfile. Whatever you set in Chef Provisioning would be silently ignored.

>
> Regards,
> Christine


HTH,

--
Daniel DeLeo

--

Chris Sibbitt | Infrastructure Systems Architect

---

WWW.THINKINGPHONES.COM

P: 613.686.1590 " style="color:rgb(17,85,204)" target="_blank">