[chef-dev] Re: [] Lightweight Search

[Justin Alan Ryan < >]

Got a chance to look closer at the whitelist cookbook today,

This is an interesting solution.  If I understand correctly:

  (a) It stores less data in couch

  (b) lucene has to index less data

  (c) the size of the returned node object is smaller, in addition to being less trouble to handle on the server

Am I correct?

Solutions we've been tossing around include:

  * Patch Chef search view to accept optional arguments and filter at that layer, which would also require patching the client.  We'd prefer not to actually launch this solution without a patch being accepted because we don't want to run a bunch of patched stuff, for maintainability sake.

  * Generate data bags of the data we need on the server via some sort of cron-ed knife scripts

  * Generate flat files that we can use via remote_file via some sort of cron, or in chef recipes running on the chef server.

The more I think about it, the more I like the idea of a whitelist because we think there may be problems with the accuracy of our search results due to failed indexing or something, it seems that if we stored and indexed less data the chef server could do more with less.

I'm not particularly excited about using whitelist as a cookbook which absolutely must be the last item, because it breaks our workflow and is likely to result in lots of relatively unproductive conversations.

Is there any roadmap to have whitelist in 0.10 series as a feature?  Are there any other ideas floating about which could address this concern, e.g. a way to force items to be sticky at the beginning or end of run_list?  We already have a recipe for measuring memory which has to run last, and adjusting the run_list on each of those handful of nodes has become:

  knife node run_list remove "recipe[chef::mem]"

  knife node run_list add "recipe[something-new-and-fancy]"

  knife node run_list add "recipe[chef::mem]"

That's annoying, but manageable for 5 nodes.  For 290 nodes to ensure that everyone with knife access knows to keep "recipe[whitelist-node-attrs]" at the end or their chef run may fail sounds like potentially trading one headache for another. ;d

It's not crucial to fix this in one day, but we need to do something soon.  I'm fully inclined for that to be something which involves contributing to making chef better for everyone.

On Wed, Aug 17, 2011 at 8:31 PM, Justin Alan Ryan < "> > wrote:

On Wed, Aug 17, 2011 at 5:47 PM, Adam Jacob < " target="_blank"> > wrote:

Also, I've been playing around a bit with getting the size of the node objects saved to be smaller. The current leading contender is to whitelist the node attributes you actually want saved at the end of the run - this way all the attributes from Ohai + Cookbooks are available during the run, but only the ones you actually care about for search are saved. A working prototype is here:

https://github.com/opscode/whitelist-node-attrs

This should help in the near term - slim down the size of the objects to the parts you care about, and the aggregate size of the results will shrink as well. We should still fix the core issue of only returning full objects on search, but this should help in the interim.

--
http://about.me/justizin