[chef-dev] Re: [] Lightweight Search

[Adam Jacob < >]

 Yep - you got it. It's not the final answer (which i think involves partial 
search results, and a change to the client API) but it will go a long way. 

To make the whitelist code as it stands part of the chef server, you can 
probably add a library that patches Chef::Node#save to execute the same code 
as the whitelist does, while keeping a copy of the 4 original hashes, so it 
can restore them post-save (in the case of someone calling node.save 
explicitly.)

Something like this showing up in core chef seems like a good idea.

Adam 

-- 
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: 

 


On Thursday, August 18, 2011 at 2:25 PM, Justin Alan Ryan wrote:

> Got a chance to look closer at the whitelist cookbook today,
> 
> This is an interesting solution. If I understand correctly:
> 
>  (a) It stores less data in couch
>  (b) lucene has to index less data
>  (c) the size of the returned node object is smaller, in addition to being 
> less trouble to handle on the server
> 
> Am I correct?
> 
> Solutions we've been tossing around include: 
> 
>  * Patch Chef search view to accept optional arguments and filter at that 
> layer, which would also require patching the client. We'd prefer not to 
> actually launch this solution without a patch being accepted because we 
> don't want to run a bunch of patched stuff, for maintainability sake. 
>  * Generate data bags of the data we need on the server via some sort of 
> cron-ed knife scripts
>  * Generate flat files that we can use via remote_file via some sort of 
> cron, or in chef recipes running on the chef server.
> 
> The more I think about it, the more I like the idea of a whitelist because 
> we think there may be problems with the accuracy of our search results due 
> to failed indexing or something, it seems that if we stored and indexed 
> less data the chef server could do more with less. 
> 
> I'm not particularly excited about using whitelist as a cookbook which 
> absolutely must be the last item, because it breaks our workflow and is 
> likely to result in lots of relatively unproductive conversations. 
> 
> Is there any roadmap to have whitelist in 0.10 series as a feature? Are 
> there any other ideas floating about which could address this concern, e.g. 
> a way to force items to be sticky at the beginning or end of run_list? We 
> already have a recipe for measuring memory which has to run last, and 
> adjusting the run_list on each of those handful of nodes has become: 
> 
>  knife node run_list remove "recipe[chef::mem]"
>  knife node run_list add "recipe[something-new-and-fancy]"
>  knife node run_list add "recipe[chef::mem]"
> 
> That's annoying, but manageable for 5 nodes. For 290 nodes to ensure that 
> everyone with knife access knows to keep "recipe[whitelist-node-attrs]" at 
> the end or their chef run may fail sounds like potentially trading one 
> headache for another. ;d 
> 
> It's not crucial to fix this in one day, but we need to do something soon. 
> I'm fully inclined for that to be something which involves contributing to 
> making chef better for everyone.
> 
>  On Wed, Aug 17, 2011 at 8:31 PM, Justin Alan Ryan 
> <
 
>  
> (mailto:
 )>
>  wrote:
> > 
> > 
> > On Wed, Aug 17, 2011 at 5:47 PM, Adam Jacob 
> > <
 
> >  
> > (mailto:
 )>
> >  wrote:
> > >  Also, I've been playing around a bit with getting the size of the node 
> > > objects saved to be smaller. The current leading contender is to 
> > > whitelist the node attributes you actually want saved at the end of the 
> > > run - this way all the attributes from Ohai + Cookbooks are available 
> > > during the run, but only the ones you actually care about for search 
> > > are saved. A working prototype is here:
> > > 
> > > https://github.com/opscode/whitelist-node-attrs
> > > 
> > >  This should help in the near term - slim down the size of the objects 
> > > to the parts you care about, and the aggregate size of the results will 
> > > shrink as well. We should still fix the core issue of only returning 
> > > full objects on search, but this should help in the interim.
> -- 
> http://about.me/justizin