- From: Adam Jacob <
>
- To: Justin Alan Ryan <
>
- Cc:
- Subject: [chef-dev] Re: [] Lightweight Search
- Date: Thu, 18 Aug 2011 14:36:58 -0700
Yep - you got it. It's not the final answer (which i think involves partial
search results, and a change to the client API) but it will go a long way.
To make the whitelist code as it stands part of the chef server, you can
probably add a library that patches Chef::Node#save to execute the same code
as the whitelist does, while keeping a copy of the 4 original hashes, so it
can restore them post-save (in the case of someone calling node.save
explicitly.)
Something like this showing up in core chef seems like a good idea.
Adam
--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E:
On Thursday, August 18, 2011 at 2:25 PM, Justin Alan Ryan wrote:
>
Got a chance to look closer at the whitelist cookbook today,
>
>
This is an interesting solution. If I understand correctly:
>
>
(a) It stores less data in couch
>
(b) lucene has to index less data
>
(c) the size of the returned node object is smaller, in addition to being
>
less trouble to handle on the server
>
>
Am I correct?
>
>
Solutions we've been tossing around include:
>
>
* Patch Chef search view to accept optional arguments and filter at that
>
layer, which would also require patching the client. We'd prefer not to
>
actually launch this solution without a patch being accepted because we
>
don't want to run a bunch of patched stuff, for maintainability sake.
>
* Generate data bags of the data we need on the server via some sort of
>
cron-ed knife scripts
>
* Generate flat files that we can use via remote_file via some sort of
>
cron, or in chef recipes running on the chef server.
>
>
The more I think about it, the more I like the idea of a whitelist because
>
we think there may be problems with the accuracy of our search results due
>
to failed indexing or something, it seems that if we stored and indexed
>
less data the chef server could do more with less.
>
>
I'm not particularly excited about using whitelist as a cookbook which
>
absolutely must be the last item, because it breaks our workflow and is
>
likely to result in lots of relatively unproductive conversations.
>
>
Is there any roadmap to have whitelist in 0.10 series as a feature? Are
>
there any other ideas floating about which could address this concern, e.g.
>
a way to force items to be sticky at the beginning or end of run_list? We
>
already have a recipe for measuring memory which has to run last, and
>
adjusting the run_list on each of those handful of nodes has become:
>
>
knife node run_list remove "recipe[chef::mem]"
>
knife node run_list add "recipe[something-new-and-fancy]"
>
knife node run_list add "recipe[chef::mem]"
>
>
That's annoying, but manageable for 5 nodes. For 290 nodes to ensure that
>
everyone with knife access knows to keep "recipe[whitelist-node-attrs]" at
>
the end or their chef run may fail sounds like potentially trading one
>
headache for another. ;d
>
>
It's not crucial to fix this in one day, but we need to do something soon.
>
I'm fully inclined for that to be something which involves contributing to
>
making chef better for everyone.
>
>
On Wed, Aug 17, 2011 at 8:31 PM, Justin Alan Ryan
>
<
>
>
(mailto:
)>
>
wrote:
>
>
>
>
>
> On Wed, Aug 17, 2011 at 5:47 PM, Adam Jacob
>
> <
>
>
>
> (mailto:
)>
>
> wrote:
>
> > Also, I've been playing around a bit with getting the size of the node
>
> > objects saved to be smaller. The current leading contender is to
>
> > whitelist the node attributes you actually want saved at the end of the
>
> > run - this way all the attributes from Ohai + Cookbooks are available
>
> > during the run, but only the ones you actually care about for search
>
> > are saved. A working prototype is here:
>
> >
>
> > https://github.com/opscode/whitelist-node-attrs
>
> >
>
> > This should help in the near term - slim down the size of the objects
>
> > to the parts you care about, and the aggregate size of the results will
>
> > shrink as well. We should still fix the core issue of only returning
>
> > full objects on search, but this should help in the interim.
>
--
>
http://about.me/justizin
Archive powered by MHonArc 2.6.16.