[chef] Re: Re: Re: Re: Latency on search indicies

[Peter Donald < >]

On Tue, Jun 25, 2013 at 6:00 PM, Jeppe Nejsum Madsen 
<
 >
 wrote:
>>>> Just be warned that the lag can be much more significant. We run the
>>>> open source variant of chef for now and have noticed lags of anywhere
>>>> between <1s to ~120s between the data being inserted and it being
>>>> accessible via search.
>>>
>>> What version of chef-server are you running?  Is it Chef 11.x, 10.x, or 
>>> something older?
>>
>> We actually use Chef 11 but to be honest most of the strategies we
>> used to address this were developed when we were on 10.x.
>
> We're using windows as well. Can you outline what strategies you used?

Mostly it involved reducing load generated via search, reduce the
amount of data that went into the search indexes and support explicit
wait until constructs to wait until search is updated.

In no particular order we did the following;

* Remove all the ohai plugins we weren't using a recipe (See
https://gist.github.com/realityforge/5863439)
* Strip out unnecessary ohai data at the end of a run. Conceptually
similar to https://github.com/opscode-cookbooks/whitelist-node-attrs
except we used a blacklist instead
* Use partial_search rather than regular search where possible (See
https://github.com/opscode-cookbooks/partial_search)
* Avoid broad searching of the node index where possible. Require
nodes to explicitly publish data that they expect other nodes to use
in a manner friendly to partial search. i.e. See
https://github.com/realityforge/chef-cutlery/blob/master/libraries/services.rb#L24
* Add an abstraction layer so that data can be stored either against a
node or in a data bag item. Sometimes we moved things from a node to a
databag. See 
https://github.com/realityforge/chef-cutlery/blob/master/libraries/services.rb#L53
* Our orchestration layer (bunch of ruby libraries using spice that
are driven by jenkins) has built in primitives to wait until the
search index has been updated.

We probably did a few other things as well. But most significant
change was the introduction of the last strategy. We now are driving
more and more of our config via data bags to try and limit the idea of
"exported" resources interacting across nodes.

HTH,

Peter Donald