[chef] Re: Re: Data Bag Search Delay

[Stephen Delano < >]

Solr 1.4, the Solr included with the chef server, is asynchronous in "commiting" saved object to the index. The rate at which Solr commits is tunable. The defaults are set to commit every 60 seconds or 1000 documents as seen here https://github.com/opscode/omnibus-chef-server/blob/master/files/chef-server-cookbooks/chef-server/attributes/default.rb#L74-L75.

You can tune these to your heart's content by editing the /etc/chef-server/chef-server.rb file to override the default values, but you should be aware of the tradeoffs that you're making by doing so.

Every time Solr commits to the index, it blocks all incoming updates. As you shorten the duration between commits, the time that chef-expander has available to send updates to Solr decreases and you may, under heavy load, find yourself in a state that your update rate outruns the rate at which you can commit objects to the index. If you're going to be putting this server under heavy load, proceed with caution.

What you may want to ask instead, is what is it about your usage of databags that necessitates real-time search?

-Stephen

On Wed, Oct 2, 2013 at 5:07 PM, Noah Kantrowitz < " target="_blank"> > wrote:

Get more CPU for Solr. There have been some experiments with replacing Solr with ElasticSearch which can have better insertion performance, so you could also look at working on that patch.

--Noah

On Oct 2, 2013, at 4:44 PM, Steven Barre < "> > wrote:

> It takes 60 seconds from when I call "knife data bag from file somebag path/to/some.json" until "knife search somebag" will return the answer. Is there anything that can be done to make that faster?
>
> http://community.opscode.com/questions/436 also describes the issue.
>
> CentOS 6.4
> chef-server-11.0.8-1.el6.x86_64
>
> --
> =================================================
> Steven Barre, RHCE, ZCE, MCP
> ">
>
> Systems Administrator / Programmer
> Real Estate Webmasters - 250-753-9893
> ==================================================
>

--
Stephen Delano
Software Development Engineer
Opscode, Inc.
1008 Western Avenue
Suite 601
Seattle, WA 98104