[chef] Re: RE: Re: RE: Re: RE: Chef Search for Node Discovery.

Title: RE: [chef] Re: RE: Re: RE: Chef Search for Node Discovery.

Kevin,

The update frequency for Solr search index is configurable on your chef-server (see below).‎ I think creating the node object first is the correct approach Doug should be attempting. May be worth investigating chef provisioning and the machine resource as it actually creates a node object before start of chef-client run and let's you populate whatever values you desire. At end of chef-client run on vm the client object is created and node object is saved with remaining attributes.

Update Frequency

At the end of every chef-client run, the node object is saved to the Chef server. From the Chef server, each node object is then added to the SOLR search index. This process is asynchronous. By default, node objects are committed to the search index every 60 seconds or per 1000 node objects, whichever occurs first.‎

When data is committed to the Apache Solr index, all incoming updates are blocked. If the duration between updates is too short, it is possible for the rate at which updates are asked to occur to be faster than the rate at which objects can be actually committed.

For Chef server, these settings are configurable:

Setting	Description
opscode_solr['commit_interval']	The frequency (in seconds) at which node objects are added to the Apache Solr search index. Default value: ‎60000 (every 60 seconds).
opscode_solr['max_commit_docs']	The frequency (in documents) at which node objects are added to the Apache Solr search index. Default value: 1000 (every 1000 documents).‎

From: Kevin Keane Subscription‎

Sent: Sunday, June 14, 2015 2:12 AM

To:

Reply To:

Subject: [chef] RE: Re: RE: Re: RE: Chef Search for Node Discovery.‎

‎

Sorry it wasn't helpful, I did misunderstand you. It was actually news to me that search wouldn't work until after the first chef run. I just tried it, and actually search *does* seem to work as soon as the node is created (at least in Chef 12.0.8 with Chef 12.3.0 client). One thing: populate the FQDN. That appears to make a bit of a difference.

Here is what I did:

> knife node create test

> knife node search 'node:test'

Works as expected.

> knife node search 'node:*'

Did not return the new node in one of my test runs, but did in another.

Now add the fqdn attribute:

> knife node edit test

Add the fqdn attribute

Then

> knife node search 'node:*'

Now the new node will show up. Whether this was due to adding the FQDN attribute, or coincidence, I don't know. It may simply take a bit of time for chef to add the new node to search indices.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang < >
Sent: Saturday 13th June 2015 22:19
To:
Subject: [chef] Re: RE: Re: RE: Chef Search for Node Discovery.

Kevin,

Seems we have a slight disconnect. It's been my experience that you can't query anything about a node with chef search, not even the node itself, until after the client has run for the first time. I actually don't need to pull any attributes from the node. This is for zookeeper, so I just need to get a list of host names that match a criteria (role, environment etc).

Alas, I can't use a data bag, or a role. My boot scripts use AWS Dynamodb (and a concatenated role and environment as a key) to pick always incrementing host name numbers. I'd have to start the CF stack, wait until the scripts had queried DynamoDB, had set the Name tag on the instance, eyeball the name in the Amazon console, and then go update the role, upload it to chef etc, before the instance came fully up.

Doug

On Sat, Jun 13, 2015 at 5:46 PM, Kevin Keane Subscription < " target="_blank" title="This external link will open in a new window"> > wrote:

When you created the node, did you also populate it with whatever attributes your search will later use?

I've run into your problem, too, several times, actually - I'm sure it's common.

For instance, I'm using bacula as a backup solution. The bacula director (server) needs to know about all of the clients, and all of the clients need to know about the director.

So when the clients search for the director, there is a chance they don't find one (yet), and similarly, the director may not (yet) find any clients. Both would lead to errors in the bacula configuration. The system won't be stable until it has run twice.

I don't look as the solution as a "hack" but rather as necessary error handling.

There is one other option: instead of a search, you can use a data bag (or even an attribute) to explicitly store the information you would otherwise get from a search. I use this approach in situations where I may have multiple similar servers, and want to be explicitly in which one gets used - for instance, to control which database is stored on which MySQL server.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang < " target="_blank" title="This external link will open in a new window"> >

Sent: Saturday 13th June 2015 14:03
To: " target="_blank" title="This external link will open in a new window">
Subject: [chef] Re: RE: Chef Search for Node Discovery.

Kevin,

My boot process runs a script that calls a knife node add, to add the node to chef. Since it's been added to chef, it should be discoverable. However, it's not discoverable until the client has run at least once. That seems more like an architectural problem with chef, rather than a fundamental flaw in the universe. :)

What prevents me from doing a two pass run is that I would have to take a process that works fine, and add a hack to make it work when a chef search is required. Every node would have to run twice, or I would have to create some criteria, maybe have CloudFormation put a 'needs-chef-hack' EC2 tag on the instance, and then my scripts check for that. This really isn't very elegant.

What if I modified my boot scripts to run chef twice. The first run would essentially be to call a dummy cookbook that does nothing. After this has run, the node would be discoverable? Plenty of other folks must have this same issue.

Doug.

On Sat, Jun 13, 2015 at 1:43 PM, Kevin Keane Subscription < " title="This external link will open in a new window" target="_blank"> > wrote:

It seems to me that this problem is fundamental, not specific to chef: you simply can't discover anything until it is discoverable. Making something discoverable in an automated fashion can happen only by running *something*. In chef, it's chef itself. The other option is to have a separate pre-registration process that pre-creates (and pre-populates) the nodes.

That is also quite possible in chef; you'd use knife (possibly from a shell script) to automatically create your nodes.

But in the end, that doesn't really solve your problem, it just moves it.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Douglas Garstang < " title="This external link will open in a new window" target="_blank"> >
Sent: Friday 12th June 2015 22:01
To: " title="This external link will open in a new window" target="_blank">
Subject: [chef] Chef Search for Node Discovery.

I haven't tried this for a while, so I thought I'd see if there was any progress.

I need to write a cookbook that discovers other chef nodes. The issue with chef search is that nodes don't report in until they've done their first chef client run, which means that the client has to run twice on the one doing the discovering, but... that also applies to the one that wants to be discovered. End result, not automated. This has basically made chef search unusable in every situation for me where it might otherwise be useful.

Is there a better way? I'm on EC2. My other option was to write an ohai plugin to scan ec2 and return instances matching the correct criteria.

Thanks,

Doug

--

Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: " title="This external link will open in a new window" target="_blank">
Cell: +1-805-340-5627

--

Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: " target="_blank" title="This external link will open in a new window">
Cell: +1-805-340-5627