[chef] Problems with Search?

[Ash Berlin < >]

So some of my nagios checks started failing just now because the search that 
fills out allowed_hosts in nrpe.cfg was empty. This is the second time this 
happened to me - the first was immediately after upgrading to chef-server 
0.9.12.

The server is The clients in question are still on 0.9.8.

My cookbook code is (probably the standard nagios::client cookbook, possibly 
a few versions out of date):

search(:node, "roles:monitoring AND 
app_environment:#{node[:app_environment]}") do |n|
  mon_host << n['ipaddress']
end

...

template "/etc/nagios/nrpe.cfg" do
  source "nrpe.cfg.erb"
  owner "nagios"
  group "nagios"
  mode "0644"
  variables :mon_host => mon_host
  notifies :restart, resources(:service => "nagios-nrpe-server")
end
__END__

The initial problem on upgrading the server was the search was previously 
"role:..." which was working fine but needed to be changed to "roles:...".

However this time nothing changed and simply logging in to the problem hosts 
and re-running chef-client fixed it.

What went wrong and how to I track it down so it doesn't happen again?

-ash