[chef] Problem accessing databag from node

[Michael Glenney < >]

I'm having problems with a 403 error when trying a new cookbook and I think I've tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 "Forbidden"
/usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in api_request'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in `retriable_rest_request'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api_request'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in `put_rest'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in `save'
/var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from_file'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in `instance_eval'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in `from_file'
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in `load_recipe'


The relevant part of that recipe is:

# Find cluster master, or create item and become master
if search(:ejabberd, "id:#{deployid}").count == 0
  masternode = "#{node[:ipaddress]}"
  h = {}
  h[deployid] = {"id" => deployid, "master" => masternode, "members" => [masternode]}

  # Create new data bag item for cluster
  databag_item = Chef::DataBagItem.new
  databag_item.data_bag("ejabberd")
  databag_item.raw_data = h[deployid]
  databag_item.save
else

This is for a first chef run on a new node.  If I run shef, switch to recipe context, and run 'search(:ejabberd, "id:deployment_000010182")' I get back:

chef:recipe > search(:ejabberd, "id:deployment_000010182")
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused connecting to localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server.  If I run the same command from my local box I get back:

chef:recipe > search(:ejabberd, "id:deployment_000010182")
 => [] 

which is what I expect.  Any ideas where I should be looking?

Thanks,

MG