[chef] Chef Server 12.01 - Spontaneous Client ERROR: 403 "Forbidden"

[Mark Selby < >]

I have recently upgraded from Chef Server 11 to 12 and have come across a very frustrating issue.

I have a set of nodes that are converging fine and then poof they spontaneously can not set their node state back up to the chef server at the end of the run. 

Can anyone point out some things I might do to troubleshoot this issue. I am including what I think are helpful logs but I am sure there is more than I can submit.

The only way that I can fix this is by deleting the node and client and reestablish them.

Thanks!

Server = CentOS 6.6 

########################

## This on the client ##

########################

Running handlers:

[2015-01-02T10:26:00-08:00] ERROR: Running exception handlers

Running handlers complete

[2015-01-02T10:26:00-08:00] ERROR: Exception handlers complete

[2015-01-02T10:26:00-08:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

Chef Client failed. 0 resources updated in 4.740723394 seconds

[2015-01-02T10:26:00-08:00] ERROR: 403 "Forbidden"

[2015-01-02T10:26:00-08:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

cat /var/chef/cache/chef-stacktrace.out

Generated at 2015-01-02 10:26:00 -0800

Net::HTTPServerException: 403 "Forbidden"

/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'

/opt/chef/embedded/apps/chef/lib/chef/http.rb:145:in `request'

/opt/chef/embedded/apps/chef/lib/chef/http.rb:118:in `put'

/opt/chef/embedded/apps/chef/lib/chef/node.rb:512:in `save'

/opt/chef/embedded/apps/chef/lib/chef/client.rb:257:in `save_updated_node'

/opt/chef/embedded/apps/chef/lib/chef/client.rb:403:in `run'

/opt/chef/embedded/apps/chef/lib/chef/application.rb:261:in `block in fork_chef_client'

/opt/chef/embedded/apps/chef/lib/chef/application.rb:249:in `fork'

/opt/chef/embedded/apps/chef/lib/chef/application.rb:249:in `fork_chef_client'

/opt/chef/embedded/apps/chef/lib/chef/application.rb:215:in `block in run_chef_client'

/opt/chef/embedded/apps/chef/lib/chef/local_mode.rb:38:in `with_server_connectivity'

/opt/chef/embedded/apps/chef/lib/chef/application.rb:201:in `run_chef_client'

/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:355:in `block in interval_run_chef_client'

/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:in `loop'

/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:in `interval_run_chef_client'

/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:335:in `run_application'

/opt/chef/embedded/apps/chef/lib/chef/application.rb:58:in `run'

/opt/chef/embedded/apps/chef/bin/chef-client:26:in `<top (required)>'

/usr/bin/chef-client:40:in `load'

/usr/bin/chef-client:40:in `<main>'

########################

## This on the server ##

########################

from /var/log/opscode/opscode-erchef/requests.log.1

2015-01-01T14:19:15Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=200; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAA4PQAAAAgAAAAA; org_name=example; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=18; rdbms_time=5; rdbms_count=4; authz_time=3; authz_count=1; user=serv.example.com; 

2015-01-01T14:52:34Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=200; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAC2vQAAAAgAAAAA; org_name=example; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=17; rdbms_time=5; rdbms_count=4; authz_time=3; authz_count=1; user=serv.example.com; 

2015-01-01T15:22:44Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=200; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAEgewAAAAgAAAAA; org_name=example; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=17; rdbms_time=6; rdbms_count=4; authz_time=3; authz_count=1; user=serv.example.com; 

2015-01-01T15:56:29Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=200; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAGgNwAAAAgAAAAA; org_name=example; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=17; rdbms_time=5; rdbms_count=4; authz_time=3; authz_count=1; user=serv.example.com; 

2015-01-01T16:39:35Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=403; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAJMIQAAAAgAAAAA; org_name=example; msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=7; rdbms_time=0; rdbms_count=3; authz_time=4; authz_count=1; user=serv.example.com; 

2015-01-01T17:02:59Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=403; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAKU/gAAAAgAAAAA; org_name=example; msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=7; rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=serv.example.com; 

2015-01-01T17:34:07Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=403; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAMKLAAAAAgAAAAA; org_name=example; msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=7; rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=serv.example.com; 

2015-01-01T18:06:20Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=403; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAN/AwAAAAgAAAAA; org_name=example; msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=7; rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=serv.example.com; 

2015-01-01T18:36:35Z "> method=PUT; path=/organizations/example/nodes/serv.example.com; status=403; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAPs6gAAAAgAAAAA; org_name=example; msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=7; rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=serv.example.com;