[chef] client registration fails with failed to authenticate - part way through series of installs.

[Tom < >]

Hi all,

I have been deploying chef-client (0.10.8) from gems to some instances using a script which delivers a validation.pem key for the chef-validator user. The script was working well with many instances deployed.

However at some point all the new instances start to fail with the following error;   (The server log is here)
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: HTTP Request Returned 401 Unauthorized: Failed to authenticate. Ensure that your client key is valid.

According to the debug log, the NTP time differences are OK, and the Hash content is ok, but the request has failed a "OpenSSL::PKey::RSAError: padding check failed"

See here...

1. DEBUG: Failed to verify request signature: OpenSSL::PKey::RSAError: padding check failed
2. DEBUG: Request time difference: 0.617022, within 900 seconds? : true
3. DEBUG: Expected content hash is: 'VD3sVc7y8Od5rhMPZqxkdaNP5Q8='
4. DEBUG:  Request Content Hash is: 'VD3sVc7y8Od5rhMPZqxkdaNP5Q8='
5. DEBUG:            Hashes match?: true

Any suggestion on what might be the problem?

This seems to intermittently happen, and if I regenerate the chef-validator private key I might get another week or 2 before that key gets screwed up.

Thanks,
Tom








Linux opencirrus-g0801.hpl.hp.com 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9 12:54:20 EST 2010 x86_64 x86_64 x86_64 GNU/Linux


~]# ruby -v
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

~]# gem search chef
*** LOCAL GEMS ***
chef (0.10.8)
chef-expander (0.10.8)
chef-server (0.10.8)
chef-server-api (0.10.8)
chef-server-webui (0.10.8)
chef-solr (0.10.8)



client log;
~]# chef-client -j /etc/chef/first-boot.json
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: *** Chef 0.10.8 ***
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: Client key /etc/chef/client.pem is not present - registering
[Tue, 10 Jan 2012 03:13:08 -0800] INFO: HTTP Request Returned 401 Unauthorized: Failed to authenticate. Ensure that your client key is valid.
[Tue, 10 Jan 2012 03:13:08 -0800] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
[Tue, 10 Jan 2012 03:13:08 -0800] FATAL: Net::HTTPServerException: 401 "Unauthorized"
~]#


The server log is here