[chef] Re: Change hostname and IP address on Chef server

[Arthur Kalmenson < >]

Hello everyone,

I'm trying to change the hostname and IP address of the Chef server
that I set up. My steps have been the following:

1. Delete all the nodes and clients (except chef-validator, chef-webui
and the user on the chef server using knife).
2. Change the hostname, /etc/hosts, and the IP address. Reboot.
3. At this point, I remove the client.pem file from all the nodes,
update the client.rb to point to the new hostname, and try to re-add
them as nodes by running "sudo chef-client"

This is where I've been running into trouble.


 :~$
 knife client list
 chef-validator
 chef-webui
 devuser

As you can see, the only remaining clients are chef-validator,
chef-webui and devuser (the knife user on the chef server). Running
"knife node list" is empty.

The first node I want to add back on is the chef server itself, so I
delete /etc/chef/client.pem and run "sudo chef-client"


 :~$
 sudo chef-client
[Fri, 23 Dec 2011 14:50:17 -0500] INFO: *** Chef 0.10.8 ***
[Fri, 23 Dec 2011 14:50:17 -0500] INFO: Client key
/etc/chef/client.pem is not present - registering
[Fri, 23 Dec 2011 14:50:18 -0500] INFO: HTTP Request Returned 500
Internal Server Error: Connection refused - connect(2)
[Fri, 23 Dec 2011 14:50:18 -0500] ERROR: Server returned error for
http://devserverconfig:4000/clients, retrying 1/5 in 3s
[Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 409
Conflict: Client already exists
[Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 403
Forbidden: You are not allowed to take this action.
[Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Stacktrace dumped to
/var/cache/chef/chef-stacktrace.out
[Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Net::HTTPServerException: 403
"Forbidden"


 :~$
 cat /var/cache/chef/chef-stacktrace.out
Generated at Fri Dec 23 14:50:21 -0500 2011
Net::HTTPServerException: 403 "Forbidden"
/usr/lib/ruby/1.8/net/http.rb:2101:in `error!'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in `retriable_rest_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in `api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in `put_rest'
/usr/lib/ruby/vendor_ruby/chef/api_client.rb:247:in `save'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:81:in `register'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `upto'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `register'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `catch'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `register'
/usr/lib/ruby/vendor_ruby/chef/client.rb:280:in `register'
/usr/lib/ruby/vendor_ruby/chef/client.rb:150:in `run'
/usr/lib/ruby/vendor_ruby/chef/application/client.rb:239:in `run_application'
/usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in `loop'
/usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in `run_application'
/usr/lib/ruby/vendor_ruby/chef/application.rb:67:in `run'
/usr/bin/chef-client:25


But, how could that client be there if it didn't exist a moment ago?
Turns out, now when I run "knife client list", this client appears:


 :~$
 knife client list
 chef-validator
 chef-webui
 devserverconfig
 devuser

But it's not a node because "knife node list" is empty. So now when I
try to delete this "ghost" client, I get the following:


 :~$
 knife client delete devserverconfig
Do you really want to delete devserverconfig? (Y/N) Y
ERROR: Server returned error for
http://localhost:4000/clients/devserverconfig, retrying 1/5 in 3s
ERROR: The object you are looking for could not be found
Response: Cannot load client devserverconfig


 :~$
 knife client list
 chef-validator
 chef-webui
 devuser


As you can see, there's an error deleting but it does get deleted.
Anyway, at this point I'm stuck and not able to add any nodes to this
chef server with a new hostname. I've posted excerpts from the
/var/log/chef/server.log file that gives more details at the errors:
https://gist.github.com/1515217. I'm still experimenting with Chef
right now, so I'm willing to reinstall everything, but it'd be good to
understand what's going on.

Any help would be greatly appreciated. Thank you in advance.