[chef] Re: Change hostname and IP address on Chef server

[Arthur Kalmenson < >]

jessicab has been helping me out on IRC and pointed me to
http://wiki.opscode.com/display/chef/Common+Errors#CommonErrors-401Unauthorized%28usingvalidatorAPIclient%29.
I tried recreating the validation.pem as follows:


 :/etc/chef$
 sudo rm validation.pem

 :/etc/chef$
 sudo /etc/init.d/chef-server restart
 * Restarting chef-server
                                                  ~ Killing pid 1407
with INT
 ~ In 15699

                                          [ OK ]

But the validation.pem was not recreated. It looks like the chef
server failed to start:


 :/etc/chef$
 sudo /etc/init.d/chef-server status
 * Checking status of chef-server
                                          [fail]

 :/etc/chef$
 sudo /etc/init.d/chef-server start
 * Starting chef-server
                                                  ~ In 16771

                                          [fail]

I've tried looking around /var/log/chef and /var/log/ to see if
anything is getting logged, but I didn't find anything.
--
Arthur Kalmenson



On Fri, Dec 23, 2011 at 3:08 PM, Arthur Kalmenson 
<
 >
 wrote:
> Hello everyone,
>
> I'm trying to change the hostname and IP address of the Chef server
> that I set up. My steps have been the following:
>
> 1. Delete all the nodes and clients (except chef-validator, chef-webui
> and the user on the chef server using knife).
> 2. Change the hostname, /etc/hosts, and the IP address. Reboot.
> 3. At this point, I remove the client.pem file from all the nodes,
> update the client.rb to point to the new hostname, and try to re-add
> them as nodes by running "sudo chef-client"
>
> This is where I've been running into trouble.
>
> 
 :~$
>  knife client list
>  chef-validator
>  chef-webui
>  devuser
>
> As you can see, the only remaining clients are chef-validator,
> chef-webui and devuser (the knife user on the chef server). Running
> "knife node list" is empty.
>
> The first node I want to add back on is the chef server itself, so I
> delete /etc/chef/client.pem and run "sudo chef-client"
>
> 
 :~$
>  sudo chef-client
> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: *** Chef 0.10.8 ***
> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: Client key
> /etc/chef/client.pem is not present - registering
> [Fri, 23 Dec 2011 14:50:18 -0500] INFO: HTTP Request Returned 500
> Internal Server Error: Connection refused - connect(2)
> [Fri, 23 Dec 2011 14:50:18 -0500] ERROR: Server returned error for
> http://devserverconfig:4000/clients, retrying 1/5 in 3s
> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 409
> Conflict: Client already exists
> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 403
> Forbidden: You are not allowed to take this action.
> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Stacktrace dumped to
> /var/cache/chef/chef-stacktrace.out
> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Net::HTTPServerException: 403
> "Forbidden"
>
> 
 :~$
>  cat /var/cache/chef/chef-stacktrace.out
> Generated at Fri Dec 23 14:50:21 -0500 2011
> Net::HTTPServerException: 403 "Forbidden"
> /usr/lib/ruby/1.8/net/http.rb:2101:in `error!'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in `retriable_rest_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in `api_request'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in `put_rest'
> /usr/lib/ruby/vendor_ruby/chef/api_client.rb:247:in `save'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:81:in `register'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `upto'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `register'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `catch'
> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `register'
> /usr/lib/ruby/vendor_ruby/chef/client.rb:280:in `register'
> /usr/lib/ruby/vendor_ruby/chef/client.rb:150:in `run'
> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:239:in 
> `run_application'
> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in `loop'
> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in 
> `run_application'
> /usr/lib/ruby/vendor_ruby/chef/application.rb:67:in `run'
> /usr/bin/chef-client:25
>
>
> But, how could that client be there if it didn't exist a moment ago?
> Turns out, now when I run "knife client list", this client appears:
>
> 
 :~$
>  knife client list
>  chef-validator
>  chef-webui
>  devserverconfig
>  devuser
>
> But it's not a node because "knife node list" is empty. So now when I
> try to delete this "ghost" client, I get the following:
>
> 
 :~$
>  knife client delete devserverconfig
> Do you really want to delete devserverconfig? (Y/N) Y
> ERROR: Server returned error for
> http://localhost:4000/clients/devserverconfig, retrying 1/5 in 3s
> ERROR: The object you are looking for could not be found
> Response: Cannot load client devserverconfig
>
> 
 :~$
>  knife client list
>  chef-validator
>  chef-webui
>  devuser
>
>
> As you can see, there's an error deleting but it does get deleted.
> Anyway, at this point I'm stuck and not able to add any nodes to this
> chef server with a new hostname. I've posted excerpts from the
> /var/log/chef/server.log file that gives more details at the errors:
> https://gist.github.com/1515217. I'm still experimenting with Chef
> right now, so I'm willing to reinstall everything, but it'd be good to
> understand what's going on.
>
> Any help would be greatly appreciated. Thank you in advance.