[chef] Re: Re: Change hostname and IP address on Chef server

[Tom Thomas < >]

From http://wiki.opscode.com/display/chef/Support

Log file locations
Check the server or client config file for the log_location.

• /etc/chef/server.rb
• /etc/chef/client.rb

- Tom 

On Tue, Dec 27, 2011 at 9:15 AM, Arthur Kalmenson < "> > wrote:

Any ideas where I could find logs to see why chef-server is failing to
start and unable to recreate the validation.pem? Thank you.

--
Arthur Kalmenson

On Fri, Dec 23, 2011 at 3:40 PM, Arthur Kalmenson < "> > wrote:
> jessicab has been helping me out on IRC and pointed me to
> http://wiki.opscode.com/display/chef/Common+Errors#CommonErrors-401Unauthorized%28usingvalidatorAPIclient%29.
> I tried recreating the validation.pem as follows:
>
> :/etc/chef$ sudo rm validation.pem
> :/etc/chef$ sudo /etc/init.d/chef-server restart
>  * Restarting chef-server
>                                                  ~ Killing pid 1407
> with INT
>  ~ In 15699
>
>                                          [ OK ]
>
> But the validation.pem was not recreated. It looks like the chef
> server failed to start:
>
> :/etc/chef$ sudo /etc/init.d/chef-server status
>  * Checking status of chef-server
>                                          [fail]
> :/etc/chef$ sudo /etc/init.d/chef-server start
>  * Starting chef-server
>                                                  ~ In 16771
>
>                                          [fail]
>
> I've tried looking around /var/log/chef and /var/log/ to see if
> anything is getting logged, but I didn't find anything.
> --
> Arthur Kalmenson
>
>
>
> On Fri, Dec 23, 2011 at 3:08 PM, Arthur Kalmenson < "> > wrote:
>> Hello everyone,
>>
>> I'm trying to change the hostname and IP address of the Chef server
>> that I set up. My steps have been the following:
>>
>> 1. Delete all the nodes and clients (except chef-validator, chef-webui
>> and the user on the chef server using knife).
>> 2. Change the hostname, /etc/hosts, and the IP address. Reboot.
>> 3. At this point, I remove the client.pem file from all the nodes,
>> update the client.rb to point to the new hostname, and try to re-add
>> them as nodes by running "sudo chef-client"
>>
>> This is where I've been running into trouble.
>>
>> :~$ knife client list
>>  chef-validator
>>  chef-webui
>>  devuser
>>
>> As you can see, the only remaining clients are chef-validator,
>> chef-webui and devuser (the knife user on the chef server). Running
>> "knife node list" is empty.
>>
>> The first node I want to add back on is the chef server itself, so I
>> delete /etc/chef/client.pem and run "sudo chef-client"
>>
>> :~$ sudo chef-client
>> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: *** Chef 0.10.8 ***
>> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: Client key
>> /etc/chef/client.pem is not present - registering
>> [Fri, 23 Dec 2011 14:50:18 -0500] INFO: HTTP Request Returned 500
>> Internal Server Error: Connection refused - connect(2)
>> [Fri, 23 Dec 2011 14:50:18 -0500] ERROR: Server returned error for
>> http://devserverconfig:4000/clients, retrying 1/5 in 3s
>> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 409
>> Conflict: Client already exists
>> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 403
>> Forbidden: You are not allowed to take this action.
>> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Stacktrace dumped to
>> /var/cache/chef/chef-stacktrace.out
>> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Net::HTTPServerException: 403
>> "Forbidden"
>>
>> :~$ cat /var/cache/chef/chef-stacktrace.out
>> Generated at Fri Dec 23 14:50:21 -0500 2011
>> Net::HTTPServerException: 403 "Forbidden"
>> /usr/lib/ruby/1.8/net/http.rb:2101:in `error!'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in `api_request'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in `retriable_rest_request'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in `api_request'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in `put_rest'
>> /usr/lib/ruby/vendor_ruby/chef/api_client.rb:247:in `save'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:81:in `register'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `upto'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `register'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `catch'
>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `register'
>> /usr/lib/ruby/vendor_ruby/chef/client.rb:280:in `register'
>> /usr/lib/ruby/vendor_ruby/chef/client.rb:150:in `run'
>> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:239:in `run_application'
>> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in `loop'
>> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in `run_application'
>> /usr/lib/ruby/vendor_ruby/chef/application.rb:67:in `run'
>> /usr/bin/chef-client:25
>>
>>
>> But, how could that client be there if it didn't exist a moment ago?
>> Turns out, now when I run "knife client list", this client appears:
>>
>> :~$ knife client list
>>  chef-validator
>>  chef-webui
>>  devserverconfig
>>  devuser
>>
>> But it's not a node because "knife node list" is empty. So now when I
>> try to delete this "ghost" client, I get the following:
>>
>> :~$ knife client delete devserverconfig
>> Do you really want to delete devserverconfig? (Y/N) Y
>> ERROR: Server returned error for
>> http://localhost:4000/clients/devserverconfig, retrying 1/5 in 3s
>> ERROR: The object you are looking for could not be found
>> Response: Cannot load client devserverconfig
>>
>> :~$ knife client list
>>  chef-validator
>>  chef-webui
>>  devuser
>>
>>
>> As you can see, there's an error deleting but it does get deleted.
>> Anyway, at this point I'm stuck and not able to add any nodes to this
>> chef server with a new hostname. I've posted excerpts from the
>> /var/log/chef/server.log file that gives more details at the errors:
>> https://gist.github.com/1515217. I'm still experimenting with Chef
>> right now, so I'm willing to reinstall everything, but it'd be good to
>> understand what's going on.
>>
>> Any help would be greatly appreciated. Thank you in advance.