[chef] Re: Change hostname and IP address on Chef server

[Steffen Gebert < >]

> It turned out to be faster to just set up the chef server again. Would
> be nice if there was an easy way to change the hostname....


Ouch, I have to do the change the DNS name of our server within the next few 
days, too :-(

We're running just a hand full of clients at the moment, so manually changing 
the address on client side would be no problem. But when I can be sure that 
the server itself will not start anymore, I'm not sure, how to proceed.

Is there an "official" recommendation? Is there an easy way to dump all data 
and restore it after new installation?

Thanks 
Steffen


On 29.12.2011, at 04:57, Arthur Kalmenson wrote:

> It turned out to be faster to just set up the chef server again. Would
> be nice if there was an easy way to change the hostname.... Anyway,
> onwards with exploring :)
> 
> All the best,
> --
> Arthur Kalmenson
> 
> 
> 
> On Wed, Dec 28, 2011 at 10:02 AM, Arthur Kalmenson
> <
 >
>  wrote:
>> Thanks Tom, it looks like it was printing to STDOUT, so I changed it
>> to a specific file location and got this:
>> 
>> [Wed, 28 Dec 2011 09:08:55 -0500] INFO: Creating validation key...
>> [Wed, 28 Dec 2011 09:08:55 -0500] INFO: Creating new key pair for 
>> chef-validator
>> [Wed, 28 Dec 2011 09:08:55 -0500] INFO: Sending
>> client(5074ede6-ef8c-46d6-b3a0-5902a80641b3) to the index queue for
>> addition.
>> [Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not connect to
>> rabbitmq. Is it running, reachable, and configured correctly?
>> [Wed, 28 Dec 2011 09:08:55 -0500] ERROR: Disconnected from the AMQP
>> Broker (RabbitMQ)
>> [Wed, 28 Dec 2011 09:08:55 -0500] INFO: Attempting to reconnect to the
>> AMQP broker
>> [Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not connect to
>> rabbitmq. Is it running, reachable, and configured correctly?
>> [Wed, 28 Dec 2011 09:08:55 -0500] ERROR: Disconnected from the AMQP
>> Broker (RabbitMQ)
>> [Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not re-connect to the
>> AMQP broker, giving up
>> [Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not index (to rabbitmq)
>> validation key - rabbitmq is down! Start rabbitmq then restart
>> chef-server to re-generate it
>> 
>> 
>> Turns out RabbitMQ cannot be started anymore, I'm getting:
>> 
>> starting database
>> ...{"init terminating in
>> do_boot",{{nocatch,{error,{cannot_start_application,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,{timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_vhost,rabbit_config,rabbit_listener,rabbit_durable_route,rabbit_route,rabbit_reverse_route,rabbit_durable_exchange,rabbit_exchange,rabbit_durable_queue,rabbit_queue]}}},[{rabbit,'-run_boot_step/1-lc$^1/1-1-',1},{rabbit,run_boot_step,1},{rabbit,'-start/2-lc$^0/1-0-',1},{rabbit,start,2},{application_master,start_it_old,4}]}}}}}}},[{init,start_it,1},{init,start_em,1}]}}
>> 
>> 
>> Which seems to be related to the hostname change. To get RabbitMQ to
>> start up again, I deleted everything under "/var/lib/rabbitmq/mnesia"
>> and RabbitMQ has started up successfully. However, when I try to start
>> up the chef server, I'm still getting:
>> 
>> [Wed, 28 Dec 2011 09:52:41 -0500] INFO: Creating validation key...
>> [Wed, 28 Dec 2011 09:52:41 -0500] INFO: Creating new key pair for 
>> chef-validator
>> [Wed, 28 Dec 2011 09:52:42 -0500] INFO: Sending
>> client(704cd294-2384-4153-a82d-dd98c37d0634) to the index queue for
>> addition.
>> [Wed, 28 Dec 2011 09:52:42 -0500] FATAL: Connection to rabbitmq
>> refused. Check your rabbitmq configuration and chef's amqp* settings
>> 
>> --
>> Arthur Kalmenson
>> 
>> 
>> 
>> On Tue, Dec 27, 2011 at 12:16 PM, Tom Thomas 
>> <
 >
>>  wrote:
>>> From http://wiki.opscode.com/display/chef/Support
>>> 
>>> Log file locations
>>> Check the server or client config file for the log_location.
>>> 
>>> /etc/chef/server.rb
>>> /etc/chef/client.rb
>>> 
>>> - Tom
>>> 
>>> 
>>> 
>>> 
>>> On Tue, Dec 27, 2011 at 9:15 AM, Arthur Kalmenson 
>>> <
 >
>>> wrote:
>>>> 
>>>> Any ideas where I could find logs to see why chef-server is failing to
>>>> start and unable to recreate the validation.pem? Thank you.
>>>> 
>>>> --
>>>> Arthur Kalmenson
>>>> 
>>>> 
>>>> 
>>>> On Fri, Dec 23, 2011 at 3:40 PM, Arthur Kalmenson 
>>>> <
 >
>>>> wrote:
>>>>> jessicab has been helping me out on IRC and pointed me to
>>>>> 
>>>>> http://wiki.opscode.com/display/chef/Common+Errors#CommonErrors-401Unauthorized%28usingvalidatorAPIclient%29.
>>>>> I tried recreating the validation.pem as follows:
>>>>> 
>>>>> 
 :/etc/chef$
>>>>>  sudo rm validation.pem
>>>>> 
 :/etc/chef$
>>>>>  sudo /etc/init.d/chef-server restart
>>>>>  * Restarting chef-server
>>>>>                                                  ~ Killing pid 1407
>>>>> with INT
>>>>>  ~ In 15699
>>>>> 
>>>>>                                          [ OK ]
>>>>> 
>>>>> But the validation.pem was not recreated. It looks like the chef
>>>>> server failed to start:
>>>>> 
>>>>> 
 :/etc/chef$
>>>>>  sudo /etc/init.d/chef-server status
>>>>>  * Checking status of chef-server
>>>>>                                          [fail]
>>>>> 
 :/etc/chef$
>>>>>  sudo /etc/init.d/chef-server start
>>>>>  * Starting chef-server
>>>>>                                                  ~ In 16771
>>>>> 
>>>>>                                          [fail]
>>>>> 
>>>>> I've tried looking around /var/log/chef and /var/log/ to see if
>>>>> anything is getting logged, but I didn't find anything.
>>>>> --
>>>>> Arthur Kalmenson
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Dec 23, 2011 at 3:08 PM, Arthur Kalmenson
>>>>> <
 >
>>>>>  wrote:
>>>>>> Hello everyone,
>>>>>> 
>>>>>> I'm trying to change the hostname and IP address of the Chef server
>>>>>> that I set up. My steps have been the following:
>>>>>> 
>>>>>> 1. Delete all the nodes and clients (except chef-validator, chef-webui
>>>>>> and the user on the chef server using knife).
>>>>>> 2. Change the hostname, /etc/hosts, and the IP address. Reboot.
>>>>>> 3. At this point, I remove the client.pem file from all the nodes,
>>>>>> update the client.rb to point to the new hostname, and try to re-add
>>>>>> them as nodes by running "sudo chef-client"
>>>>>> 
>>>>>> This is where I've been running into trouble.
>>>>>> 
>>>>>> 
 :~$
>>>>>>  knife client list
>>>>>>  chef-validator
>>>>>>  chef-webui
>>>>>>  devuser
>>>>>> 
>>>>>> As you can see, the only remaining clients are chef-validator,
>>>>>> chef-webui and devuser (the knife user on the chef server). Running
>>>>>> "knife node list" is empty.
>>>>>> 
>>>>>> The first node I want to add back on is the chef server itself, so I
>>>>>> delete /etc/chef/client.pem and run "sudo chef-client"
>>>>>> 
>>>>>> 
 :~$
>>>>>>  sudo chef-client
>>>>>> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: *** Chef 0.10.8 ***
>>>>>> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: Client key
>>>>>> /etc/chef/client.pem is not present - registering
>>>>>> [Fri, 23 Dec 2011 14:50:18 -0500] INFO: HTTP Request Returned 500
>>>>>> Internal Server Error: Connection refused - connect(2)
>>>>>> [Fri, 23 Dec 2011 14:50:18 -0500] ERROR: Server returned error for
>>>>>> http://devserverconfig:4000/clients, retrying 1/5 in 3s
>>>>>> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 409
>>>>>> Conflict: Client already exists
>>>>>> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 403
>>>>>> Forbidden: You are not allowed to take this action.
>>>>>> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Stacktrace dumped to
>>>>>> /var/cache/chef/chef-stacktrace.out
>>>>>> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Net::HTTPServerException: 403
>>>>>> "Forbidden"
>>>>>> 
>>>>>> 
 :~$
>>>>>>  cat /var/cache/chef/chef-stacktrace.out
>>>>>> Generated at Fri Dec 23 14:50:21 -0500 2011
>>>>>> Net::HTTPServerException: 403 "Forbidden"
>>>>>> /usr/lib/ruby/1.8/net/http.rb:2101:in `error!'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in `api_request'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in `retriable_rest_request'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in `api_request'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in `put_rest'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/api_client.rb:247:in `save'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:81:in `register'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `upto'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `register'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `catch'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `register'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/client.rb:280:in `register'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/client.rb:150:in `run'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:239:in
>>>>>> `run_application'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in `loop'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in
>>>>>> `run_application'
>>>>>> /usr/lib/ruby/vendor_ruby/chef/application.rb:67:in `run'
>>>>>> /usr/bin/chef-client:25
>>>>>> 
>>>>>> 
>>>>>> But, how could that client be there if it didn't exist a moment ago?
>>>>>> Turns out, now when I run "knife client list", this client appears:
>>>>>> 
>>>>>> 
 :~$
>>>>>>  knife client list
>>>>>>  chef-validator
>>>>>>  chef-webui
>>>>>>  devserverconfig
>>>>>>  devuser
>>>>>> 
>>>>>> But it's not a node because "knife node list" is empty. So now when I
>>>>>> try to delete this "ghost" client, I get the following:
>>>>>> 
>>>>>> 
 :~$
>>>>>>  knife client delete devserverconfig
>>>>>> Do you really want to delete devserverconfig? (Y/N) Y
>>>>>> ERROR: Server returned error for
>>>>>> http://localhost:4000/clients/devserverconfig, retrying 1/5 in 3s
>>>>>> ERROR: The object you are looking for could not be found
>>>>>> Response: Cannot load client devserverconfig
>>>>>> 
>>>>>> 
 :~$
>>>>>>  knife client list
>>>>>>  chef-validator
>>>>>>  chef-webui
>>>>>>  devuser
>>>>>> 
>>>>>> 
>>>>>> As you can see, there's an error deleting but it does get deleted.
>>>>>> Anyway, at this point I'm stuck and not able to add any nodes to this
>>>>>> chef server with a new hostname. I've posted excerpts from the
>>>>>> /var/log/chef/server.log file that gives more details at the errors:
>>>>>> https://gist.github.com/1515217. I'm still experimenting with Chef
>>>>>> right now, so I'm willing to reinstall everything, but it'd be good to
>>>>>> understand what's going on.
>>>>>> 
>>>>>> Any help would be greatly appreciated. Thank you in advance.
>>> 
>>>