[chef] Re: Re: Re: Change hostname and IP address on Chef server

[Arthur Kalmenson < >]

Thanks Tom, it looks like it was printing to STDOUT, so I changed it
to a specific file location and got this:

[Wed, 28 Dec 2011 09:08:55 -0500] INFO: Creating validation key...
[Wed, 28 Dec 2011 09:08:55 -0500] INFO: Creating new key pair for 
chef-validator
[Wed, 28 Dec 2011 09:08:55 -0500] INFO: Sending
client(5074ede6-ef8c-46d6-b3a0-5902a80641b3) to the index queue for
addition.
[Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not connect to
rabbitmq. Is it running, reachable, and configured correctly?
[Wed, 28 Dec 2011 09:08:55 -0500] ERROR: Disconnected from the AMQP
Broker (RabbitMQ)
[Wed, 28 Dec 2011 09:08:55 -0500] INFO: Attempting to reconnect to the
AMQP broker
[Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not connect to
rabbitmq. Is it running, reachable, and configured correctly?
[Wed, 28 Dec 2011 09:08:55 -0500] ERROR: Disconnected from the AMQP
Broker (RabbitMQ)
[Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not re-connect to the
AMQP broker, giving up
[Wed, 28 Dec 2011 09:08:55 -0500] FATAL: Could not index (to rabbitmq)
validation key - rabbitmq is down! Start rabbitmq then restart
chef-server to re-generate it


Turns out RabbitMQ cannot be started anymore, I'm getting:

starting database
...{"init terminating in
do_boot",{{nocatch,{error,{cannot_start_application,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,{timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_vhost,rabbit_config,rabbit_listener,rabbit_durable_route,rabbit_route,rabbit_reverse_route,rabbit_durable_exchange,rabbit_exchange,rabbit_durable_queue,rabbit_queue]}}},[{rabbit,'-run_boot_step/1-lc$^1/1-1-',1},{rabbit,run_boot_step,1},{rabbit,'-start/2-lc$^0/1-0-',1},{rabbit,start,2},{application_master,start_it_old,4}]}}}}}}},[{init,start_it,1},{init,start_em,1}]}}


Which seems to be related to the hostname change. To get RabbitMQ to
start up again, I deleted everything under "/var/lib/rabbitmq/mnesia"
and RabbitMQ has started up successfully. However, when I try to start
up the chef server, I'm still getting:

[Wed, 28 Dec 2011 09:52:41 -0500] INFO: Creating validation key...
[Wed, 28 Dec 2011 09:52:41 -0500] INFO: Creating new key pair for 
chef-validator
[Wed, 28 Dec 2011 09:52:42 -0500] INFO: Sending
client(704cd294-2384-4153-a82d-dd98c37d0634) to the index queue for
addition.
[Wed, 28 Dec 2011 09:52:42 -0500] FATAL: Connection to rabbitmq
refused. Check your rabbitmq configuration and chef's amqp* settings

--
Arthur Kalmenson



On Tue, Dec 27, 2011 at 12:16 PM, Tom Thomas 
<
 >
 wrote:
> From http://wiki.opscode.com/display/chef/Support
>
> Log file locations
> Check the server or client config file for the log_location.
>
> /etc/chef/server.rb
> /etc/chef/client.rb
>
> - Tom
>
>
>
>
> On Tue, Dec 27, 2011 at 9:15 AM, Arthur Kalmenson 
> <
 >
> wrote:
>>
>> Any ideas where I could find logs to see why chef-server is failing to
>> start and unable to recreate the validation.pem? Thank you.
>>
>> --
>> Arthur Kalmenson
>>
>>
>>
>> On Fri, Dec 23, 2011 at 3:40 PM, Arthur Kalmenson 
>> <
 >
>> wrote:
>> > jessicab has been helping me out on IRC and pointed me to
>> >
>> > http://wiki.opscode.com/display/chef/Common+Errors#CommonErrors-401Unauthorized%28usingvalidatorAPIclient%29.
>> > I tried recreating the validation.pem as follows:
>> >
>> > 
 :/etc/chef$
>> >  sudo rm validation.pem
>> > 
 :/etc/chef$
>> >  sudo /etc/init.d/chef-server restart
>> >  * Restarting chef-server
>> >                                                  ~ Killing pid 1407
>> > with INT
>> >  ~ In 15699
>> >
>> >                                          [ OK ]
>> >
>> > But the validation.pem was not recreated. It looks like the chef
>> > server failed to start:
>> >
>> > 
 :/etc/chef$
>> >  sudo /etc/init.d/chef-server status
>> >  * Checking status of chef-server
>> >                                          [fail]
>> > 
 :/etc/chef$
>> >  sudo /etc/init.d/chef-server start
>> >  * Starting chef-server
>> >                                                  ~ In 16771
>> >
>> >                                          [fail]
>> >
>> > I've tried looking around /var/log/chef and /var/log/ to see if
>> > anything is getting logged, but I didn't find anything.
>> > --
>> > Arthur Kalmenson
>> >
>> >
>> >
>> > On Fri, Dec 23, 2011 at 3:08 PM, Arthur Kalmenson
>> > <
 >
>> >  wrote:
>> >> Hello everyone,
>> >>
>> >> I'm trying to change the hostname and IP address of the Chef server
>> >> that I set up. My steps have been the following:
>> >>
>> >> 1. Delete all the nodes and clients (except chef-validator, chef-webui
>> >> and the user on the chef server using knife).
>> >> 2. Change the hostname, /etc/hosts, and the IP address. Reboot.
>> >> 3. At this point, I remove the client.pem file from all the nodes,
>> >> update the client.rb to point to the new hostname, and try to re-add
>> >> them as nodes by running "sudo chef-client"
>> >>
>> >> This is where I've been running into trouble.
>> >>
>> >> 
 :~$
>> >>  knife client list
>> >>  chef-validator
>> >>  chef-webui
>> >>  devuser
>> >>
>> >> As you can see, the only remaining clients are chef-validator,
>> >> chef-webui and devuser (the knife user on the chef server). Running
>> >> "knife node list" is empty.
>> >>
>> >> The first node I want to add back on is the chef server itself, so I
>> >> delete /etc/chef/client.pem and run "sudo chef-client"
>> >>
>> >> 
 :~$
>> >>  sudo chef-client
>> >> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: *** Chef 0.10.8 ***
>> >> [Fri, 23 Dec 2011 14:50:17 -0500] INFO: Client key
>> >> /etc/chef/client.pem is not present - registering
>> >> [Fri, 23 Dec 2011 14:50:18 -0500] INFO: HTTP Request Returned 500
>> >> Internal Server Error: Connection refused - connect(2)
>> >> [Fri, 23 Dec 2011 14:50:18 -0500] ERROR: Server returned error for
>> >> http://devserverconfig:4000/clients, retrying 1/5 in 3s
>> >> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 409
>> >> Conflict: Client already exists
>> >> [Fri, 23 Dec 2011 14:50:21 -0500] INFO: HTTP Request Returned 403
>> >> Forbidden: You are not allowed to take this action.
>> >> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Stacktrace dumped to
>> >> /var/cache/chef/chef-stacktrace.out
>> >> [Fri, 23 Dec 2011 14:50:21 -0500] FATAL: Net::HTTPServerException: 403
>> >> "Forbidden"
>> >>
>> >> 
 :~$
>> >>  cat /var/cache/chef/chef-stacktrace.out
>> >> Generated at Fri Dec 23 14:50:21 -0500 2011
>> >> Net::HTTPServerException: 403 "Forbidden"
>> >> /usr/lib/ruby/1.8/net/http.rb:2101:in `error!'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in `api_request'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in `retriable_rest_request'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in `api_request'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in `put_rest'
>> >> /usr/lib/ruby/vendor_ruby/chef/api_client.rb:247:in `save'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:81:in `register'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `upto'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:79:in `register'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `catch'
>> >> /usr/lib/ruby/vendor_ruby/chef/rest.rb:77:in `register'
>> >> /usr/lib/ruby/vendor_ruby/chef/client.rb:280:in `register'
>> >> /usr/lib/ruby/vendor_ruby/chef/client.rb:150:in `run'
>> >> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:239:in
>> >> `run_application'
>> >> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in `loop'
>> >> /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in
>> >> `run_application'
>> >> /usr/lib/ruby/vendor_ruby/chef/application.rb:67:in `run'
>> >> /usr/bin/chef-client:25
>> >>
>> >>
>> >> But, how could that client be there if it didn't exist a moment ago?
>> >> Turns out, now when I run "knife client list", this client appears:
>> >>
>> >> 
 :~$
>> >>  knife client list
>> >>  chef-validator
>> >>  chef-webui
>> >>  devserverconfig
>> >>  devuser
>> >>
>> >> But it's not a node because "knife node list" is empty. So now when I
>> >> try to delete this "ghost" client, I get the following:
>> >>
>> >> 
 :~$
>> >>  knife client delete devserverconfig
>> >> Do you really want to delete devserverconfig? (Y/N) Y
>> >> ERROR: Server returned error for
>> >> http://localhost:4000/clients/devserverconfig, retrying 1/5 in 3s
>> >> ERROR: The object you are looking for could not be found
>> >> Response: Cannot load client devserverconfig
>> >>
>> >> 
 :~$
>> >>  knife client list
>> >>  chef-validator
>> >>  chef-webui
>> >>  devuser
>> >>
>> >>
>> >> As you can see, there's an error deleting but it does get deleted.
>> >> Anyway, at this point I'm stuck and not able to add any nodes to this
>> >> chef server with a new hostname. I've posted excerpts from the
>> >> /var/log/chef/server.log file that gives more details at the errors:
>> >> https://gist.github.com/1515217. I'm still experimenting with Chef
>> >> right now, so I'm willing to reinstall everything, but it'd be good to
>> >> understand what's going on.
>> >>
>> >> Any help would be greatly appreciated. Thank you in advance.
>
>