[chef] rabbitmq connection error on newly installed chef server

[Haim Ashkenazi < >]

Hi

I've just installed chef server from gems on ubuntu 10.04 (like I did numerous times before) but this time something went wrong, chef-server refuses to start because of errors connecting to rabbitmq:

2012-07-31_09:39:31.03184 [2012-07-31T05:39:31-04:00] INFO: Creating validation key...

2012-07-31_09:39:31.03195 [2012-07-31T05:39:31-04:00] INFO: Creating new key pair for chef-validator

2012-07-31_09:39:31.36981 [2012-07-31T05:39:31-04:00] INFO: Sending client(ca1c95c7-cdfc-4ad3-8aae-9b7a8d953050) to the index queue for addition.

2012-07-31_09:39:31.45811 [2012-07-31T05:39:31-04:00] FATAL: Connection to rabbitmq refused. Check your rabbitmq configuration and chef's amqp* settings

2012-07-31_09:39:31.45822 /usr/lib/ruby/gems/1.8/gems/bunny-0.8.0/lib/bunny/client.rb:117:in `init_connection': Connection initiation failed (Bunny::ProtocolError)

2012-07-31_09:39:31.45824       from /usr/lib/ruby/gems/1.8/gems/bunny-0.8.0/lib/bunny/client.rb:298:in `start'

2012-07-31_09:39:31.45825       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/index_queue/amqp_client.rb:45:in `amqp_client'

2012-07-31_09:39:31.45826       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/index_queue/amqp_client.rb:72:in `queue_for_object'

2012-07-31_09:39:31.45826       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/index_queue/indexable.rb:95:in `publish_object'

2012-07-31_09:39:31.45827       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/index_queue/indexable.rb:74:in `add_to_index'

2012-07-31_09:39:31.45828       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/couchdb.rb:118:in `store'

2012-07-31_09:39:31.45832       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/api_client.rb:231:in `cdb_save'

2012-07-31_09:39:31.45835       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/certificate.rb:139:in `create_validation_key'

2012-07-31_09:39:31.45836       from /usr/lib/ruby/gems/1.8/gems/chef-10.12.0/lib/chef/certificate.rb:114:in `gen_validation_key'

2012-07-31_09:39:31.45837       from /usr/lib/ruby/gems/1.8/gems/chef-server-api-10.12.0/config/init.rb:84

2012-07-31_09:39:31.45838       from /usr/lib/ruby/gems/1.8/gems/merb-core-1.1.3/lib/merb-core/bootloader.rb:529:in `load'

2012-07-31_09:39:31.45838       from /usr/lib/ruby/gems/1.8/gems/merb-core-1.1.3/lib/merb-core/bootloader.rb:529:in `load_initfile'

2012-07-31_09:39:31.45839       from /usr/lib/ruby/gems/1.8/gems/merb-core-1.1.3/lib/merb-core/bootloader.rb:377:in `run'

2012-07-31_09:39:31.45840       from /usr/lib/ruby/gems/1.8/gems/merb-core-1.1.3/lib/merb-core/bootloader.rb:97:in `run'

2012-07-31_09:39:31.45840       from /usr/lib/ruby/gems/1.8/gems/merb-core-1.1.3/lib/merb-core/server.rb:172:in `bootup'

2012-07-31_09:39:31.45841       from /usr/lib/ruby/gems/1.8/gems/merb-core-1.1.3/lib/merb-core/server.rb:42:in `start'

2012-07-31_09:39:31.45842       from /usr/lib/ruby/gems/1.8/gems/merb-core-1.1.3/lib/merb-core.rb:165:in `start'

2012-07-31_09:39:31.45842       from /usr/lib/ruby/gems/1.8/gems/chef-server-api-10.12.0/bin/chef-server:84

2012-07-31_09:39:31.45843       from /usr/bin/chef-server:19:in `load'

2012-07-31_09:39:31.45845       from /usr/bin/chef-server:19

On the rabbitmq log I see:

=INFO REPORT==== 31-Jul-2012::05:41:43 ===

starting TCP connection <0.1468.0> from 127.0.0.1:41019

=ERROR REPORT==== 31-Jul-2012::05:41:43 ===

exception on TCP connection <0.1468.0> from 127.0.0.1:41019

{bad_header,<<65,77,81,80,0,0,9,1>>}

=INFO REPORT==== 31-Jul-2012::05:41:43 ===

closing TCP connection <0.1468.0> from 127.0.0.1:41019

=INFO REPORT==== 31-Jul-2012::05:41:45 ===

accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:41034

=INFO REPORT==== 31-Jul-2012::05:41:45 ===

starting TCP connection <0.1471.0> from 127.0.0.1:41034

=ERROR REPORT==== 31-Jul-2012::05:41:45 ===

exception on TCP connection <0.1471.0> from 127.0.0.1:41034

{bad_header,<<65,77,81,80,0,0,9,1>>}

=INFO REPORT==== 31-Jul-2012::05:41:45 ===

closing TCP connection <0.1471.0> from 127.0.0.1:41034

=INFO REPORT==== 31-Jul-2012::05:41:48 ===

accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:41051

=INFO REPORT==== 31-Jul-2012::05:41:48 ===

starting TCP connection <0.1475.0> from 127.0.0.1:41051

=ERROR REPORT==== 31-Jul-2012::05:41:48 ===

exception on TCP connection <0.1475.0> from 127.0.0.1:41051

{bad_header,<<65,77,81,80,0,0,9,1>>}

=INFO REPORT==== 31-Jul-2012::05:41:48 ===

closing TCP connection <0.1475.0> from 127.0.0.1:41051

rabbitmq version is: 1.7.2-1ubuntu1

chef-server version is: 10.12.0

Any ideas how to debug/solve this?

Thanks in advance

--
Haim