[chef] Re: Failing to configure a client via user-data on Amazon EC2

[Clif Smith < >]

On Jul 28, 2011, at 11:30 AM, Daniel DeLeo wrote:

> On Thursday, July 28, 2011 at 8:56 AM, 
> 
 
>  wrote:
>> I'm failing to configure a client via user-data on Amazon EC2. The 2 
>> biggest
>> issues I see in the log are "Cannot load environment prod" (i haven't 
>> setup an
>> env yet, should i?) and "Connection refused connecting to localhost:4000" 
>> which
>> isn't the server and not what's set in the client.rb.
> You shouldn't need to worry about environments until you're ready to use 
> them.
> 
>> Here's my client.rb:
>> log_level :info
>> log_location "/var/log/chef/client.log"
>> ssl_verify_mode :verify_none
>> validation_client_name "chef-validator"
>> validation_key "/etc/chef/validation.pem"
>> client_key "/etc/chef/client.pem"
>> chef_server_url "http://chef.pri.mydomain.com:4000";
>> file_cache_path "/var/cache/chef"
>> file_backup_path "/var/backups/chef"
>> pid_file "/var/run/chef/client.pid"
>> node_name "i-e5912c84"
>> Chef::Log::Formatter.show_time = true
>> 
>> Here's my client.log
>> # Logfile created on Thu Jul 28 15:22:42 +0000 2011 by logger.rb/22285
>> [Thu, 28 Jul 2011 15:22:42 +0000] INFO: Daemonizing..
>> [Thu, 28 Jul 2011 15:22:42 +0000] INFO: Forked, in 6051. Priveleges: 0 0
>> [Thu, 28 Jul 2011 15:22:44 +0000] INFO: *** Chef 0.10.2 ***
>> [Thu, 28 Jul 2011 15:22:45 +0000] INFO: Client key /etc/chef/client.pem is 
>> not
>> present - registering
>> [Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not 
>> Found:
>> Cannot load node i-e5912c84
>> [Thu, 28 Jul 2011 15:22:47 +0000] INFO: Setting the run_list to
>> ["role[ubuntu-11_04-prod-worker-sched-mail]"] from JSON
>> [Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List is
>> [role[ubuntu-11_04-prod-worker-sched-mail]]
>> [Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List expands to
>> [ubuntu-11_04-prod-worker-sched-mail]
>> [Thu, 28 Jul 2011 15:22:47 +0000] INFO: Starting Chef Run for i-e5912c84
>> [Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not 
>> Found:
>> Cannot load environment prod
>> [Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Running exception handlers
>> [Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Saving node information to
>> /var/cache/chef/failed-run-data.json
>> [Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Exception handlers complete
>> [Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Stacktrace dumped to
>> /var/cache/chef/chef-stacktrace.out
>> [Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Net::HTTPServerException: 404 "Not
>> Found"
>> [Thu, 28 Jul 2011 15:22:50 +0000] INFO: *** Chef 0.10.2 ***
>> [Thu, 28 Jul 2011 15:22:51 +0000] ERROR: Connection refused connecting to
>> localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 1/5
>> [Thu, 28 Jul 2011 15:22:56 +0000] ERROR: Connection refused connecting to
>> localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 2/5
>> [Thu, 28 Jul 2011 15:23:01 +0000] ERROR: Connection refused connecting to
>> localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 3/5
>> [Thu, 28 Jul 2011 15:23:06 +0000] ERROR: Connection refused connecting to
>> localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 4/5
>> [Thu, 28 Jul 2011 15:23:11 +0000] ERROR: Connection refused connecting to
>> localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 5/5
>> [Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Errno::ECONNREFUSED: Connection
>> refused - Connection refused connecting to localhost:4000 for
>> /nodes/ip-10-196-89-221.ec2.internal, giving up
>> [Thu, 28 Jul 2011 15:23:16 +0000] FATAL: Stacktrace dumped to
>> /var/chef/cache/chef-stacktrace.out
>> [Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Sleeping for 1800 seconds before
>> trying again
> It appears that a different configuration is being used between these two 
> runs. Could anything be changing the client.rb file? Also, how are you 
> invoking chef-client, i.e., what's the command line? And how are you 
> installing chef? Is it baked in to your AMI? 
> 
> Thanks,
> Dan
> 
I don't think anything is changing the file:
# ls -al /etc/chef/client.rb 
-rw-r----- 1 root root 484 2011-07-28 16:40 /etc/chef/client.rb

# head /var/log/chef/client.log 
# Logfile created on Thu Jul 28 16:40:44 +0000 2011 by logger.rb/22285

I'm wanting to use a base Ubuntu AMI.  Because of this I'm using the 
following script which is passed to the system upon initial boot up (via 
Amazon EC2's user-data): 

#!/bin/bash
###################################################
## Setup our logfile
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 
2>&1
###################################################
## Setup the OpsCode apt repo
echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" ;>> 
/etc/apt/sources.list.d/opscode.list
mkdir -p /etc/apt/trusted.gpg.d
gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
gpg --export 

 
 | tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
apt-get update
apt-get install opscode-keyring # permanent upgradeable keyring
###################################################
APT_GET="env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get 
-q"
#$APT_GET -y remove ruby1.8*
$APT_GET -y install ruby ruby-dev libruby
$APT_GET -y install build-essential
$APT_GET -y install chef
#ln -sf gem1.9.1 /usr/bin/gem
#gem install --no-rdoc --no-ri chef
#ln -sf ruby1.9.1 /usr/bin/ruby
mkdir -p /var/log/chef
mkdir -p /var/backups/chef
mkdir -p /var/run/chef
mkdir -p /var/cache/chef
mkdir -p /var/lib/chef
#mkdir /etc/chef
#ln -s /var/lib/gems/1.9.1/bin/chef-client /usr/bin/chef-client
cat - >/etc/chef/bootstrap.json <<EOF
{ 
"run_list": [
"role[ubuntu-11_04-prod-worker-sched-mail]"
],
"default_attributes": {
},
"override_attributes": {
} 
} 
EOF
cat - >/etc/chef/client.rb <<EOF
log_level          :info
log_location       "/var/log/chef/client.log"
ssl_verify_mode    :verify_none
validation_client_name "chef-validator"
validation_key         "/etc/chef/validation.pem"
client_key               "/etc/chef/client.pem"
chef_server_url    "http://chef.pri.mydomain.com:4000";
file_cache_path    "/var/cache/chef"
file_backup_path  "/var/backups/chef"
pid_file           "/var/run/chef/client.pid"
node_name       "`curl http://169.254.169.254/latest/meta-data/instance-id`";
Chef::Log::Formatter.show_time = true
EOF
cat - >/etc/chef/validation.pem <<EOF
-----BEGIN RSA PRIVATE KEY-----
<snip>
-----END RSA PRIVATE KEY-----
EOF
/usr/bin/chef-client -j /etc/chef/bootstrap.json