[chef] Odd behavior with Chef Server

[Aaron Abramson < >]

Hello,

First time on the list...

I have a vanilla chef-server deployed on Ubuntu 11.04 on a m1.small EC2 instance.  I just followed the opscode ubuntu apt-get install instructions.

My chef-server EC2 instance is associated with an elasticIP, and chef.mydomain.com is a CNAME to the public DNS of the elastic IP.  I'm pretty new to both EC2 and Chef, so I'm not sure if this is an EC2 security group issue, or a chef-server issue.

Everything worked at first, but now I'm getting errors; when deleting nodes or clients from the chef-server, I get errors every time, but they are successfully deleted:

aabramson$ knife client list

  aabramson

  chef-validator

  chef-webui

  ip-10-90-186-175.ec2.internal

  retest

  ubuntu

  yetanothertest

aabramson$ knife client delete yetanothertest

Do you really want to delete yetanothertest? (Y/N) y

ERROR: Server returned error for http://chef.mydomain.com:4000/clients/yetanothertest, retrying 1/5 in 3s

ERROR: The object you are looking for could not be found

Response: Cannot load client yetanothertest

aabramson$ knife client delete ip-10-90-186-175.ec2.internal

Do you really want to delete ip-10-90-186-175.ec2.internal? (Y/N) y

ERROR: Server returned error for http://chef.mydomain.com:4000/clients/ip-10-90-186-175.ec2.internal, retrying 1/5 in 3s

ERROR: The object you are looking for could not be found

Response: Cannot load client ip-10-90-186-175.ec2.internal

aabramson$ knife client list

  aabramson

  chef-validator

  chef-webui

  retest

  ubuntu

aabramson$ 

And now when I try to launch/bootstrap new instances on EC2, the bootstrap fails:

aabramson$ knife ec2 server create -I ami-06ad526f -f t1.micro -S mykeypair -G default,management,www -Z us-east-1a -N newinstance -x ubuntu

.

.

.

ec2-67-202-33-226.compute-1.amazonaws.com Successfully installed chef-0.10.0

ec2-67-202-33-226.compute-1.amazonaws.com 15 gems installed

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:03 +0000] INFO: *** Chef 0.10.0 ***

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:03 +0000] INFO: Client key /etc/chef/client.pem is not present - registering

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:08 +0000] INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:08 +0000] ERROR: Server returned error for http://chef.mydomain.com:4000/clients, retrying 1/5 in 3s

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] INFO: HTTP Request Returned 409 Conflict: Client already exists

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action.

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] FATAL: Net::HTTPServerException: 403 "Forbidden"

Instance ID: i-2b87554a

Flavor: t1.micro

Image: ami-06ad526f

Availability Zone: us-east-1a

Security Groups: 

Public DNS Name: ec2-67-202-33-226.compute-1.amazonaws.com

Public IP Address: 67.202.33.226

Private DNS Name: domU-12-31-38-04-E1-8F.compute-1.internal

SSH Key: wfg-ec2keypair

Private IP Address: 10.220.226.125

Root Device Type: ebs

Root Volume ID: vol-b9585ad2

Root Device Name: /dev/sda1

Root Device Delete on Terminate: true

Environment: _default

Run List: 

abramson$ knife client list

  aabramson

  chef-validator

  chef-webui

  newinstance

  retest

  ubuntu

aabramson$ ssh  ">

Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-virtual i686)

.

..

:~$ sudo chef-client

[Thu, 21 Jul 2011 16:51:33 +0000] INFO: *** Chef 0.10.0 ***

[Thu, 21 Jul 2011 16:51:33 +0000] INFO: Client key /etc/chef/client.pem is not present - registering

[Thu, 21 Jul 2011 16:51:33 +0000] INFO: HTTP Request Returned 409 Conflict: Client already exists

[Thu, 21 Jul 2011 16:51:33 +0000] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action.

[Thu, 21 Jul 2011 16:51:33 +0000] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

[Thu, 21 Jul 2011 16:51:33 +0000] FATAL: Net::HTTPServerException: 403 "Forbidden"

:~$