[chef] reconfiguring server vs continuous upgrades

[Spike Grobstein < >]

ohai chefs,

We manage around 80 servers using chef-server that we host internally and 
have been doing so for about 18 months. Until a couple months ago, our base 
image was Ubuntu 11.04, but, in an effort to stay up to date and have access 
to the latest packages and security features, I moved that up to 12.04. 
Starting this week, I've begun to upgrade our servers using 
`do-release-upgrade` and after getting through 4 nodes, I started thinking...

Is there a better way to do this? Right now, I've got to ssh into each node, 
run do-release-upgrade (11.04 -> 11.10), answer some questions, let it run, 
tell it to replace some files, tell it to not replace some files, let it run 
some more... let it finish, reboot, run it again (11.10 -> 12.04) and do the 
same thing again.

Would it be better to just trash the node and configure a new one in place 
from the image using our chef recipes?

If I did the latter, I'm not entirely sure what the workflow would be. The 
node would have to have the client.pem file from the old node, but would that 
Just Work™?

The advantages of configuring from our base image are that it ensures that 
our cookbooks are all up to date. Many of these nodes were configured over a 
year ago using completely different cookbooks and updates have been run on 
top of already configured nodes. The only reason I'm confident that this will 
work is that I've configured nodes from most of the roles recently on the new 
ubuntu.

So, what do you guys do in cases like this? What's the workflow look like? Is 
it as easy as just saving the client.pem, zapping the node, cloning from the 
base image template, putting client.pem back and running chef-client? Are 
there any gotchas I should be worried about?

Or is there a better way?

Thanks!



...spike