[chef] Re: Re: Re: reconfiguring server vs continuous upgrades

[Ranjib Dey < >]

If you are on some cloud provider, +1 for trashing the node. It  not only ensures that you have a clean system to begin with, it also ensures you have the ability to spawn instances, whenever required,

On Fri, Apr 12, 2013 at 12:22 PM, Geoff Papilion < " target="_blank"> > wrote:

Trash the node, and everything with it. You really want to pretend its
a new system.

Upgrades typically contain nasty surprises, and finding the edge cases
for them can be pretty difficult. Also, this method gets rid of any
admin cruft (i.e. random apt-get install ) that might be there.

I don't bother trying to save the pem files, since its easy enough to
blow away the client records and start over.

//geoff

On Fri, Apr 12, 2013 at 12:13 PM, Andrew Gross < "> > wrote:
> Hey Spike,
>
> Our workflow for replacing nodes is to completely trash them and start from
> scratch.  New node, new client, fresh bootstrapping.
>
> A few things that make this possible for us:
>
> Base Images:  We start with an Amazon AMI that has our validation key.  We
> bootstrap the node, get the client key, and remove the master key.
>
> No unique node data: No node.set for Normal attributes, everything is set
> before the Chef run starts.  There is no data in Chef that depends on the
> "state" of the node.
>
> I would recommend discarding the Client information on the old nodes as you
> bring up their replacements. IMO you should not view nodes/clients as
> anything more than disposable identifiers/authentication information.  The
> bonus of this approach is that if you have the spare capacity, you can bring
> up the new nodes before you remove the old ones, allowing you to guard for
> failures for any particularly fragile machines.
>
> Andrew
>
>
>
> On Fri, Apr 12, 2013 at 3:02 PM, Spike Grobstein < "> >
> wrote:
>>
>> ohai chefs,
>>
>> We manage around 80 servers using chef-server that we host internally and
>> have been doing so for about 18 months. Until a couple months ago, our base
>> image was Ubuntu 11.04, but, in an effort to stay up to date and have access
>> to the latest packages and security features, I moved that up to 12.04.
>> Starting this week, I've begun to upgrade our servers using
>> `do-release-upgrade` and after getting through 4 nodes, I started
>> thinking...
>>
>> Is there a better way to do this? Right now, I've got to ssh into each
>> node, run do-release-upgrade (11.04 -> 11.10), answer some questions, let it
>> run, tell it to replace some files, tell it to not replace some files, let
>> it run some more... let it finish, reboot, run it again (11.10 -> 12.04) and
>> do the same thing again.
>>
>> Would it be better to just trash the node and configure a new one in place
>> from the image using our chef recipes?
>>
>> If I did the latter, I'm not entirely sure what the workflow would be. The
>> node would have to have the client.pem file from the old node, but would
>> that Just Work™?
>>
>> The advantages of configuring from our base image are that it ensures that
>> our cookbooks are all up to date. Many of these nodes were configured over a
>> year ago using completely different cookbooks and updates have been run on
>> top of already configured nodes. The only reason I'm confident that this
>> will work is that I've configured nodes from most of the roles recently on
>> the new ubuntu.
>>
>> So, what do you guys do in cases like this? What's the workflow look like?
>> Is it as easy as just saving the client.pem, zapping the node, cloning from
>> the base image template, putting client.pem back and running chef-client?
>> Are there any gotchas I should be worried about?
>>
>> Or is there a better way?
>>
>> Thanks!
>>
>>
>>
>> ...spike
>
>

--
//geoff