[chef] Re: Re: Re: Re: Installing large numbers of packages

[Matt Palmer < >]

On Thu, Sep 1, 2011 at 10:34 AM, Brad Knowles 
<
 >
 wrote:
> On Aug 31, 2011, at 7:08 PM, Matt Palmer wrote:
>> There's also resilience issues, a strong aversion to centralisation,
>> and too many painful memories of Puppet scaling nightmares to get
>> over.
>
> Well, all CM systems are about centralization, regularization, 
> categorization, and management of information, so I don't think you're 
> going to get away from that.  In the case of Chef, what you're doing is 
> trying to get all this information about your internal systems & network 
> infrastructure recorded into a reliable and version-controlled CM system.

Configuration management doesn't imply reliability on a giant central
server infrastructure that's going to have to be scaled and managed
itself.

>> Even without that, though, I'm having trouble working out how it's
>> better to have a list of packages in one place, and a resource
>> specification that installs those packages somewhere else.  I can
>> almost convince myself that putting attributes in an external JSON
>> file makes sense for roles (although I think it's codifying the same
>> mistake that practically everyone makes using Puppet, where you define
>> a pile of global variables and cross your fingers that everything
>> works, rather than having locally-passed parameters that define how
>> you want to use something Here and Now), but making a list somewhere
>> external just so I can avoid having to walk an array is insane.  My
>> recipe says "this is how you configure a workstation", and the list of
>> packages you have to install in order to do that should be in that
>> recipe.
>
> I think it comes down to a separation of "code" from "data".  You should 
> put your "code" into a code repository, but when the data that the code is 
> going to be operating needs to be changed, you shouldn't necessarily have 
> to change the code just to accommodate the change in the data.

My system configuration is *all* data... "this is what I want to
happen".  A list of packages is no more code or data than the fact
that I want those packages installed, and not removed.  I want to
revision control it all.

> The installation script should be simple and easy to read, regardless of 
> how many packages are being installed -- that information should come from 
> the database.  And when all that is changing is the data, it should work as 
> you want with the existing code that is already in place on all your nodes.

The installation script *is* simple and easy to read, if the syntax is
appropriate.

> When you're talking about a small number of nodes to be managed, I'm not 
> sure that this makes such a difference.  But as you try to scale up, it's 
> going to become more and more important that you keep this separation 
> between code & data.
>
> So, do you want to learn the right way from Day One, or do you want to 
> learn a single-file method that you will have to unlearn as you try to 
> scale up?

And this is where I feel like I should stop listening to you, because
you assume I've never managed large scale systems.  I've done 500+
nodes with Puppet, and 2,500+ systems under management in bodgy
semi-manual ways.

If there are better ways to do it with Chef, I'm open to them, but I
*do* have plenty of experience in this field, and so far my
experiences are telling me that the way Chef does it is a monumental
pain in the arse at scale.  However, I'm willing to learn that I'm
wrong, so point me at the documentation that explains clearly and
simply why the Chef way works better.

> Maybe your problem with CM systems isn't with the systems themselves, but 
> with the way you're trying to use them -- or maybe misuse them?

Perhaps.  Feel free to show me where I'm wrong, but with specifics,
not platitudes.

- Matt