[chef] Re: Re: Best practices for multiple data centers

[Sascha Bates < >]

We have two local data centers and are in the midst of provisioning a 3rd VMware-only data center.  We have chef servers in each environment in each location.  This was dictated in some ways by ops because the management team for stage and production is a 3rd party company and they allow almost zero access to their systems.  We keep everything for Chef in Subversion in a cloud environment and every Chef server has one firewall rule that allows it to connect to an Apache proxy that allows us to pull chef code from svn and application code coming from Artifactory.  We also version our data bags although we are discussing the wisdom of versioned data bags as it has resulted in too much config data being kept in data bags instead of attribute files.

What we don't have is a common repository for 3rd party packages and tar balls.  Our Chef servers do triple duty as Chef, package repos and kickstart servers.  Generally we put stuff into dev when requested and sync the repository to other servers as needed.  It also requires us to maintain 7 different Chef servers across 3 different data centers and we have only been so-so at automating our chef servers themselves.  I could probably do a presentation on the challenges of implementing Chef in a large firewalled environment with a very protective client.  Security is not part of the org we're in so we tend to struggle a lot with what we need and what corporate security will allow.  None of our compromises are pretty, but all were necessary and none were close to what we actually wanted.

We have a long list of to-dos that revolve around automation and sustainability of the env.  We are a small team of 2 full-timers and 3 part-timers supporting new development initiatives and new provisioning initiatives while simultaneously trying to migrate over an enormous aged infrastructure.  It's been a real challenge and I don't know if I would describe our current condition as the outcome of "best practices" but more of the best possible outcome under current constraints.  We are constantly refactoring and looking for ways to simplify and automate but it's slow going sometimes.

Sascha

On Fri, Jun 24, 2011 at 1:34 PM, Jason J. W. Williams < " target="_blank"> > wrote:

I'd be interested to find out what others are doing in this area.
Right now we host our Chef server out of Seattle and have our DCs in
Boise and NYC use the Seattle Chef server. Is anyone using Couch
replication in a ring to accomplish distribution?

-J

On Fri, Jun 24, 2011 at 12:30 PM, John Alberts < " target="_blank"> > wrote:
> I'm wondering how others use Chef for multiple datacenters and if they
> use multiple Chef servers.
> In my case, we have multiple data centers around the world and each
> data center has a private network that all of the servers are on.  I
> currently have a single Chef server at one of the data centers that
> every client connects to.  This works fine, but it does have
> disadvantages.
> 1. chef-client runs slow at data centers located on the opposite side
> of the world because of latency and bandwidth.
> 2. While the bandwidth usage right now is not much, I'm worried that
> it will be significant as our usage of Chef for various things
> increases.
>
> One obvious solution is to have a separate Chef server at each data
> center; however, I am then managing multiple Chef installations that
> use 99.9% of the same code.  The primary disadvantage of this is that
> I then have my nodes on different Chef servers and no way to search
> for all of the nodes.  Specifically, my monitoring is automated and
> needs to be able to get a list of all servers, roles, etc.  Having
> separate Chef servers really ends up creating a large barrier to
> managing all of my nodes.
>
> I'm sure others have run into something similar, so I'm wondering what
> others are doing.
>
> One thought was that it would be great if there was some sort of Chef
> proxy server that I could have at each location that cached the files
> and node data.  That way my nodes at a dc could contact the proxy for
> all of it's needs and the proxy would sync up with the main Chef
> server every so often.  Anyone working on anything like that?  :)
>
>
>
> --
> John Alberts
>