[chef] Re: Re: Re: chef-client memory usage

[Brad Knowles < >]

On Dec 8, 2011, at 5:09 PM, Chris wrote:

> I agree with you, there are other problems surrounding our memory 
> constraints. Namely, I'm not allowed to take memory away from Java and Java 
> is never allowed to go into swap. We routinely provision 8GB VMs and give 
> Java 7 of that, and the Java dev teams won't budge on that. 

Ouch.  Yeah, politics is going to hurt.

> I'm really surprised that your client is so lean, but you're CentOS version 
> is also newer then ours. We have systems ranging from 5.2-5.5, most have 
> never been patched. I'm pretty sure we're missing out on some 
> optimizations. I'm curious, are most of your cookbooks from Opscode or home 
> grown?

We started with the Opscode cookbooks, but some of them have been modified 
pretty heavily, and we've created a few of our own.  Some of the cookbooks 
have come from elsewhere on github, with some local modifications.

I'm hoping that I will be able to get up to speed enough on git in the near 
future that we can contribute pretty much all our work back to github, so 
that the community can upgrade the Opscode cookbooks as appropriate, or the 
few non-Opscode cookbooks that we have used.  Most importantly, I think we 
may be the first non-Ubuntu site to make use of the edelight cookbook for 
MongoDB, and I'd really like to get our enhancements folded back in.

We've already signed all the CLA and CCLA forms, so now it's more a matter of 
me finding the time and inclination to "git" up to speed.



One other observation I'd like to make about VSS & RSS -- I think this might 
depend on your HyperVisor implementation, but I would be surprised if you 
didn't get "shared" pages with multiple different VMs on the system each with 
their own copy of chef-client that is running.

So, the real memory impact would not be the number of VMs times the VSS (as 
they claimed), but more like some cost-reduction factor times the number of 
VMs times the RSS for each of those chef-client instances.  The more VMs 
you've got on a single machine, I think the more memory overall that you 
would save as a result of getting something akin to deduplication being 
performed by the HyperVisor in conjunction with whatever OS is running in 
Ring Zero.  All they need to do is implement standard "copy-on-write" 
functionality for each of the affected pages.

-- 
Brad Knowles 
<
 >
SAGE Level IV, Chef Level 0.0.1