[chef] Re: Debugging memory leak issues with chef-client?

[Daniel DeLeo < >]

On Monday, July 8, 2013 at 12:34 PM, Brad Knowles wrote:

Folks,

I've seen CHEF-3432 and CHEF-3985, but I'm still seeing memory leak issues with chef-client 10.18.2, even though it's not running in daemon mode. The VM is configured for 2GB of "RAM" and 8GB of swap, and I'm now seeing chef-client reliably running up to ~9GB VSZ and 1.7GB RSS. This just started today, as I've been debugging and resolving various other issues in the cookbooks and recipes that we're using for this system.

However, it's not exactly clear to me what the best debugging process is, to try and figure out why it's leaking memory. I did check the node with a "knife node show -l", and the output is about 3400 lines long, comprising some 113KB of data. That's a little bigger than usual, but it doesn't seem to be totally out of whack.

And no, we're not using search at all. I wish we were, but that's a different story for a different day.

Are there particular tools or command-line options I should be using to try and figure out why chef-client is growing without bounds?

Ruby memory profiling is in a rough spot right now, as many of the useful tools developed for Ruby 1.8 have not been updated, and efforts to update them and incorporate them into ruby core have not yet landed.

When I was investigating CHEF-3432, I used remote pry to get a shell in the running ruby process and then `ObjectSpace.each_object` to count various object types. If you can reproduce your memory usage issue reliably, I'd suggest putting one remote pry invocation at the beginning of a run, to get a baseline, and then a second later in the run after the memory leak has occurred.

 

--

Brad Knowles < "> >

LinkedIn Profile: <http://tinyurl.com/y8kpxu>

-- 

Daniel DeLeo