[chef] Re: Re: Re: Re: Re: Re: Re: disk running out of space

[Spike Grobstein < >]

Yeah, I was having compaction issues due to a bug in couchdb 1.0.1, the solution being to delete that file. compaction was failing, but not visibly (I only discovered that it was failing after looking at my couchdb logs).

...spike

On Nov 26, 2012, at 2:10 PM, Chris wrote:

Ah I see. No, compaction is working normally. It's just the .view files that need cleanup on a regular basis

Sent from a phone

On Nov 26, 2012, at 11:07 AM, Spike Grobstein < "> > wrote:

yes.

/var/lib/couchdb/1.0.1/chef.couch.compact

apparently it was left over from some process that was doing a compaction process and wasn't properly cleaned up. After deleting, it, I was suddenly able to properly compact and reclaim space.

Are you also having an issue with your database not compacting?

...spike

On Nov 26, 2012, at 2:04 PM, Chris wrote:

And that's in your $database_dir?

I see chef.couch, but not chef.couch.compact

Sent from a phone

On Nov 26, 2012, at 10:48 AM, Spike Grobstein < "> > wrote:

I'm running couchdb 1.0.1

chef 10.16.2

...spike

On Nov 26, 2012, at 1:42 PM, Chris wrote:

Interesting.  I don't have that file. What version are you running again?

Sent from a phone

On Nov 26, 2012, at 9:27 AM, Spike Grobstein < "> > wrote:

I finally solved this issue.

For those that are curious, the solution was:

After emailing the couchdb mailing list, it turned out that the issue was that I had an empty chef.couch.compact file. After deleting that and running the compaction script again, I went from 40GB of data down to 11MB.

The issue with the empty .compact file is the thing that's fixed in newer versions of couch, so this might not be an issue for new users, but it was a real problem for us.

...spike

On Nov 20, 2012, at 1:01 PM, Spike Grobstein wrote:

good catch.

I'm running 1.0.2 according to this:

chef-server:~$ curl localhost:5984

{"couchdb":"Welcome","version":"1.0.1"}

I'll look into getting that upgraded.

I'm still looking for a way to delete the old sandbox data, though. I have no idea how to do that. I don't think compaction will help since it's not revisions, it's straight-up obsolete documents.

Thanks!

...spike

On Nov 20, 2012, at 12:36 PM, Brian Hatfield wrote:

Ah, that looks correct enough to me.

Apologies if I've also missed this, but what version of Couch are you running?

If it's not 1.2, I strongly recommend you upgrade; that may help ensure that the compaction and cleanup tasks actually work. I noticed a huge improvement in compaction and such under 1.2 vs. previous versions myself.

Brian

On Tue, Nov 20, 2012 at 12:26 PM, Spike Grobstein < " target="_blank"> > wrote:

Hi Brian,

The end result of my daily cleanup cron is basically the same.

I didn't know about the way to list the views, but when I looked at my views (based on the URL you're hitting in your code) I get:

chef-server:~$ curl 'localhost:5984/chef/_all_docs?startkey="_design/"&endkey="_design0"'

{"total_rows":11243,"offset":7018,"rows":[

{"id":"_design/checksums","key":"_design/checksums","value":{"rev":"1-62a9807afbf0177330dc9989e99ae9a3"}},

{"id":"_design/clients","key":"_design/clients","value":{"rev":"1-f800279f9b454738aa1cf34eadaa5e9c"}},

{"id":"_design/cookbooks","key":"_design/cookbooks","value":{"rev":"1-12de613d3cfa8f0c0dde9a78b95346dc"}},

{"id":"_design/data_bags","key":"_design/data_bags","value":{"rev":"1-7f86059f3c4f9b81a9a0823de971efac"}},

{"id":"_design/environments","key":"_design/environments","value":{"rev":"1-0e657fdd2e990977704d0643adc3c7ed"}},

{"id":"_design/id_map","key":"_design/id_map","value":{"rev":"1-8f687bcb28aa335d35c93f6f2824cc46"}},

{"id":"_design/nodes","key":"_design/nodes","value":{"rev":"1-be0ed0a94ae8dbd9bd1caf2ac499d99e"}},

{"id":"_design/roles","key":"_design/roles","value":{"rev":"1-0f44c80a2244351cbd1f39ce72e37602"}},

{"id":"_design/sandboxes","key":"_design/sandboxes","value":{"rev":"1-9631200dcb18b22d313f41ee7a084a83"}},

{"id":"_design/users","key":"_design/users","value":{"rev":"1-3a045dbe31e7eb10588661d955c4455d"}}

]}

and I'm hitting all of those with my script:

https://gist.github.com/4091940

...spike

On Nov 20, 2012, at 12:15 PM, Brian Hatfield wrote:

Hi there,

Cleaning up CouchDB actually requires 3 different compaction commands, but some of the responses in this thread don't mention all 3:

View Compaction, View Cleanup, and overall Compaction. Without running all 3, not much space will be cleared.

I've written a small python command line tool that aides me in running these commands by automatically seeking all the views and running the compaction commands against them. Additionally, you can dump a GZ backup of your database as well.

I hope this helps: https://github.com/bmhatfield/couch-compaction

Brian

On Tue, Nov 20, 2012 at 12:12 PM, Spike Grobstein < " target="_blank"> > wrote:

Hi Daniel,

I don't have much experience with couchdb. What's the best way to do this?

I'd do it through the couchdb admin interface, but it's like 10,000 documents.

...spike

On Nov 20, 2012, at 11:51 AM, Daniel DeLeo wrote:

On Tuesday, November 20, 2012 at 8:45 AM, Spike Grobstein wrote:

Hi Ketan,

Yeah, I've got something similar set up:

https://gist.github.com/4091940

I  have that running once a day, but it's not affecting our free space and at the rate it's going, I'll be out of space in about a month. The couchdb size has already passed 40GB.

I really want to be able to just delete old data. I've got tons of Sandbox data going back as far as march.

...spike

Sandboxes should be safe to delete, but I'm not sure it will fix your issue. Would be interested to hear your results.

-- 

Daniel DeLeo