[chef] Re: Re: Re: Intermittent chef-expander problem

[Alex Kiernan < >]

On Fri, Aug 26, 2011 at 4:53 PM, Daniel DeLeo 
<
 >
 wrote:
> On Friday, August 26, 2011 at 3:37 AM, Alex Kiernan wrote:
>> On Fri, Aug 26, 2011 at 11:16 AM, Alex Kiernan 
>> <
 
>>  
>> (mailto:
 )>
>>  wrote:
>> > We've an intermittent chef-expander problem... every few days it'll
>> > get invalid JSON from rabbitmq, the pattern's always the same, 8
>> > octets of corruption, the last 4 of which are always NUL. If we
>> > restart rabbitmq & chef-expander the problem goes away for a few days,
>> > then comes back... a (heavily snipped) example is:
>
> Why restart chef-expander? Does the problem persist until you restart it?
>

Pretty sure we could just restart rabbitmq... not actually tried that,
but it would make sense for that to fix it since all that happens now
is the chef-expander child process dies, is respawned and promptly
dies with the exact same error (repeat until your disk fills with the
expander.log)

>> >
>> > [Thu, 25 Aug 2011 18:24:48 +0100] ERROR: cannot index object because it 
>> > is inval
>> > id JSON: 
>> > {"action":"add","payload":{"enqueued_at":1314293088,"id":"92a1d57c-4579
>> > -446c-8fda-c3331f8477d9","type":"node","database":"chef","item":{
>> > ...............
>> > "dmi":^@<80><9E>
 @
 @{"address":"0xEA2E0","firmware_revision":"0.0",
>> >
>> > Any ideas?
>> >
>> > This is Chef 0.10.0 on RHEL 5.5
>>
>> Sorry should have added RabbitMQ is 2.5.1, Erlang is R14B.
>>

> I've seen something like this in a different context that leads me to 
> believe it's a bug in the JSON gem or possibly Ruby. It seems to only occur 
> on Red Hat systems. What version of ruby are you using? Is your system 64 
> bit? Is ruby 64 bit?

System is 64 bit, ruby is 64 bit:

ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770,
Ruby Enterprise Edition 2010.02

I've been hacking on an omnibus version of chef-server
(https://github.com/akiernan/omnibus) maybe I should expend a bit more
effort getting that so we can actually roll it out, though I guess I'm
not clear that might not suffer from the same problems, though at
least we'd have an entirely isolated/known environment.

Thinking about it some more I think we've the same problem where
chef-client gets bogus JSON at startup on our clients (also 64bit
RHEL) - I get this problem on my workstation when using an omnibus
chef-client (0.10.4), so it'd suggest that the problem's somewhere
underlying :( In fact just checking the logs the corruption that sees
is also in the dmi object.

-- 
Alex Kiernan