[chef-dev] Re: Re: [chef] Re: Re: Intermittent chef-expander problem

[Alex Kiernan < >]

On Wed, Aug 31, 2011 at 5:23 PM, Alex Kiernan 
<
 >
 wrote:
> On Wed, Aug 31, 2011 at 4:21 PM, Daniel DeLeo 
> <
 >
>  wrote:
>> On Wednesday, August 31, 2011 at 5:00 AM, Alex Kiernan wrote:
>>> [moved to chef-dev]
>>>
>>> On Wed, Aug 31, 2011 at 11:12 AM, Alex Kiernan 
>>> <
 
>>>  
>>> (mailto:
 )>
>>>  wrote:
>>> > > > > I've seen something like this in a different context that leads 
>>> > > > > me to believe it's a bug in the JSON gem or possibly Ruby. It 
>>> > > > > seems to only occur on Red Hat systems. What version of ruby are 
>>> > > > > you using? Is your system 64 bit? Is ruby 64 bit?
>>> > >
>>> > > I'm guessing this:
>>> > >
>>> > > https://github.com/flori/json/issues/46
>>> > >
>>> > > might be the problem. Time to get that omnibus build working!
>>> >
>>> > Tried this just on a client (once I'd hacked the gemspecs so it didn't
>>> > continue to pick up 1.5.2) and I still got some corrupt JSON through,
>>> > but I'm guessing the JSON that a client starts with comes from the
>>> > server, so any corruption there would propagate through?
>>> >
>>> > To test that theory I've just hacked up the server similarly...
>>>
>>> And it's just died in the same way. Though this time I just restarted
>>> rabbitmq and chef-expander is (as expected) happy.
>>>
>>> Anyone any ideas? It definitely looks like it's native code corruption
>>> :( My thinking at the moment is to patch chef-expander to discard (and
>>> log) invalid messages since there's little point in it getting into a
>>> die/fork/die loop.
>>>
>> Just to be clear, you're now running with the json gem version 1.5.4 [1] 
>> on both the client and the server and you're still seeing this issue?
>
> Almost but not quite 1.5.4 - what I had was HEAD from github with
> commits up to fe046d68c5ed88b32b1cf3343babcf367b5cc79f, but I see
> there's work since then with more GC guard stuff so I'll pull that and
> give it a whirl (and the log references that exact defect - I could've
> sworn I had those changes :|)
>

Well... 16 hours in from pushing 1.5.4 on and we're still running,
which is about as good as we've ever managed previously. We've
occasionally had several days so will leave it running and see how it
goes.

>> Also, I agree that chef-expander should drop and log invalid messages. 
>> It's a tricky balance between ensuring that messages get retried for 
>> intermittent errors and dropping messages that will always cause an error, 
>> but in this case log and drop is definitely correct.
>>
>
> Cool will give that a go when I get a chance!
>

Had a look last night... I can't for the life of me figure out what
the code path is which makes it exit when it gets invalid JSON as it
looks like it's designed to cope :|

-- 
Alex Kiernan