[chef] Re: Re: BUG: unable to handle kernel NULL pointer dereference at pick_next_task_fair

[Ran Tavory < >]

Daniel thanks, I'm running an experiment - disabled all ohai plugins I could find, and wait a few days/weeks to see if something cashes again.

On Tue, Oct 16, 2012 at 5:21 PM, Daniel DeLeo < " target="_blank"> > wrote:

On Tuesday, October 16, 2012 at 5:21 AM, " target="_blank"> wrote:

I'm running chef on aws ubuntu's and a few of my ubuntu instances crashed.

When looking at the log I found that the chef-client process which runs as

init.d script, stumbles upon what seems to be a kernel bug.

Full details are here: https://gist.github.com/3885828

And here's the gist of the gist is:

[119832.732086] BUG: unable to handle kernel NULL pointer dereference at

0000000000000038

[119832.732111] IP: [<ffffffff81053527>] pick_next_task_fair+0xa7/0x1a0

[119832.732124] PGD 1cee6f067 PUD 1cd5e7067 PMD 0

[119832.732132] Oops: 0000 [#1] SMP

[119832.732137] last sysfs file:

/sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map

[119832.732145] CPU 1

[119832.732147] Modules linked in: acpiphp raid456 async_pq async_xor xor

async_memcpy async_raid6_recov raid10 raid6_pq async_tx raid1 raid0 multipath

linear

[119832.732172]

[119832.732177] Pid: 7896, comm: chef-client Not tainted 2.6.38-8-virtual

#42-Ubuntu

...

$ uname -a

Linux hostname 2.6.38-8-virtual #42-Ubuntu SMP Mon Apr 11 04:06:34 UTC 2011

x86_64 x86_64 x86_64 GNU/Linux

Had anyone seem that?

Been googling that but nothing of note came up...

I don't know anything about this particular bug, but it looks probable that ohai is triggering this bug reading some entry in /proc (we've seen different bugs when reading /proc entries in ohai before).

I see a few ways out...

1. upgrade ubuntu and hope I don't see this again. (it doesn't happen every

day, but of the month I'd been using chef it happened four times to three

different hosts, out of 20 hosts)

2. Instead of using init script, use a cron or other method to run chef-client

on the hour.

3. OK, move to centos or something else desperate...

To have confidence in #1, you'll probably have to track down the specific kernel bug, and if there's a fix, find out what version it was fixed it. Then see if Ubuntu has ported the patch to your release. Additionally, I'd recommend adding some checks to your monitoring system, lest you accidentally spin up a new box with an old ISO/AMI/whatever.

Unless something about this bug means that it is only triggered during system startup, #2 probably won't work.

As for option #3, you could try to pinpoint which ohai plugin is triggering the bug and then disable it until a kernel patch is available (or a workaround is implemented in ohai, if possible).

 

I'd be happy to get to the bottom of it and see where the bug really comes from

and whether it was fixed in newer versions of the kernel but I shamefully admit

that so far I'd failed to get to the bottom of this bug.

what would you do?

-- 

Daniel DeLeo