[chef] Re: BUG: unable to handle kernel NULL pointer dereference at pick_next_task_fair


Chronological Thread 
  • From: Daniel DeLeo < >
  • To:
  • Subject: [chef] Re: BUG: unable to handle kernel NULL pointer dereference at pick_next_task_fair
  • Date: Tue, 16 Oct 2012 08:21:48 -0700


On Tuesday, October 16, 2012 at 5:21 AM, wrote:

I'm running chef on aws ubuntu's and a few of my ubuntu instances crashed.
When looking at the log I found that the chef-client process which runs as
init.d script, stumbles upon what seems to be a kernel bug.

Full details are here: https://gist.github.com/3885828

And here's the gist of the gist is:

[119832.732086] BUG: unable to handle kernel NULL pointer dereference at
0000000000000038
[119832.732111] IP: [<ffffffff81053527>] pick_next_task_fair+0xa7/0x1a0
[119832.732124] PGD 1cee6f067 PUD 1cd5e7067 PMD 0
[119832.732132] Oops: 0000 [#1] SMP
[119832.732137] last sysfs file:
/sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map
[119832.732145] CPU 1
[119832.732147] Modules linked in: acpiphp raid456 async_pq async_xor xor
async_memcpy async_raid6_recov raid10 raid6_pq async_tx raid1 raid0 multipath
linear
[119832.732172]
[119832.732177] Pid: 7896, comm: chef-client Not tainted 2.6.38-8-virtual
#42-Ubuntu
...

$ uname -a
Linux hostname 2.6.38-8-virtual #42-Ubuntu SMP Mon Apr 11 04:06:34 UTC 2011
x86_64 x86_64 x86_64 GNU/Linux


Had anyone seem that?
Been googling that but nothing of note came up...

I don't know anything about this particular bug, but it looks probable that ohai is triggering this bug reading some entry in /proc (we've seen different bugs when reading /proc entries in ohai before).

I see a few ways out...

1. upgrade ubuntu and hope I don't see this again. (it doesn't happen every
day, but of the month I'd been using chef it happened four times to three
different hosts, out of 20 hosts)
2. Instead of using init script, use a cron or other method to run chef-client
on the hour.
3. OK, move to centos or something else desperate...
To have confidence in #1, you'll probably have to track down the specific kernel bug, and if there's a fix, find out what version it was fixed it. Then see if Ubuntu has ported the patch to your release. Additionally, I'd recommend adding some checks to your monitoring system, lest you accidentally spin up a new box with an old ISO/AMI/whatever.

Unless something about this bug means that it is only triggered during system startup, #2 probably won't work.

As for option #3, you could try to pinpoint which ohai plugin is triggering the bug and then disable it until a kernel patch is available (or a workaround is implemented in ohai, if possible).
 

I'd be happy to get to the bottom of it and see where the bug really comes from
and whether it was fixed in newer versions of the kernel but I shamefully admit
that so far I'd failed to get to the bottom of this bug.
what would you do?


-- 
Daniel DeLeo




Archive powered by MHonArc 2.6.16.

§