[chef] Re: Re: Re: Re: Re: Re: Re: segfault with CentOS 5.4/5.5 + ruby 1.8.7 + chef 0.10.0

[Sergio Rubio < >]

Hey guys,

On Fri, Jun 3, 2011 at 11:05 AM, Raymond Tham < "> > wrote:

That's right, we noticed the seg faults happening on our i386 systems; we don't have chef-client set up on any x86_64 instances at the moment.

I wonder if you made any progress with this stuff. Been swamped lately, but I closely follow they list and RPM related threads. I'll get some fresh air soon and I was wondering if testing i386 packages is the best way to help here right now.

Thanks!

 

Thanks for looking into this!

On Fri, Jun 3, 2011 at 1:44 AM, James < " target="_blank"> > wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is using. We'll work on rebuilding that this weekend. All of your systems are i386, correct? 

Thanks a lot for the report and information!

James

On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham < " target="_blank"> > wrote:

Hi James,

Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US West, not East. It should be an S3 backed RightScale AMI.

I checked /root/.gemrc, and it points to http://ec2-us-west-mirror.rightscale.com/rubygems/archive/latest/. However, the gems should have been installed through yum through the RBEL5 i386 repo.

If it helps, here's the output from "gem list" after installing through yum.

~]# gem list

*** LOCAL GEMS ***

abstract (1.0.0)

activesupport (3.0.3)

allison (2.0.3)

bunny (0.6.0)

chef (0.10.0)

diff-lcs (1.1.2)

echoe (4.5.6)

erubis (2.6.6)

highline (1.6.1)

json (1.4.6)

mime-types (1.16)

mixlib-authentication (1.1.4)

mixlib-cli (1.2.0)

mixlib-config (1.1.2)

mixlib-log (1.3.0)

moneta (0.6.0)

net-sftp (2.0.4)

net-ssh (2.1.4, 2.0.23)

net-ssh-gateway (1.0.1)

net-ssh-multi (1.0.1)

ohai (0.6.4)

polyglot (0.3.1)

rake (0.8.7)

rake-compiler (0.7.5)

rest-client (1.6.1)

rspec (2.5.0)

rspec-core (2.5.1)

rspec-expectations (2.5.0)

rspec-mocks (2.5.0)

systemu (1.2.0)

treetop (1.4.9)

uuidtools (2.1.1)

xml-simple (1.0.12)

yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James < " target="_blank"> > wrote:

Ray,

The ami you gave doesn't seem to be public, but I found one which the same CentOS and RightImage numbers: ami-2342a94a.

Can you double-check that it's a public image, so I can make sure to mirror you as closely as possible? 

Did you also experience this: http://tickets.opscode.com/browse/CHEF-2402

Thanks, 

James

On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham < " target="_blank"> > wrote:

Hi James,

I originally noticed this happening on our older physical servers running i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was able to reproduce it. Here's what I did:

1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI (ami-31df8e74) 
2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5

3. Installed rubygem-chef (excluding EPEL so all the ruby packages are from RBEL): yum install --disablerepo=epel rubygem-chef

4. Manually created a /etc/chef/client.rb and copied over validation.pem.

5. Ran chef-client for the first time to register the node: /usr/bin/chef-client

6. Added recipe[logrotate] to the run list (this is the cookbook from opscode)

7. Re-ran chef-client which picked up the recipe and seg faults trying to run it 

With regards to the ruby 1.8.6 that ended up working:

1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in 1.8.6.420 and repackaged on CentOS 5.5.

2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)

3. Excluded ruby-*, rubygems-*, and rubygem-chef-0.10* in the RBEL yum config.

4. Ran yum install rubygem-chef with my new RPMs.

It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10 with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,

Ray

On Thu, Jun 2, 2011 at 6:45 PM, James < " target="_blank"> > wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install from the RBEL repo, and the AMI / image info? Are these i386 instances? 

We maintain packages at rpm.aegisco.com, but I'm happy to try to help debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had trouble getting 0.10 to work on anything older than 1.8.7 and rubygems 1.6.2. 

James

On Thu, Jun 2, 2011 at 6:29 PM, < " target="_blank"> > wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and didn't see
any resolution so I wanted to share my experience with the same thing happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the client on 5.6
yet). I'm consistently able to get our chef-client to segfault whenever it
handles this package resource (haven't had time to try other packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386 repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x, ruby 1.8.7
and Chef?

Thanks,
Ray