[chef] Re: Re: Re: Re: ohai 6.18 and 6.20 failing on ec2 hi1.xlarge

On Thu, 12 Dec 2013, 

 
 wrote:

> On Thu, 12 Dec 2013, 
> 
 
>  wrote:
>
> > On Thu, 12 Dec 2013, Andrew Gross wrote:
> >
> > > Are you running with IAM roles with an '=' in the name?
> >
> > No, we didn't launch this instance with an IAM instance profile.
>
> I added some debugging to the ohai code, ran it, and I can see which URI 
> it's
> barfing on on once instance vs another:
>
> http://169.254.169.254/2012-01-12/meta-data/metrics/vhostmd
>
> That returns 200 on all of our instances except the 2 having problems right 
> now.
> And the 2 having problems only just started having this problem today. I 
> think
> I'll go talk to AWS support now.

After talking with AWS support, that URL now returns 200, and ohai and chef
works again (because node['ec2']['userdata'] no longer returns nil).

I'm wondering, if just one URI returns 404 to ohai, but all others don't, why 
is
the ohai result an empty Hash? By querying ec2 metadata by hand, I could see
all of what my recipes cared about. Is ohai all or nothing?

kallen


> > > On Thu, Dec 12, 2013 at 3:40 PM, 
> > > <
 >
> > >  wrote:
> > >
> > > >
> > > > hai.
> > > >
> > > > I don't know if the instance type has anything to do with this, but 
> > > > ohai is
> > > > failing on the only ones we use of this ec2 instance type. Ohai and 
> > > > chef
> > > > had
> > > > been running fine on these for many weeks, but today started failing, 
> > > > and
> > > > I don't know why. Nothing changed that I know of.
> > > >
> > > > In general, we run chef 11.6.0 (ohai 6.18.0), but I grabbed 11.8.2 /
> > > > 6.20.0 to
> > > > see if the behavior would change, and it didn't.
> > > >
> > > > On the problem host, accessing user-data and meta-data work just fine 
> > > > using
> > > > "curl http://169.254.169.254/latest/user-data", etc.
> > > >
> > > > But "ohai -l debug ec2" throws:
> > > >  #<RuntimeError: Encountered error retrieving EC2 metadata (returned 
> > > > 404
> > > > response)
> > > >
> > > >
> > > > [2013-12-12T20:36:14+00:00] DEBUG: Loading plugin ec2
> > > > [2013-12-12T20:36:14+00:00] DEBUG: has_ec2_mac? == true
> > > > [2013-12-12T20:36:14+00:00] DEBUG: can_metadata_connect? == true
> > > > [2013-12-12T20:36:14+00:00] DEBUG: looks_like_ec2? == true
> > > > [2013-12-12T20:36:14+00:00] DEBUG: EC2 metadata version: 2012-01-12
> > > > [2013-12-12T20:36:14+00:00] DEBUG: Plugin ec2 threw exception
> > > > #<RuntimeError: Encountered error retrieving EC2 metadata (returned 
> > > > 404
> > > > response)>
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:104:in
> > > > `metadata_get'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:120:in
> > > > `block in fetch_metadata'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
> > > > `each'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
> > > > `fetch_metadata'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:130:in
> > > > `block in fetch_metadata'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
> > > > `each'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
> > > > `fetch_metadata'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/ec2.rb:52:in
> > > > `from_file'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
> > > > `instance_eval'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
> > > > `from_file'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
> > > > `block in require_plugin'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
> > > > `each'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
> > > > `require_plugin'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/cloud.rb:19:in
> > > > `from_file'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
> > > > `instance_eval'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
> > > > `from_file'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
> > > > `block in require_plugin'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
> > > > `each'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
> > > > `require_plugin'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:140:in
> > > > `block (2 levels) in all_plugins'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in
> > > > `each'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in
> > > > `block in all_plugins'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in
> > > > `each'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in
> > > > `all_plugins'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:97:in
> > > > `run_application'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:75:in
> > > > `run'
> > > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/bin/ohai:51:in
> > > > `<top (required)>'
> > > > /opt/chef/bin/ohai:23:in `load'
> > > > /opt/chef/bin/ohai:23:in `<main>'
> > > >
> > > >
> > > > Is this known? Any help or work arounds?
> > > >
> > > > thanks!
> > > > kallen
> > > >
> > > >
> > > >
> > > >