Ohai! chefs!
My Recipe has to import from S3, extract and seed the data from the mysqldump.
To do this, I want it using ruby and the aws-sdk gem.
AFAIK, the ruby code in the recipe file is evaluated at compil time and the ones that are inside `ruby_block` is evaluated at convergence time.
So if I do `require 'aws'`, it won't since its not installed. When I add it inside the ruby_block, it errors out.
So to fix it, I used the `chef_gem 'aws-sdk'`, but this will run at the compile time which is fine.
But to install it, it depends on packages like lib-xml2 to be installed too at the compile time.
What I want is, what is a better way to do this using the ruby_block way?
.....
if app['db_import']
gem_package 'aws-sdk'
ruby_block "import_backed-up_files_from_S3" do
block do
# Getting the files from S3 backup
require 'aws'
.....
[2013-01-26T09:12:25+00:00] INFO: Processing gem_package[aws-sdk] action install (empowerme::extract_files_from_s3_backup line 6)
[2013-01-26T09:12:25+00:00] DEBUG: gem_package[aws-sdk] detected omnibus installation in /opt/chef/embedded/bin
[2013-01-26T09:12:25+00:00] DEBUG: gem_package[aws-sdk] searching for 'gem' binary in path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/chef/embedded/bin:/opt/chef/embedded/bin
[2013-01-26T09:12:25+00:00] DEBUG: gem_package[aws-sdk] using gem '/usr/bin/gem'
[2013-01-26T09:12:26+00:00] DEBUG: gem_package[aws-sdk] no installed version found for aws-sdk (>= 0)
RubyGems Environment:
- RUBYGEMS VERSION: 1.8.23
- RUBY VERSION: 1.9.3 (2012-11-10 patchlevel 327) [i686-linux]
- INSTALLATION DIRECTORY: /var/lib/gems/1.9.1
- RUBY EXECUTABLE: /usr/bin/ruby1.9.1
- EXECUTABLE DIRECTORY: /usr/local/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86-linux
- GEM PATHS:
- /var/lib/gems/1.9.1
- /root/.gem/ruby/1.9.1
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- http://rubygems.org/
[2013-01-26T09:12:27+00:00] DEBUG: found gem aws-sdk version 1.8.0 for platform ruby from http://rubygems.org/
When you HTTParty, you must party hard!
Building native extensions. This could take a while...
Successfully installed uuidtools-2.1.3
Successfully installed multi_xml-0.5.2
Successfully installed httparty-0.10.0
Successfully installed json-1.7.6
Successfully installed aws-sdk-1.8.0
5 gems installed
[2013-01-26T09:12:42+00:00] DEBUG: gem_package[aws-sdk] resetting gem environment to default
[2013-01-26T09:12:42+00:00] INFO: Processing ruby_block[import_backed-up_files_from_S3] action create (empowerme::extract_files_from_s3_backup line 8)
================================================================================
Error executing action `create` on resource 'ruby_block[import_backed-up_files_from_S3]'
================================================================================
LoadError
---------
cannot load such file -- aws
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:11:in `block (2 levels) in from_file'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb
8: ruby_block "import_backed-up_files_from_S3" do
9: block do
10: # Getting the files from S3 backup
11: require 'aws'
12: AWS.config(access_key_id: node['aws']['access_key_id'], secret_access_key: node['aws']['secret_access_key'])
13: s3 = AWS::S3.new
14:
15: bucket = s3.buckets[node['empowerme']['s3_bucket']]
16:
17: file_to_download = bucket.objects.map{|ob| ob.key}.sort.last # e.g. "empowerme/2012.10.14.20.40.02/empowerme.tar"
18:
19: Chef::Log.info("\n\nBackup Import from S3\n---------- \n #{file_to_download}\n-------------\n")
20:
21: obj = bucket.objects[file_to_download]
22:
23: local_file = Pathname.new("/tmp/#{node['empowerme']['backup_model_name']}.tar")
24:
25: File.open(local_file, "w") do |f|
26: obj.read do |chunk|
27: f.write chunk
28: end
29: end
30:
31: execute "extracting downloaded file" do
32: cwd local_file.dirname.to_s
33: command "tar -xf ./#{local_file.basename.to_path}"
34: user "root"
35: end
36:
37: # execute "change permission on #{app['deploy_to']}" do
38: # command "chmod g+rw #{app['deploy_to']}; chown -R #{app['owner']}:#{app['group']} #{app['deploy_to']}"
39: # end
40:
41: # execute "change permission on #{node['empowerme']['ebs_mount_dir']}" do
42: # command "chmod g+rw #{node['empowerme']['ebs_mount_dir']} && chown -R #{app['owner']}:#{app['group']} #{node['empowerme']['ebs_mount_dir']}"
43: # end
44: end
45: action :create
46: end
47: end
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:8:in `from_file'
ruby_block("import_backed-up_files_from_S3") do
action [:create]
retries 0
retry_delay 2
block_name "import_backed-up_files_from_S3"
cookbook_name "empowerme"
recipe_name "extract_files_from_s3_backup"
block #<Proc:0x85c7300@/var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:9>
end
[2013-01-26T09:12:43+00:00] DEBUG: Re-raising exception: LoadError - ruby_block[import_backed-up_files_from_S3] (empowerme::extract_files_from_s3_backup line 8) had an error: LoadError: cannot load such file -- aws
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
/var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:11:in `block (2 levels) in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider/ruby_block.rb:33:in `call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider/ruby_block.rb:33:in `block in action_create'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/mixin/why_run.rb:52:in `call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider.rb:152:in `converge_by'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider/ruby_block.rb:32:in `action_create'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider.rb:115:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:597:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:49:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:81:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:81:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection.rb:94:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection.rb:92:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:80:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:378:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:420:in `do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:176:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:140:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:274:in `block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:in `loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:in `run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:72:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:23:in `load'
/usr/bin/chef-client:23:in `<main>'
[2013-01-26T09:12:43+00:00] ERROR: Running exception handlers
[2013-01-26T09:12:43+00:00] ERROR: Report handler Chef::Handler::ErrorReport raised #<Encoding::UndefinedConversionError: "\xE2" from ASCII-8BIT to UTF-8>
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:500:in `encode'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:500:in `to_json'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:500:in `to_json'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:in `generate'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:in `pretty_generate'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/json_compat.rb:49:in `to_json_pretty'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler/error_report.rb:27:in `report'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:226:in `run_report_unsafe'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:214:in `run_report_safely'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:118:in `block in run_exception_handlers'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:117:in `each'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:117:in `run_exception_handlers'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:127:in `block in <class:Handler>'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:118:in `call'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:118:in `block in run_failed'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:117:in `each'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:117:in `run_failed'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:436:in `rescue in do_run'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:444:in `do_run'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:176:in `run'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:140:in `run_chef_client'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:274:in `block in run_application'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:in `loop'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:in `run_application'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:72:in `run'
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/bin/chef-client:26:in `<top (required)>'
[2013-01-26T09:12:43+00:00] ERROR: /usr/bin/chef-client:23:in `load'
[2013-01-26T09:12:43+00:00] ERROR: /usr/bin/chef-client:23:in `<main>'
[2013-01-26T09:12:43+00:00] ERROR: Exception handlers complete
[2013-01-26T09:12:43+00:00] DEBUG: Server doesn't support resource history, skipping resource report.
[2013-01-26T09:12:43+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-01-26T09:12:43+00:00] DEBUG: LoadError: ruby_block[import_backed-up_files_from_S3] (empowerme::extract_files_from_s3_backup line 8) had an error: LoadError: cannot load such file -- aws