[chef] Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes


Chronological Thread 
  • From: Jesse Hu < >
  • To: ,
  • Subject: [chef] Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes
  • Date: Tue, 29 Jul 2014 11:57:34 +0800

Hi Chefs,

I'm using Chef to bootstrap a 300 nodes cluster and a few chef-clients throws the error "503 Service Unavailable" when the 300 nodes started syncing cookbooks at the same time. The chef server log shows erchef crashed. Most of the time, syncing cookbooks on all the nodes again after the 1st failure will succeed.

There is a ticket Chef -4423 Chef Synchronization of Cookbook Slow to speed up syncing cookbooks. I applied the new chef-client which contains the fix.  However, the fix doesn't work in my case.

Is there any tips to tune Open Source Chef Server 11 to serve hundreds of nodes ?  Thanks a lot.


Here is my chef configuration:

## Chef Server ##

2 cores + 8G Mem

$ cat /etc/issue
CentOS release 5.10 (Final)

$ rpm -q chef-server
chef-server-11.1.1-1.el5

$ cat /etc/chef-server/chef-server.rb
chef_server_webui['enable'] = false
nginx['ssl_port'] = 9443
nginx['non_ssl_port'] = 9080
chef_solr['heap_size'] = 3072
chef_solr['commit_interval'] = 3000
chef_solr['poll_seconds'] = 6
chef_expander['nodes'] = 4
erchef['s3_parallel_ops_timeout'] = 30000
erchef['s3_url_ttl'] = 3600
erchef['db_pool_size'] = 150
postgresql['max_connections'] = 200
rabbitmq['node_port'] = 5672


## Chef Client ##

$ cat /etc/issue
CentOS release 6.4 (Final)

$ rpm -q chef
chef-11.14.0.rc.2-1.el6.x86_64

$ cat /etc/chef/client.rb
log_location     STDOUT
chef_server_url  "https://<fqdn>:9443"
validation_client_name "chef-validator"
node_name "bigtop-master-0"

no_lazy_load true
ssl_verify_mode :verify_peer
ssl_ca_path "/etc/chef/trusted_certs"


## chef-server-ctl tail ==

full output of "chef-server-ctl tail" is attached.

==> /var/log/chef-server/erchef/crash.log.1 <==
** Last event was: {log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.23096.41> '}],error,{[">"},{node,' '}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"080"]},{1405,896002,80894},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == {state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] ",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-20 22:40:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}2014-07-20 22:40:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: {log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"085"]},{1405,896002,85731},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] \",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...]) in lager_console_backend:handle_event/2 line 95"]}}
** When handler state == {state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] ",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] \",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...]) in lager_console_backend:handle_event/2 line 95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}


==> /var/log/chef-server/erchef/erchef.log <==
2014-07-22 04:15:48.514 [error] Supervisor stats_hero_worker_sup had child stats_hero started with {stats_hero,start_link,undefined} at <0.30506.3> exit with reason {timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"358","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".application.byRequestType.","roles",".","GET"],58,"1866",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1834",124,"h","\n"],[["chefAPI",".upstreamReques...",...],...],...]]}]}} in context child_terminated
2014-07-22 04:15:48.600 [error] Supervisor stats_hero_worker_sup had child stats_hero started with {stats_hero,start_link,undefined} at <0.30501.3> exit with reason {timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"356","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"2000",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1987",124,"h","\n"],[["chefAPI",".upstreamRequest...",...],...],...]]}]}} in context child_terminated
2014-07-22 04:15:48.613 [error] Supervisor stats_hero_worker_sup had child stats_hero started with {stats_hero,start_link,undefined} at <0.30527.3> exit with reason {timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"110","\n",[[["chefAPI",".application.allRequests"],58,"1",124,"m","\n"],[["chefAPI",".","10",".allRequests"],58,"1",124,"m","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"1",124,"m","\n"]]]}]}} in context child_terminated
2014-07-22 04:16:12.850 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:16:13.010 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:16:13.024 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:05.852 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:13.356 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:13.363 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:13.433 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}



Thanks
Jesse Hu

==> /var/log/chef-server/rabbitmq/current <==
2014-07-04_10:36:36.40961 starting exchange, queue and binding recovery       
                  ...done
2014-07-04_10:36:43.19757 starting mirror queue slave sup                     
                  ...done
2014-07-04_10:36:43.19817 starting adding mirrors to queues                   
                  ...done
2014-07-04_10:36:43.21618 -- message delivery logic ready
2014-07-04_10:36:43.21624 starting error log relay                            
                  ...done
2014-07-04_10:36:43.21749 starting networking                                 
                  ...done
2014-07-04_10:36:43.25202 starting direct_client                              
                  ...done
2014-07-04_10:36:43.25258 starting notify cluster nodes                       
                  ...done
2014-07-04_10:36:43.25283 
2014-07-04_10:36:43.25284 broker running

==> /var/log/chef-server/nginx/error.log <==
2014/07/22 04:17:27 [error] 29511#0: *219321 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.119.101, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-c5baac1af8b112b9c95445552423ce9e?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006233&Signature=p9tm3SAD0C7eLIt2Jx5mv3KnY7U%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-c5baac1af8b112b9c95445552423ce9e?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006233&Signature=p9tm3SAD0C7eLIt2Jx5mv3KnY7U%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219722 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.118.179, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-02ed14d31c4b9c0dc70b8db9615e10f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006235&Signature=X3P15Y%2BhWkJv26e7ifpnlpnP8/8%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-02ed14d31c4b9c0dc70b8db9615e10f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006235&Signature=X3P15Y%2BhWkJv26e7ifpnlpnP8/8%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219739 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.118.15, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-366f8b6e2b40b8625276f585ae58088a?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006243&Signature=qDs1shFdZxs1B%2BH6WJOrSVvI1gw%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-366f8b6e2b40b8625276f585ae58088a?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006243&Signature=qDs1shFdZxs1B%2BH6WJOrSVvI1gw%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219732 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.118.42, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-67508cd66f9068d6cea88482293ed926?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=exjJCUCjucOWoqWe0%2BnxPvPkod4%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-67508cd66f9068d6cea88482293ed926?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=exjJCUCjucOWoqWe0%2BnxPvPkod4%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219735 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.118.135, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-8faf8ca68a8e2015403dae6a22b5be5f?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006225&Signature=gdma/MO4XouXrSm4vvQqxbDo93Q%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-8faf8ca68a8e2015403dae6a22b5be5f?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006225&Signature=gdma/MO4XouXrSm4vvQqxbDo93Q%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219737 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.119.231, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-9b1e25962e3d912cd3d638a645e522f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=0z2XeEv%2B5Kp7XwWhpsgJxS4AYj8%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-9b1e25962e3d912cd3d638a645e522f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=0z2XeEv%2B5Kp7XwWhpsgJxS4AYj8%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219738 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.118.53, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-be774f68b2fda4de95aef63054395bd3?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=SBglEXBnORK51EG1Hiq0XSnQmm4%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-be774f68b2fda4de95aef63054395bd3?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=SBglEXBnORK51EG1Hiq0XSnQmm4%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219736 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.118.188, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-ba9dd8717b41b8bc532aec2e32c49262?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=ETrIWTpKGPgBs5x0xEvzMGr%2Bwdk%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-ba9dd8717b41b8bc532aec2e32c49262?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=ETrIWTpKGPgBs5x0xEvzMGr%2Bwdk%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219741 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.119.124, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-8360ee7e5a56fd2ab116330392792e06?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006238&Signature=NUUQr8tO5lgOBcbG5T7zGuzRIMk%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-8360ee7e5a56fd2ab116330392792e06?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006238&Signature=NUUQr8tO5lgOBcbG5T7zGuzRIMk%3D";,
 host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219743 upstream timed out (110: 
Connection timed out) while connecting to upstream, client: 10.111.119.105, 
server: 10.111.119.252, request: "GET 
/bookshelf/organization-00000000000000000000000000000000/checksum-240de3895846451a6b46d76fe6f4b95c?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006241&Signature=NQXlHqR5eW7czd0hEPG3bLO3Ex0%3D
 HTTP/1.1", upstream: 
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-240de3895846451a6b46d76fe6f4b95c?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006241&Signature=NQXlHqR5eW7czd0hEPG3bLO3Ex0%3D";,
 host: "10.111.119.252:9443"

==> /var/log/chef-server/nginx/access.log <==
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98 
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98 
HTTP/1.1" 200 "0.013" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.013" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98 
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98 
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-98 
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99 
HTTP/1.1" 200 "0.030" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.030" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99 
HTTP/1.1" 200 "0.010" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.010" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99 
HTTP/1.1" 200 "0.008" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.008" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99 
HTTP/1.1" 200 "0.009" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.009" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:07 +0000]  "POST 
/search/node?q=cluster_name:huhui300&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000
 HTTP/1.1" 200 "0.862" 9087 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484; 
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200" 
"0.820" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti" 
"2014-07-22T04:20:06Z" "sZC8+2gGsGaHgTLbMbifW+YEwpg=" 1183

==> /var/log/chef-server/nginx/rewrite-port-9080.log <==

==> /var/log/chef-server/nginx/current <==

==> /var/log/chef-server/erchef/requests.log.1 <==
2014-07-22T04:20:04Z 

 method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti; 
req_id=dM3kh7n+3aU5xzJDINYjGQ==; req_time=9; rdbms_time=3; rdbms_count=3; 
2014-07-22T04:20:04Z 

 method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti; 
req_id=pKdQMRwImn5kgyON69kffA==; req_time=9; rdbms_time=4; rdbms_count=3; 
2014-07-22T04:20:04Z 

 method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti; 
req_id=Dl9ZwTcmwV60EuhQtIScWw==; req_time=10; rdbms_time=4; rdbms_count=3; 
2014-07-22T04:20:04Z 

 method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti; 
req_id=9kPj+T49GqMzErP2Q1azoQ==; req_time=9; rdbms_time=4; rdbms_count=3; 
2014-07-22T04:20:05Z 

 method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti; 
req_id=XtI6SJWBdlHzjMx2+TXSOw==; req_time=9; rdbms_time=3; rdbms_count=3; 
2014-07-22T04:20:05Z 

 method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti; 
req_id=vrb/TNT31kck94ByPcyk7w==; req_time=24; rdbms_time=19; rdbms_count=3; 
2014-07-22T04:20:05Z 

 method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti; 
req_id=BpPCCkPHQkVxRQ3bhU8ZLA==; req_time=7; rdbms_time=3; rdbms_count=3; 
2014-07-22T04:20:05Z 

 method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti; 
req_id=JIPfmMp1LmODcEY1vsIAIQ==; req_time=5; rdbms_time=0; rdbms_count=3; 
2014-07-22T04:20:05Z 

 method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti; 
req_id=/aWydPlwNnRibso/cP8y4A==; req_time=6; rdbms_time=2; rdbms_count=3; 
2014-07-22T04:20:07Z 

 method=POST; 
path=/search/node?q=cluster_name:huhui300&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000;
 status=200; user=serengeti; req_id=aPJAxKemU2wgI+FYV78MFQ==; 
msg={search,301,301,301}; req_time=815; rdbms_time=104; rdbms_count=63; 
solr_time=31; solr_count=1; 

==> /var/log/chef-server/erchef/crash.log.6 <==
** Last event was: 
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"15"],["21",58,"50",58,"02",46,"009"]},{1405,461002,9150},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 95"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-15 23:00:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server 
Error"}2014-07-15 23:00:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.10212.34>"},{node,' '}],error,{["2014",45,"07",45,"15"],["23",58,"00",58,"02",46,"231"]},{1405,465202,231822},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/crash.log.9 <==
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-12 23:00:01 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server 
Error"}2014-07-12 23:25:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.28620.29>"},{node,' '}],error,{["2014",45,"07",45,"12"],["23",58,"25",58,"02",46,"307"]},{1405,207502,307523},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-12 23:25:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/erchef.log.3 <==
2014-07-20 22:05:02.278 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-20 22:05:02.323 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=GET; 
path=/_status; status=500; \"",">>"],44,"\"Internal ..."],...],...]) in 
lager_console_backend:handle_event/2 line 95
2014-07-20 22:05:02.365 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=GET; path=/_status; status=500; 
\"",">>"],44,"\"Internal Server 
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l..."},...]}]}}
2014-07-20 22:40:02.080 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-20 22:40:02.084 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-20 22:40:02.085 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-20 22:40:02.085 [error] {<<"method=GET; path=/_status; status=500; 
">>,"Internal Server Error"}
2014-07-20 22:40:02.327 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-20 22:40:02.342 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}

==> /var/log/chef-server/erchef/crash.log.2 <==
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.3043.40>"},{node,' '}],error,{["2014",45,"07",45,"19"],["20",58,"45",58,"01",46,"949"]},{1405,802701,949318},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-19 20:45:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server 
Error"}2014-07-19 20:45:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"19"],["20",58,"45",58,"02",46,"010"]},{1405,802702,10847},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 95"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/erchef.log.1 <==
2014-07-21 09:38:31.072 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions; 
status=503; \"",">>"],44,"\"Service 
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}
2014-07-21 09:38:31.091 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-21 09:38:31.342 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=POST; 
path=/environments/_default/cookbook_versions; sta...",...],...],...],...]) 
in lager_console_backend:handle_event/2 line 95
2014-07-21 09:38:31.347 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions; 
status=503; \"",">>"],44,"\"Service 
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}
2014-07-21 09:39:34.234 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-21 09:39:34.235 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=POST; 
path=/environments/_default/cookbook_versions; sta...",...],...],...],...]) 
in lager_console_backend:handle_event/2 line 95
2014-07-21 09:39:34.236 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions; 
status=503; \"",">>"],44,"\"Service 
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}
2014-07-21 09:39:34.303 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-21 09:39:34.548 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=POST; 
path=/environments/_default/cookbook_versions; sta...",...],...],...],...]) 
in lager_console_backend:handle_event/2 line 95
2014-07-21 09:39:34.569 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions; 
status=503; \"",">>"],44,"\"Service 
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}

==> /var/log/chef-server/erchef/erchef.log.7 <==
2014-07-18 20:15:02.177 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-18 20:15:02.177 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-18 20:15:02.178 [error] {<<"method=GET; path=/_status; status=500; 
">>,"Internal Server Error"}
2014-07-18 20:15:02.266 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=GET; 
path=/_status; status=500; \"",">>"],44,"\"Internal ..."],...],...]) in 
lager_console_backend:handle_event/2 line 95
2014-07-18 20:15:02.322 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=GET; path=/_status; status=500; 
\"",">>"],44,"\"Internal Server 
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l..."},...]}]}}
2014-07-18 20:50:02.040 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-18 20:50:02.106 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-18 20:50:02.107 [error] {<<"method=GET; path=/_status; status=500; 
">>,"Internal Server Error"}
2014-07-18 20:50:02.108 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}

==> /var/log/chef-server/erchef/crash.log.4 <==
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.11517.37>"},{node,' '}],error,{["2014",45,"07",45,"17"],["23",58,"40",58,"02",46,"041"]},{1405,640402,41466},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-17 23:40:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server 
Error"}2014-07-17 23:40:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"17"],["23",58,"40",58,"02",46,"092"]},{1405,640402,92084},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 95"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/crash.log.7 <==
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.29083.32>"},{node,' '}],error,{["2014",45,"07",45,"14"],["23",58,"30",58,"01",46,"930"]},{1405,380601,930506},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-14 23:30:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{pid,<0.67.0>}],error,{["2014",45,"07",45,"14"],["23",58,"30",58,"01",46,"948"]},{1405,380601,948680},[123,[["<<","\"method=GET;
 path=/_status; status=500; \"",">>"],44,"\"Internal Server Error\""],125]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=GET; path=/_status; status=500; 
\"",">>"],44,"\"Internal Server 
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/crash.log.1 <==
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.23096.41>"},{node,' '}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"080"]},{1405,896002,80894},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-20 22:40:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server 
Error"}2014-07-20 22:40:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"085"]},{1405,896002,85731},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 95"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/requests.log.siz <==

==> /var/log/chef-server/erchef/crash.log.3 <==
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=GET; path=/_status; status=500; 
\"",">>"],44,"\"Internal Server 
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-18 20:50:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.18908.38>"},{node,' '}],error,{["2014",45,"07",45,"18"],["20",58,"50",58,"02",46,"040"]},{1405,716602,40838},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-18 20:50:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/crash.log <==
     Offender:   
[{pid,<0.30527.3>},{name,stats_hero},{mfargs,{stats_hero,start_link,undefined}},{restart_type,temporary},{shutdown,brutal_kill},{child_type,worker}]

2014-07-22 04:16:12 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}2014-07-22 04:16:12 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}2014-07-22 04:16:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}2014-07-22 04:17:05 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}2014-07-22 04:17:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}2014-07-22 04:17:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}2014-07-22 04:17:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}
==> /var/log/chef-server/erchef/crash.log.8 <==
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-13 22:30:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server 
Error"}2014-07-13 23:50:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.12852.31>"},{node,' '}],error,{["2014",45,"07",45,"13"],["23",58,"50",58,"02",46,"282"]},{1405,295402,282443},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-13 23:50:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/crash.log.0 <==
** Last event was: 
{log,{lager_msg,[],[{pid,<0.67.0>}],error,{["2014",45,"07",45,"21"],["09",58,"39",58,"34",46,"234"]},{1405,935574,234523},[123,[["<<","\"method=POST;
 path=/environments/_default/cookbook_versions; status=503; 
\"",">>"],44,"\"Service Unavailable\""],125]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions; 
status=503; \"",">>"],44,"\"Service 
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-21 09:39:34 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503; 
">>,"Service Unavailable"}2014-07-21 09:39:34 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{pid,<0.67.0>}],error,{["2014",45,"07",45,"21"],["09",58,"39",58,"34",46,"303"]},{1405,935574,303535},[123,[["<<","\"method=POST;
 path=/environments/_default/cookbook_versions; status=503; 
\"",">>"],44,"\"Service Unavailable\""],125]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions; 
status=503; \"",">>"],44,"\"Service 
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/erchef.log <==
2014-07-22 04:15:48.514 [error] Supervisor stats_hero_worker_sup had child 
stats_hero started with {stats_hero,start_link,undefined} at <0.30506.3> exit 
with reason 
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"358","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".application.byRequestType.","roles",".","GET"],58,"1866",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1834",124,"h","\n"],[["chefAPI",".upstreamReques...",...],...],...]]}]}}
 in context child_terminated
2014-07-22 04:15:48.600 [error] Supervisor stats_hero_worker_sup had child 
stats_hero started with {stats_hero,start_link,undefined} at <0.30501.3> exit 
with reason 
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"356","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"2000",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1987",124,"h","\n"],[["chefAPI",".upstreamRequest...",...],...],...]]}]}}
 in context child_terminated
2014-07-22 04:15:48.613 [error] Supervisor stats_hero_worker_sup had child 
stats_hero started with {stats_hero,start_link,undefined} at <0.30527.3> exit 
with reason 
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"110","\n",[[["chefAPI",".application.allRequests"],58,"1",124,"m","\n"],[["chefAPI",".","10",".allRequests"],58,"1",124,"m","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"1",124,"m","\n"]]]}]}}
 in context child_terminated
2014-07-22 04:16:12.850 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22 04:16:13.010 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22 04:16:13.024 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22 04:17:05.852 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22 04:17:13.356 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22 04:17:13.363 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22 04:17:13.433 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}

==> /var/log/chef-server/erchef/erchef.log.9 <==
2014-07-17 23:10:02.139 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-17 23:10:02.140 [error] {<<"method=GET; path=/_status; status=500; 
">>,"Internal Server Error"}
2014-07-17 23:10:02.140 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-17 23:40:02.041 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-17 23:40:02.091 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-17 23:40:02.092 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-17 23:40:02.092 [error] {<<"method=GET; path=/_status; status=500; 
">>,"Internal Server Error"}
2014-07-17 23:40:02.284 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-17 23:40:02.342 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}

==> /var/log/chef-server/erchef/crash.log.5 <==
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-16 20:35:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: 
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.21616.35>"},{node,' '}],error,{["2014",45,"07",45,"16"],["20",58,"35",58,"02",46,"317"]},{1405,542902,317837},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == 
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-16 20:35:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/erchef.log.5 <==
2014-07-19 19:40:02.294 [error] {<<"method=GET; path=/_status; status=500; 
">>,"Internal Server Error"}
2014-07-19 19:40:02.348 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-19 19:40:02.371 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}
2014-07-19 20:45:01.949 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-19 20:45:02.008 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-19 20:45:02.010 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-19 20:45:02.011 [error] {<<"method=GET; path=/_status; status=500; 
">>,"Internal Server Error"}
2014-07-19 20:45:02.104 [error] gen_event lager_console_backend installed in 
lager_event terminated with reason: bad argument in call to 
io:put_chars(user, unicode, ["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
 in lager_console_backend:handle_event/2 line 95
2014-07-19 20:45:02.120 [error] Lager event handler lager_console_backend 
exited with reason 
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","] 
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
 argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] 
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
 in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}

==> /var/log/chef-server/erchef/requests.log.idx <==

==> /var/log/chef-server/erchef/current <==
2014-07-22_04:15:51.44630 [error] Supervisor stats_hero_worker_sup had child 
stats_hero started with {stats_hero,start_link,undefined} at <0.30506.3> exit 
with reason 
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"358","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".application.byRequestType.","roles",".","GET"],58,"1866",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1834",124,"h","\n"],[["chefAPI",".upstreamReques...",...],...],...]]}]}}
 in context child_terminated
2014-07-22_04:15:51.48864 [error] Supervisor stats_hero_worker_sup had child 
stats_hero started with {stats_hero,start_link,undefined} at <0.30501.3> exit 
with reason 
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"356","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"2000",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1987",124,"h","\n"],[["chefAPI",".upstreamRequest...",...],...],...]]}]}}
 in context child_terminated
2014-07-22_04:15:51.52487 [error] Supervisor stats_hero_worker_sup had child 
stats_hero started with {stats_hero,start_link,undefined} at <0.30527.3> exit 
with reason 
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"110","\n",[[["chefAPI",".application.allRequests"],58,"1",124,"m","\n"],[["chefAPI",".","10",".allRequests"],58,"1",124,"m","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"1",124,"m","\n"]]]}]}}
 in context child_terminated
2014-07-22_04:16:14.96525 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22_04:16:15.03710 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22_04:16:15.13050 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22_04:17:06.87099 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22_04:17:13.49920 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22_04:17:13.51575 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}
2014-07-22_04:17:13.52022 [error] {<<"method=POST; 
path=/environments/_default/cookbook_versions; status=503; ">>,"Service 
Unavailable"}

==> /var/log/chef-server/chef-solr/current <==
2014-07-22_04:18:17.02212 Jul 22, 2014 4:18:17 AM 
org.apache.solr.core.SolrCore registerSearcher
2014-07-22_04:18:17.02212 INFO: [] Registered new searcher 

 main
2014-07-22_04:18:17.02235 Jul 22, 2014 4:18:17 AM 
org.apache.solr.search.SolrIndexSearcher close
2014-07-22_04:18:17.02236 INFO: Closing 

 main
2014-07-22_04:18:17.02236       
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2014-07-22_04:18:17.02236       
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=186,cumulative_hits=112,cumulative_hitratio=0.60,cumulative_inserts=74,cumulative_evictions=0}
2014-07-22_04:18:17.02237       
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1425,cumulative_hits=1239,cumulative_hitratio=0.86,cumulative_inserts=178,cumulative_evictions=0}
2014-07-22_04:18:17.02237       
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=4110,cumulative_hits=1555,cumulative_hitratio=0.37,cumulative_inserts=2555,cumulative_evictions=391}
2014-07-22_04:20:06.53909 Jul 22, 2014 4:20:06 AM 
org.apache.solr.core.SolrCore execute
2014-07-22_04:20:06.53912 INFO: [] webapp=/solr path=/select 
params={sort=X_CHEF_id_CHEF_X+asc&indent=off&start=0&q=content:cluster_name__%3D__huhui300&wt=json&fq=%2BX_CHEF_database_CHEF_X:chef_00000000000000000000000000000000+%2BX_CHEF_type_CHEF_X:node&rows=1000}
 hits=301 status=0 QTime=15 

==> /var/log/chef-server/chef-expander/current <==
2014-07-22_04:18:15.53363 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[0000000000007d7cb69aa5c0bbf5ea5f] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.009324073791503906,0.03338289260864258 |
2014-07-22_04:18:15.58465 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[0000000000008f90faa176bb868009dc] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.023186922073364258,0.028589725494384766 |
2014-07-22_04:18:15.61224 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[000000000000014982c024d3172875f4] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.009051799774169922,0.014837026596069336 |
2014-07-22_04:18:15.69051 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[0000000000004fd193602747e45ebe3a] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.008976936340332031,0.01736593246459961 |
2014-07-22_04:18:15.70876 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[000000000000e9fd3d07ee0b6a08daee] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.00881505012512207,0.014605283737182617 |
2014-07-22_04:18:15.75109 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[000000000000d1fc219733ae55a4ec18] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.009511947631835938,0.015871047973632812 |
2014-07-22_04:18:15.76214 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[00000000000002a18a228ff70d67e367] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.017444133758544922,0.027033090591430664 |
2014-07-22_04:18:15.78708 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[000000000000a620ce66813e0105eaa5] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.01209402084350586,0.019784927368164062 |
2014-07-22_04:18:15.86782 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[000000000000417951022e5d8f1697bd] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.019954919815063477,0.032138824462890625 |
2014-07-22_04:18:15.87531 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed 
node[000000000000eb849c19727be785306e] 
database[chef_00000000000000000000000000000000] transit,xml,solr-post | 
0,0.018274784088134766,0.0351099967956543 |

==> /var/log/chef-server/postgresql/current <==
2014-07-04_10:36:23.06733 LOG:  PID 30048 in cancel request did not match any 
process
2014-07-04_10:36:23.06733 LOG:  PID 30070 in cancel request did not match any 
process
2014-07-04_10:36:23.06733 LOG:  PID 30006 in cancel request did not match any 
process
2014-07-04_10:36:23.06734 LOG:  PID 17175 in cancel request did not match any 
process
2014-07-04_10:36:23.06734 LOG:  PID 30087 in cancel request did not match any 
process
2014-07-04_10:36:23.06734 LOG:  PID 29952 in cancel request did not match any 
process
2014-07-04_10:36:23.06734 LOG:  PID 29938 in cancel request did not match any 
process
2014-07-04_10:36:23.06735 LOG:  PID 29937 in cancel request did not match any 
process
2014-07-04_10:36:23.06736 LOG:  PID 17168 in cancel request did not match any 
process
2014-07-04_10:36:23.06736 LOG:  PID 30088 in cancel request did not match any 
process



Archive powered by MHonArc 2.6.16.

§