chef-dev - [chef-dev] Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes

Subscribers: 756
Owners
Adam Jacob
Bryan McLellan
Joshua Timberman
Nathen Harvey
Seth Chisamore
Serdar Sutay

Subscribe
Unsubscribe
Info
Archive

Post

RSS
Shared documents

Chef Developers

[chef-dev] Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes

From: Jesse Hu < >
To: ,
Subject: [chef-dev] Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes
Date: Tue, 29 Jul 2014 11:57:34 +0800

Hi Chefs,

I'm using Chef to bootstrap a 300 nodes cluster and a few chef-clients throws the error "503 Service Unavailable" when the 300 nodes started syncing cookbooks at the same time. The chef server log shows erchef crashed. Most of the time, syncing cookbooks on all the nodes again after the 1st failure will succeed.

There is a ticket Chef -4423 Chef Synchronization of Cookbook Slow to speed up syncing cookbooks. I applied the new chef-client which contains the fix. However, the fix doesn't work in my case.

Is there any tips to tune Open Source Chef Server 11 to serve hundreds of nodes ? Thanks a lot.

Here is my chef configuration:

## Chef Server ##

2 cores + 8G Mem

$ cat /etc/issue
CentOS release 5.10 (Final)

$ rpm -q chef-server
chef-server-11.1.1-1.el5

$ cat /etc/chef-server/chef-server.rb
chef_server_webui['enable'] = false
nginx['ssl_port'] = 9443
nginx['non_ssl_port'] = 9080
chef_solr['heap_size'] = 3072
chef_solr['commit_interval'] = 3000
chef_solr['poll_seconds'] = 6
chef_expander['nodes'] = 4
erchef['s3_parallel_ops_timeout'] = 30000
erchef['s3_url_ttl'] = 3600
erchef['db_pool_size'] = 150
postgresql['max_connections'] = 200
rabbitmq['node_port'] = 5672

## Chef Client ##

$ cat /etc/issue
CentOS release 6.4 (Final)

$ rpm -q chef
chef-11.14.0.rc.2-1.el6.x86_64

$ cat /etc/chef/client.rb
log_location STDOUT
chef_server_url "https://<fqdn>:9443"
validation_client_name "chef-validator"
node_name "bigtop-master-0"

no_lazy_load true
ssl_verify_mode :verify_peer
ssl_ca_path "/etc/chef/trusted_certs"

## chef-server-ctl tail ==

full output of "chef-server-ctl tail" is attached.

==> /var/log/chef-server/erchef/crash.log.1 <==
** Last event was: {log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.23096.41> '}],error,{[">"},{node,' '}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"080"]},{1405,896002,80894},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state == {state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] ",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-20 22:40:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}2014-07-20 22:40:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was: {log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"085"]},{1405,896002,85731},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] \",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...]) in lager_console_backend:handle_event/2 line 95"]}}
** When handler state == {state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","] ",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"] \",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...]) in lager_console_backend:handle_event/2 line 95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/erchef.log <==
2014-07-22 04:15:48.514 [error] Supervisor stats_hero_worker_sup had child stats_hero started with {stats_hero,start_link,undefined} at <0.30506.3> exit with reason {timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"358","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".application.byRequestType.","roles",".","GET"],58,"1866",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1834",124,"h","\n"],[["chefAPI",".upstreamReques...",...],...],...]]}]}} in context child_terminated
2014-07-22 04:15:48.600 [error] Supervisor stats_hero_worker_sup had child stats_hero started with {stats_hero,start_link,undefined} at <0.30501.3> exit with reason {timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"356","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"2000",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1987",124,"h","\n"],[["chefAPI",".upstreamRequest...",...],...],...]]}]}} in context child_terminated
2014-07-22 04:15:48.613 [error] Supervisor stats_hero_worker_sup had child stats_hero started with {stats_hero,start_link,undefined} at <0.30527.3> exit with reason {timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"110","\n",[[["chefAPI",".application.allRequests"],58,"1",124,"m","\n"],[["chefAPI",".","10",".allRequests"],58,"1",124,"m","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"1",124,"m","\n"]]]}]}} in context child_terminated
2014-07-22 04:16:12.850 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:16:13.010 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:16:13.024 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:05.852 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:13.356 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:13.363 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}
2014-07-22 04:17:13.433 [error] {<<"method=POST; path=/environments/_default/cookbook_versions; status=503; ">>,"Service Unavailable"}

Thanks
Jesse Hu

==> /var/log/chef-server/rabbitmq/current <==
2014-07-04_10:36:36.40961 starting exchange, queue and binding recovery
                  ...done
2014-07-04_10:36:43.19757 starting mirror queue slave sup
                  ...done
2014-07-04_10:36:43.19817 starting adding mirrors to queues
                  ...done
2014-07-04_10:36:43.21618 -- message delivery logic ready
2014-07-04_10:36:43.21624 starting error log relay
                  ...done
2014-07-04_10:36:43.21749 starting networking
                  ...done
2014-07-04_10:36:43.25202 starting direct_client
                  ...done
2014-07-04_10:36:43.25258 starting notify cluster nodes
                  ...done
2014-07-04_10:36:43.25283
2014-07-04_10:36:43.25284 broker running

==> /var/log/chef-server/nginx/error.log <==
2014/07/22 04:17:27 [error] 29511#0: *219321 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.119.101,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-c5baac1af8b112b9c95445552423ce9e?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006233&Signature=p9tm3SAD0C7eLIt2Jx5mv3KnY7U%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-c5baac1af8b112b9c95445552423ce9e?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006233&Signature=p9tm3SAD0C7eLIt2Jx5mv3KnY7U%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219722 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.118.179,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-02ed14d31c4b9c0dc70b8db9615e10f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006235&Signature=X3P15Y%2BhWkJv26e7ifpnlpnP8/8%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-02ed14d31c4b9c0dc70b8db9615e10f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006235&Signature=X3P15Y%2BhWkJv26e7ifpnlpnP8/8%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219739 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.118.15,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-366f8b6e2b40b8625276f585ae58088a?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006243&Signature=qDs1shFdZxs1B%2BH6WJOrSVvI1gw%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-366f8b6e2b40b8625276f585ae58088a?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006243&Signature=qDs1shFdZxs1B%2BH6WJOrSVvI1gw%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219732 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.118.42,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-67508cd66f9068d6cea88482293ed926?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=exjJCUCjucOWoqWe0%2BnxPvPkod4%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-67508cd66f9068d6cea88482293ed926?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=exjJCUCjucOWoqWe0%2BnxPvPkod4%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219735 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.118.135,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-8faf8ca68a8e2015403dae6a22b5be5f?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006225&Signature=gdma/MO4XouXrSm4vvQqxbDo93Q%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-8faf8ca68a8e2015403dae6a22b5be5f?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006225&Signature=gdma/MO4XouXrSm4vvQqxbDo93Q%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219737 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.119.231,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-9b1e25962e3d912cd3d638a645e522f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=0z2XeEv%2B5Kp7XwWhpsgJxS4AYj8%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-9b1e25962e3d912cd3d638a645e522f7?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006229&Signature=0z2XeEv%2B5Kp7XwWhpsgJxS4AYj8%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219738 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.118.53,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-be774f68b2fda4de95aef63054395bd3?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=SBglEXBnORK51EG1Hiq0XSnQmm4%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-be774f68b2fda4de95aef63054395bd3?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=SBglEXBnORK51EG1Hiq0XSnQmm4%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219736 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.118.188,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-ba9dd8717b41b8bc532aec2e32c49262?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=ETrIWTpKGPgBs5x0xEvzMGr%2Bwdk%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-ba9dd8717b41b8bc532aec2e32c49262?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006224&Signature=ETrIWTpKGPgBs5x0xEvzMGr%2Bwdk%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219741 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.119.124,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-8360ee7e5a56fd2ab116330392792e06?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006238&Signature=NUUQr8tO5lgOBcbG5T7zGuzRIMk%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-8360ee7e5a56fd2ab116330392792e06?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006238&Signature=NUUQr8tO5lgOBcbG5T7zGuzRIMk%3D";,
host: "10.111.119.252:9443"
2014/07/22 04:17:27 [error] 29511#0: *219743 upstream timed out (110:
Connection timed out) while connecting to upstream, client: 10.111.119.105,
server: 10.111.119.252, request: "GET
/bookshelf/organization-00000000000000000000000000000000/checksum-240de3895846451a6b46d76fe6f4b95c?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006241&Signature=NQXlHqR5eW7czd0hEPG3bLO3Ex0%3D
HTTP/1.1", upstream:
"http://127.0.0.1:4321/bookshelf/organization-00000000000000000000000000000000/checksum-240de3895846451a6b46d76fe6f4b95c?AWSAccessKeyId=276ae67f2dad89951e6a3447398232e80070f869&Expires=1406006241&Signature=NQXlHqR5eW7czd0hEPG3bLO3Ex0%3D";,
host: "10.111.119.252:9443"

==> /var/log/chef-server/nginx/access.log <==
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98
HTTP/1.1" 200 "0.013" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.013" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:04 +0000]  "GET /nodes/huhui300-data-98
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:04Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-98
HTTP/1.1" 200 "0.014" 4674 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.014" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99
HTTP/1.1" 200 "0.030" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.030" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99
HTTP/1.1" 200 "0.010" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.010" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99
HTTP/1.1" 200 "0.008" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.008" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:05 +0000]  "GET /nodes/huhui300-data-99
HTTP/1.1" 200 "0.009" 4647 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.009" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:05Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1000
10.111.119.252 - - [22/Jul/2014:04:20:07 +0000]  "POST
/search/node?q=cluster_name:huhui300&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000
HTTP/1.1" 200 "0.862" 9087 "-" "Chef Knife/11.12.2 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.820" "11.12.2" "algorithm=sha1;version=1.0;" "serengeti"
"2014-07-22T04:20:06Z" "sZC8+2gGsGaHgTLbMbifW+YEwpg=" 1183

==> /var/log/chef-server/nginx/rewrite-port-9080.log <==

==> /var/log/chef-server/nginx/current <==

==> /var/log/chef-server/erchef/requests.log.1 <==
2014-07-22T04:20:04Z

method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti;
req_id=dM3kh7n+3aU5xzJDINYjGQ==; req_time=9; rdbms_time=3; rdbms_count=3;
2014-07-22T04:20:04Z

method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti;
req_id=pKdQMRwImn5kgyON69kffA==; req_time=9; rdbms_time=4; rdbms_count=3;
2014-07-22T04:20:04Z

method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti;
req_id=Dl9ZwTcmwV60EuhQtIScWw==; req_time=10; rdbms_time=4; rdbms_count=3;
2014-07-22T04:20:04Z

method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti;
req_id=9kPj+T49GqMzErP2Q1azoQ==; req_time=9; rdbms_time=4; rdbms_count=3;
2014-07-22T04:20:05Z

method=GET; path=/nodes/huhui300-data-98; status=200; user=serengeti;
req_id=XtI6SJWBdlHzjMx2+TXSOw==; req_time=9; rdbms_time=3; rdbms_count=3;
2014-07-22T04:20:05Z

method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti;
req_id=vrb/TNT31kck94ByPcyk7w==; req_time=24; rdbms_time=19; rdbms_count=3;
2014-07-22T04:20:05Z

method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti;
req_id=BpPCCkPHQkVxRQ3bhU8ZLA==; req_time=7; rdbms_time=3; rdbms_count=3;
2014-07-22T04:20:05Z

method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti;
req_id=JIPfmMp1LmODcEY1vsIAIQ==; req_time=5; rdbms_time=0; rdbms_count=3;
2014-07-22T04:20:05Z

method=GET; path=/nodes/huhui300-data-99; status=200; user=serengeti;
req_id=/aWydPlwNnRibso/cP8y4A==; req_time=6; rdbms_time=2; rdbms_count=3;
2014-07-22T04:20:07Z

method=POST;
path=/search/node?q=cluster_name:huhui300&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000;
status=200; user=serengeti; req_id=aPJAxKemU2wgI+FYV78MFQ==;
msg={search,301,301,301}; req_time=815; rdbms_time=104; rdbms_count=63;
solr_time=31; solr_count=1;

==> /var/log/chef-server/erchef/crash.log.6 <==
** Last event was:
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"15"],["21",58,"50",58,"02",46,"009"]},{1405,461002,9150},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 95"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-15 23:00:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server
Error"}2014-07-15 23:00:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.10212.34>"},{node,' '}],error,{["2014",45,"07",45,"15"],["23",58,"00",58,"02",46,"231"]},{1405,465202,231822},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/crash.log.9 <==
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-12 23:00:01 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server
Error"}2014-07-12 23:25:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.28620.29>"},{node,' '}],error,{["2014",45,"07",45,"12"],["23",58,"25",58,"02",46,"307"]},{1405,207502,307523},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-12 23:25:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/erchef.log.3 <==
2014-07-20 22:05:02.278 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
in lager_console_backend:handle_event/2 line 95
2014-07-20 22:05:02.323 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=GET;
path=/_status; status=500; \"",">>"],44,"\"Internal ..."],...],...]) in
lager_console_backend:handle_event/2 line 95
2014-07-20 22:05:02.365 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=GET; path=/_status; status=500;
\"",">>"],44,"\"Internal Server
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l..."},...]}]}}
2014-07-20 22:40:02.080 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-20 22:40:02.084 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-20 22:40:02.085 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
in lager_console_backend:handle_event/2 line 95
2014-07-20 22:40:02.085 [error] {<<"method=GET; path=/_status; status=500;
">>,"Internal Server Error"}
2014-07-20 22:40:02.327 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
in lager_console_backend:handle_event/2 line 95
2014-07-20 22:40:02.342 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}

==> /var/log/chef-server/erchef/crash.log.2 <==
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.3043.40>"},{node,' '}],error,{["2014",45,"07",45,"19"],["20",58,"45",58,"01",46,"949"]},{1405,802701,949318},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-19 20:45:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server
Error"}2014-07-19 20:45:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"19"],["20",58,"45",58,"02",46,"010"]},{1405,802702,10847},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 95"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/erchef.log.1 <==
2014-07-21 09:38:31.072 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions;
status=503; \"",">>"],44,"\"Service
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}
2014-07-21 09:38:31.091 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-21 09:38:31.342 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=POST;
path=/environments/_default/cookbook_versions; sta...",...],...],...],...])
in lager_console_backend:handle_event/2 line 95
2014-07-21 09:38:31.347 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions;
status=503; \"",">>"],44,"\"Service
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}
2014-07-21 09:39:34.234 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-21 09:39:34.235 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=POST;
path=/environments/_default/cookbook_versions; sta...",...],...],...],...])
in lager_console_backend:handle_event/2 line 95
2014-07-21 09:39:34.236 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions;
status=503; \"",">>"],44,"\"Service
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}
2014-07-21 09:39:34.303 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-21 09:39:34.548 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=POST;
path=/environments/_default/cookbook_versions; sta...",...],...],...],...])
in lager_console_backend:handle_event/2 line 95
2014-07-21 09:39:34.569 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions;
status=503; \"",">>"],44,"\"Service
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,...}]}}

==> /var/log/chef-server/erchef/erchef.log.7 <==
2014-07-18 20:15:02.177 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
in lager_console_backend:handle_event/2 line 95
2014-07-18 20:15:02.177 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-18 20:15:02.178 [error] {<<"method=GET; path=/_status; status=500;
">>,"Internal Server Error"}
2014-07-18 20:15:02.266 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","] ",[123,[["<<","\"method=GET;
path=/_status; status=500; \"",">>"],44,"\"Internal ..."],...],...]) in
lager_console_backend:handle_event/2 line 95
2014-07-18 20:15:02.322 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=GET; path=/_status; status=500;
\"",">>"],44,"\"Internal Server
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l..."},...]}]}}
2014-07-18 20:50:02.040 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-18 20:50:02.106 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
in lager_console_backend:handle_event/2 line 95
2014-07-18 20:50:02.107 [error] {<<"method=GET; path=/_status; status=500;
">>,"Internal Server Error"}
2014-07-18 20:50:02.108 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}

==> /var/log/chef-server/erchef/crash.log.4 <==
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.11517.37>"},{node,' '}],error,{["2014",45,"07",45,"17"],["23",58,"40",58,"02",46,"041"]},{1405,640402,41466},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-17 23:40:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server
Error"}2014-07-17 23:40:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"17"],["23",58,"40",58,"02",46,"092"]},{1405,640402,92084},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 95"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/crash.log.7 <==
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.29083.32>"},{node,' '}],error,{["2014",45,"07",45,"14"],["23",58,"30",58,"01",46,"930"]},{1405,380601,930506},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-14 23:30:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{pid,<0.67.0>}],error,{["2014",45,"07",45,"14"],["23",58,"30",58,"01",46,"948"]},{1405,380601,948680},[123,[["<<","\"method=GET;
path=/_status; status=500; \"",">>"],44,"\"Internal Server Error\""],125]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=GET; path=/_status; status=500;
\"",">>"],44,"\"Internal Server
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/crash.log.1 <==
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.23096.41>"},{node,' '}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"080"]},{1405,896002,80894},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-20 22:40:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server
Error"}2014-07-20 22:40:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{pid,<0.101.0>}],error,{["2014",45,"07",45,"20"],["22",58,"40",58,"02",46,"085"]},{1405,896002,85731},[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 95"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line
95"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/requests.log.siz <==

==> /var/log/chef-server/erchef/crash.log.3 <==
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=GET; path=/_status; status=500;
\"",">>"],44,"\"Internal Server
Error\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-18 20:50:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.18908.38>"},{node,' '}],error,{["2014",45,"07",45,"18"],["20",58,"50",58,"02",46,"040"]},{1405,716602,40838},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-18 20:50:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/crash.log <==
     Offender:
[{pid,<0.30527.3>},{name,stats_hero},{mfargs,{stats_hero,start_link,undefined}},{restart_type,temporary},{shutdown,brutal_kill},{child_type,worker}]

2014-07-22 04:16:12 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}2014-07-22 04:16:12 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}2014-07-22 04:16:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}2014-07-22 04:17:05 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}2014-07-22 04:17:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}2014-07-22 04:17:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}2014-07-22 04:17:13 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}
==> /var/log/chef-server/erchef/crash.log.8 <==
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-13 22:30:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server
Error"}2014-07-13 23:50:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.12852.31>"},{node,' '}],error,{["2014",45,"07",45,"13"],["23",58,"50",58,"02",46,"282"]},{1405,295402,282443},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-13 23:50:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/crash.log.0 <==
** Last event was:
{log,{lager_msg,[],[{pid,<0.67.0>}],error,{["2014",45,"07",45,"21"],["09",58,"39",58,"34",46,"234"]},{1405,935574,234523},[123,[["<<","\"method=POST;
path=/environments/_default/cookbook_versions; status=503;
\"",">>"],44,"\"Service Unavailable\""],125]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions;
status=503; \"",">>"],44,"\"Service
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-21 09:39:34 =ERROR REPORT====
{<<"method=POST; path=/environments/_default/cookbook_versions; status=503;
">>,"Service Unavailable"}2014-07-21 09:39:34 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{pid,<0.67.0>}],error,{["2014",45,"07",45,"21"],["09",58,"39",58,"34",46,"303"]},{1405,935574,303535},[123,[["<<","\"method=POST;
path=/environments/_default/cookbook_versions; status=503;
\"",">>"],44,"\"Service Unavailable\""],125]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[123,[["<<","\"method=POST; path=/environments/_default/cookbook_versions;
status=503; \"",">>"],44,"\"Service
Unavailable\""],125],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

==> /var/log/chef-server/erchef/erchef.log <==
2014-07-22 04:15:48.514 [error] Supervisor stats_hero_worker_sup had child
stats_hero started with {stats_hero,start_link,undefined} at <0.30506.3> exit
with reason
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"358","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".application.byRequestType.","roles",".","GET"],58,"1866",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1834",124,"h","\n"],[["chefAPI",".upstreamReques...",...],...],...]]}]}}
in context child_terminated
2014-07-22 04:15:48.600 [error] Supervisor stats_hero_worker_sup had child
stats_hero started with {stats_hero,start_link,undefined} at <0.30501.3> exit
with reason
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"356","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"2000",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1987",124,"h","\n"],[["chefAPI",".upstreamRequest...",...],...],...]]}]}}
in context child_terminated
2014-07-22 04:15:48.613 [error] Supervisor stats_hero_worker_sup had child
stats_hero started with {stats_hero,start_link,undefined} at <0.30527.3> exit
with reason
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"110","\n",[[["chefAPI",".application.allRequests"],58,"1",124,"m","\n"],[["chefAPI",".","10",".allRequests"],58,"1",124,"m","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"1",124,"m","\n"]]]}]}}
in context child_terminated
2014-07-22 04:16:12.850 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22 04:16:13.010 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22 04:16:13.024 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22 04:17:05.852 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22 04:17:13.356 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22 04:17:13.363 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22 04:17:13.433 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}

==> /var/log/chef-server/erchef/erchef.log.9 <==
2014-07-17 23:10:02.139 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
in lager_console_backend:handle_event/2 line 95
2014-07-17 23:10:02.140 [error] {<<"method=GET; path=/_status; status=500;
">>,"Internal Server Error"}
2014-07-17 23:10:02.140 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-17 23:40:02.041 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-17 23:40:02.091 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-17 23:40:02.092 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
in lager_console_backend:handle_event/2 line 95
2014-07-17 23:40:02.092 [error] {<<"method=GET; path=/_status; status=500;
">>,"Internal Server Error"}
2014-07-17 23:40:02.284 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
in lager_console_backend:handle_event/2 line 95
2014-07-17 23:40:02.342 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}

==> /var/log/chef-server/erchef/crash.log.5 <==
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-16 20:35:02 =ERROR REPORT====
** gen_event handler lager_console_backend crashed.
** Was installed in lager_event
** Last event was:
{log,{lager_msg,[],[{application,chef_wm},{module,chef_wm_status},{function,log_failure},{line,77},{pid,"<0.21616.35>"},{node,' '}],error,{["2014",45,"07",45,"16"],["20",58,"35",58,"02",46,"317"]},{1405,542902,317837},[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"]}}
** When handler state ==
{state,{mask,127},lager_default_formatter,["[",severity,"] ",message,"\n"],[]}
** Reason == {badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2014-07-16 20:35:02 =ERROR REPORT====
{<<"method=GET; path=/_status; status=500; ">>,"Internal Server Error"}
==> /var/log/chef-server/erchef/erchef.log.5 <==
2014-07-19 19:40:02.294 [error] {<<"method=GET; path=/_status; status=500;
">>,"Internal Server Error"}
2014-07-19 19:40:02.348 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
in lager_console_backend:handle_event/2 line 95
2014-07-19 19:40:02.371 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}
2014-07-19 20:45:01.949 [error] /_status
{{status,fail},{upstreams,{[{<<"chef_solr">>,<<"fail">>},{<<"chef_sql">>,<<"pong">>}]}}}
2014-07-19 20:45:02.008 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">>,<<\"fail\">>},{<<\"chef_sql\">>,<<\"pong\">>}]}}}"],"\n"]],[]},{lager_console_backend,handle_event,2,[{file,"src/lager_console_backend.erl"},{line,95}]},{gen_event,server_update,4,[{file,"gen_event.erl"},{line,504}]},{gen_event,server_notify,4,[{file,"gen_event.erl"},{line,486}]},{gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,248}]},{proc_lib,...}]}}
2014-07-19 20:45:02.010 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[47,95,115,116,97,116,117,115,"\n","{{status,fail},{upstreams,{[{<<\"chef_solr\">..."],...])
in lager_console_backend:handle_event/2 line 95
2014-07-19 20:45:02.011 [error] {<<"method=GET; path=/_status; status=500;
">>,"Internal Server Error"}
2014-07-19 20:45:02.104 [error] gen_event lager_console_backend installed in
lager_event terminated with reason: bad argument in call to
io:put_chars(user, unicode, ["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,...],...])
in lager_console_backend:handle_event/2 line 95
2014-07-19 20:45:02.120 [error] Lager event handler lager_console_backend
exited with reason
{'EXIT',{badarg,[{io,put_chars,[user,unicode,["[","error","]
",[103,101,110,95,101,118,101,110,116,32,"lager_console_backend",32,105,110,115,116,97,108,108,101,100,32,105,110,32,"lager_event",32,116,101,114,109,105,110,97,116,101,100,32,119,105,116,104,32,114,101,97,115,111,110,58,32,"bad
argument in call to io:put_chars(user, unicode, [\"[\",\"error\",\"]
\",[47,95,115,116,97,116,117,115,\"\\n\",\"{{status,fail},{upstreams,{[{<<\\\"chef_solr\\\">...\"],...])
in lager_console_backend:handle_event/2 line 9..."],...]],...},...]}}

==> /var/log/chef-server/erchef/requests.log.idx <==

==> /var/log/chef-server/erchef/current <==
2014-07-22_04:15:51.44630 [error] Supervisor stats_hero_worker_sup had child
stats_hero started with {stats_hero,start_link,undefined} at <0.30506.3> exit
with reason
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"358","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"1866",124,"h","\n"],[["chefAPI",".application.byRequestType.","roles",".","GET"],58,"1866",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1834",124,"h","\n"],[["chefAPI",".upstreamReques...",...],...],...]]}]}}
in context child_terminated
2014-07-22_04:15:51.48864 [error] Supervisor stats_hero_worker_sup had child
stats_hero started with {stats_hero,start_link,undefined} at <0.30501.3> exit
with reason
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"356","\n",[[["chefAPI",".application.byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".","10",".byStatusCode.","200"],58,"1",124,"m","\n"],[["chefAPI",".application.allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".","10",".allRequests"],58,"2000",124,"h","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"2000",124,"h","\n"],[["chefAPI",".upstreamRequests.","rdbms"],58,"1987",124,"h","\n"],[["chefAPI",".upstreamRequest...",...],...],...]]}]}}
in context child_terminated
2014-07-22_04:15:51.52487 [error] Supervisor stats_hero_worker_sup had child
stats_hero started with {stats_hero,start_link,undefined} at <0.30527.3> exit
with reason
{timeout,{gen_server,call,[<0.925.0>,{send,[49,124,"110","\n",[[["chefAPI",".application.allRequests"],58,"1",124,"m","\n"],[["chefAPI",".","10",".allRequests"],58,"1",124,"m","\n"],[["chefAPI",".application.byRequestType.","data",".","GET"],58,"1",124,"m","\n"]]]}]}}
in context child_terminated
2014-07-22_04:16:14.96525 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22_04:16:15.03710 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22_04:16:15.13050 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22_04:17:06.87099 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22_04:17:13.49920 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22_04:17:13.51575 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}
2014-07-22_04:17:13.52022 [error] {<<"method=POST;
path=/environments/_default/cookbook_versions; status=503; ">>,"Service
Unavailable"}

==> /var/log/chef-server/chef-solr/current <==
2014-07-22_04:18:17.02212 Jul 22, 2014 4:18:17 AM
org.apache.solr.core.SolrCore registerSearcher
2014-07-22_04:18:17.02212 INFO: [] Registered new searcher

main
2014-07-22_04:18:17.02235 Jul 22, 2014 4:18:17 AM
org.apache.solr.search.SolrIndexSearcher close
2014-07-22_04:18:17.02236 INFO: Closing

main
2014-07-22_04:18:17.02236
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2014-07-22_04:18:17.02236
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=186,cumulative_hits=112,cumulative_hitratio=0.60,cumulative_inserts=74,cumulative_evictions=0}
2014-07-22_04:18:17.02237
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1425,cumulative_hits=1239,cumulative_hitratio=0.86,cumulative_inserts=178,cumulative_evictions=0}
2014-07-22_04:18:17.02237
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=4110,cumulative_hits=1555,cumulative_hitratio=0.37,cumulative_inserts=2555,cumulative_evictions=391}
2014-07-22_04:20:06.53909 Jul 22, 2014 4:20:06 AM
org.apache.solr.core.SolrCore execute
2014-07-22_04:20:06.53912 INFO: [] webapp=/solr path=/select
params={sort=X_CHEF_id_CHEF_X+asc&indent=off&start=0&q=content:cluster_name__%3D__huhui300&wt=json&fq=%2BX_CHEF_database_CHEF_X:chef_00000000000000000000000000000000+%2BX_CHEF_type_CHEF_X:node&rows=1000}
hits=301 status=0 QTime=15

==> /var/log/chef-server/chef-expander/current <==
2014-07-22_04:18:15.53363 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[0000000000007d7cb69aa5c0bbf5ea5f]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.009324073791503906,0.03338289260864258 |
2014-07-22_04:18:15.58465 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[0000000000008f90faa176bb868009dc]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.023186922073364258,0.028589725494384766 |
2014-07-22_04:18:15.61224 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[000000000000014982c024d3172875f4]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.009051799774169922,0.014837026596069336 |
2014-07-22_04:18:15.69051 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[0000000000004fd193602747e45ebe3a]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.008976936340332031,0.01736593246459961 |
2014-07-22_04:18:15.70876 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[000000000000e9fd3d07ee0b6a08daee]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.00881505012512207,0.014605283737182617 |
2014-07-22_04:18:15.75109 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[000000000000d1fc219733ae55a4ec18]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.009511947631835938,0.015871047973632812 |
2014-07-22_04:18:15.76214 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[00000000000002a18a228ff70d67e367]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.017444133758544922,0.027033090591430664 |
2014-07-22_04:18:15.78708 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[000000000000a620ce66813e0105eaa5]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.01209402084350586,0.019784927368164062 |
2014-07-22_04:18:15.86782 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[000000000000417951022e5d8f1697bd]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.019954919815063477,0.032138824462890625 |
2014-07-22_04:18:15.87531 [Tue, 22 Jul 2014 04:18:15 +0000] INFO: indexed
node[000000000000eb849c19727be785306e]
database[chef_00000000000000000000000000000000] transit,xml,solr-post |
0,0.018274784088134766,0.0351099967956543 |

==> /var/log/chef-server/postgresql/current <==
2014-07-04_10:36:23.06733 LOG:  PID 30048 in cancel request did not match any
process
2014-07-04_10:36:23.06733 LOG:  PID 30070 in cancel request did not match any
process
2014-07-04_10:36:23.06733 LOG:  PID 30006 in cancel request did not match any
process
2014-07-04_10:36:23.06734 LOG:  PID 17175 in cancel request did not match any
process
2014-07-04_10:36:23.06734 LOG:  PID 30087 in cancel request did not match any
process
2014-07-04_10:36:23.06734 LOG:  PID 29952 in cancel request did not match any
process
2014-07-04_10:36:23.06734 LOG:  PID 29938 in cancel request did not match any
process
2014-07-04_10:36:23.06735 LOG:  PID 29937 in cancel request did not match any
process
2014-07-04_10:36:23.06736 LOG:  PID 17168 in cancel request did not match any
process
2014-07-04_10:36:23.06736 LOG:  PID 30088 in cancel request did not match any
process

[chef-dev] Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes, Jesse Hu, 07/28/2014
- [chef-dev] Re: Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes, Adam Jacob, 07/28/2014
  - [chef-dev] Re: Re: Performance Tuning for Open Source Chef Server 11 to serve hundreds of nodes, Joseph Anthony Pasquale Holsten, 07/28/2014