Ok, so I âsolvedâ this issue by deleting all cookbooks from the server, then re-uploading them while carefully checking versions pinned down for the environment at the same time.
Some of them had lower versions than the current one pinned down and something must have gone awry because of that, but thereâs no way I could pinpoint the problem by looking at the logs.
I really wish there was more information spit out. One good example would be that when a node doesnât find a cookbook *version* it wouldnât log that it couldnât find *the cookbook* on the server. Printing out which version it wants along with the name would go a long way here. :)
-- Cassiano Lealhttp://cassianoleal.com http://twitter.com/cassianoleal
On July 24, 2013 at 16:17:46, Cassiano Leal (
) wrote: I got new ones:
==> error.log <== 2013/07/24 16:15:56 [error] 859#0: *1771 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.137, server: ip-10-0-1-10.sa-east-1.compute.internal, request: "POST /environments/production/cookbook_versions HTTP/1.1", upstream: "http://127.0.0.1:8000/environments/production/cookbook_versions", host: "10.0.1.10:443"
==> access.log <== 10.0.1.137 - - [24/Jul/2013:16:15:56 -0300] "POST /environments/production/cookbook_versions HTTP/1.1" 504 "300.113" 182 "-" "Chef Client/11.4.4 (ruby-1.9.3-p286; ohai-6.16.0; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "504" "300.065" "11.4.4" "algorithm=sha1;version=1.0;" "apps" "2013-07-24T19:11:11Z" "FkL+xFBHbcjWA94iv8c+Izoud/w=â 1059
Another thing I noticed is that beam.smp is consuming 100% of a CPU core, and itâs been that way for hours.
-- Cassiano Lealhttp://cassianoleal.com http://twitter.com/cassianoleal On July 24, 2013 at 15:46:40, Stephen Delano (
) wrote:
Hi there,
Can you find the corresponding POST request in /var/log/chef-server/nginx/access.log? The post will be to "/environments/production/cookbook_versions" and
might be able to shed some more light on where the timeout occurred (e.g. connection timeout vs. read timeout).
Cheers!
Stephen
From: Cassiano Leal <
">
>
Reply-To: "
">
" <
">
>
Date: Wednesday, July 24, 2013 10:58 AM
To: "
">
" <
">
>
Subject: [chef] Re: Re: Chef Server timeouts
I ran chef-server-ctl test and got these failures:
http://pastie.org/private/6jqnpte37herj8jyomhmq
Any idea whatâs happening? Some indexing gone wrong maybe?
On July 24, 2013 at 14:16:41, Cassiano Leal (
">
) wrote:
The only things logged for erchef are:
2013-07-24T17:08:09Z
">
INFO req_id=x+AfrOJ1AUtORncdS9mIfg==; status=200; method=GET; path=/nodes/apps; user=apps; msg=[]; req_time=504; rdbms_time=422; rdbms_count=2
2013-07-24T17:08:09Z
">
INFO req_id=4QrUsBd/rJ77XlBB2hWogw==; status=200; method=GET; path=/roles/api; user=apps; msg=[]; req_time=479; rdbms_time=373; rdbms_count=2
2013-07-24T17:08:10Z
">
INFO req_id=55zYGCExTCxAYJAaXdGMGA==; status=200; method=GET; path=/roles/manager; user=apps; msg=[]; req_time=488; rdbms_time=356; rdbms_count=2
2013-07-24T17:08:10Z
">
INFO req_id=a8NzqLLMByrRSPSdbvazQw==; status=200; method=GET; path=/roles/web; user=apps; msg=[]; req_time=476; rdbms_time=349; rdbms_count=2
2013-07-24T17:08:11Z
">
INFO req_id=Ec3C1KgepqDadQu/WJpTPA==; status=200; method=GET; path=/environments/production; user=apps; msg=[]; req_time=469; rdbms_time=342; rdbms_count=2
Iâm not sure how this is helpful..
On July 24, 2013 at 12:42:19, Chris (
">
) wrote:
BTW, port 8000 should be the erChef process
|