[chef-dev] Re: chef server cannot work for timeout

[Daniel DeLeo < >]

On Wednesday, July 29, 2015 at 10:32 AM, Stanley Shen wrote:

> Hello, all
>  
> I had a ubuntu server running chef-server_11.0.12-1.ubuntu.12.04_amd64.deb
> Recently I had restart the chef server, and then I cannot bootstrap VM and 
> ran chef client on it.
>  
> Here are some related errors, what could be the root cause?
> And how can I fix it?
>  
> =============================================================================
>  
> [2015-07-29T08:10:28-07:00] INFO: Running start handlers
> [2015-07-29T08:10:28-07:00] INFO: Start handlers complete.
> [2015-07-29T08:15:28-07:00] ERROR: Timeout connecting to test.com:443 
> (http://test.com:443) for /environments/_default/cookbook_versions, retry 
> 1/5
> [2015-07-29T08:20:34-07:00] ERROR: Timeout connecting to test.com:443 
> (http://test.com:443) for /environments/_default/cookbook_versions, retry 
> 2/5
> [2015-07-29T08:25:39-07:00] ERROR: Timeout connecting to test.com:443 
> (http://test.com:443) for /environments/_default/cookbook_versions, retry 
> 3/5
>  
> ================================================================================
> Error Resolving Cookbooks for Run List:
> ================================================================================
>  
> Authentication Error:
> ---------------------
> Failed to authenticate to the chef server (http 401).
> The request failed because your clock has drifted by more than 15 minutes. 
> ( I tried to sync time, but it doesn’t work)
> Syncing your clock to an NTP Time source should resolve the issue.

Since you recently rebooted your Chef Server, it’s possible that the clock on 
the Chef Server is the issue, rather than the node running chef-client.
  
>  
> [2015-07-29T08:25:44-07:00] ERROR: Running exception handlers
> [2015-07-29T08:25:44-07:00] FATAL: Saving node information to 
> /var/chef/cache/failed-run-data.json
> [2015-07-29T08:25:44-07:00] ERROR: Exception handlers complete
> [2015-07-29T08:25:44-07:00] FATAL: Stacktrace dumped to 
> /var/chef/cache/chef-stacktrace.out
> [2015-07-29T08:25:44-07:00] FATAL: Net::HTTPServerException: 401 
> "Unauthorized"
>  
> And in chef server I can such errors:
>  
> 2015/07/29 07:34:37 [error] 2664#0: *51 upstream timed out (110: Connection 
> timed out) while reading response header from upstream, client: 
> 38.102.156.135, server: chef3.cloud.intalio.com 
> (http://chef3.cloud.intalio.com), request: "POST 
> /environments/_default/cookbook_versions HTTP/1.1", upstream: 
> "http://127.0.0.1:8000/environments/_default/cookbook_versions", host: ;
> "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
> 2015/07/29 07:39:41 [error] 2665#0: *61 upstream timed out (110: Connection 
> timed out) while reading response header from upstream, client: 
> 38.102.156.135, server: chef3.cloud.intalio.com 
> (http://chef3.cloud.intalio.com), request: "POST 
> /environments/_default/cookbook_versions HTTP/1.1", upstream: 
> "http://127.0.0.1:8000/environments/_default/cookbook_versions", host: ;
> "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
> 2015/07/29 07:44:46 [error] 2665#0: *63 upstream timed out (110: Connection 
> timed out) while reading response header from upstream, client: 
> 38.102.156.135, server: chef3.cloud.intalio.com 
> (http://chef3.cloud.intalio.com), request: "POST 
> /environments/_default/cookbook_versions HTTP/1.1", upstream: 
> "http://127.0.0.1:8000/environments/_default/cookbook_versions", host: ;
> "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
> 2015/07/29 08:15:29 [error] 2664#0: *805 upstream timed out (110: 
> Connection timed out) while reading response header from upstream, client: 
> 38.102.156.135, server: chef3.cloud.intalio.com 
> (http://chef3.cloud.intalio.com), request: "POST 
> /environments/_default/cookbook_versions HTTP/1.1", upstream: 
> "http://127.0.0.1:8000/environments/_default/cookbook_versions", host: ;
> "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
> 2015/07/29 08:20:34 [error] 2664#0: *807 upstream timed out (110: 
> Connection timed out) while reading response header from upstream, client: 
> 38.102.156.135, server: chef3.cloud.intalio.com 
> (http://chef3.cloud.intalio.com), request: "POST 
> /environments/_default/cookbook_versions HTTP/1.1", upstream: 
> "http://127.0.0.1:8000/environments/_default/cookbook_versions", host: ;
> "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
> 2015/07/29 08:25:39 [error] 2664#0: *809 upstream timed out (110: 
> Connection timed out) while reading response header from upstream, client: 
> 38.102.156.135, server: chef3.cloud.intalio.com 
> (http://chef3.cloud.intalio.com), request: "POST 
> /environments/_default/cookbook_versions HTTP/1.1", upstream: 
> "http://127.0.0.1:8000/environments/_default/cookbook_versions", host: ;
> "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)”
>  
>  
> And in erchef code, I can see such:
>  
> 2015-07-29_17:04:04.97022 ** Found 0 name clashes in code paths
> 2015-07-29_17:04:05.20423
> 2015-07-29_17:04:05.20425 =INFO REPORT==== 29-Jul-2015::10:04:05 ===
> 2015-07-29_17:04:05.20425 Starting sqerl supervisor
> 2015-07-29_17:04:07.13848
> 2015-07-29_17:04:07.13850 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.13850 starting stats_hero_sup
> 2015-07-29_17:04:07.14937
> 2015-07-29_17:04:07.14938 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.14939 starting stats_hero_sender_sup with 1 senders
> 2015-07-29_17:04:07.16031
> 2015-07-29_17:04:07.16032 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.16033 stats_hero_sender starting pid: <0.216.0> port: 
> 42498
> 2015-07-29_17:04:07.17695
> 2015-07-29_17:04:07.17696 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.17697 bulk_fetch_batch_size set to 5
> 2015-07-29_17:04:07.35016
> 2015-07-29_17:04:07.35018 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.35018 Chef Keyring starting
> 2015-07-29_17:04:07.42652
> 2015-07-29_17:04:07.42655 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.42656 Loaded key default of type RSAPrivateKey from 
> file /etc/chef-server/chef-webui.pem
> 2015-07-29_17:04:07.44778
> 2015-07-29_17:04:07.44779 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.44780 starting chef_index_sup
> 2015-07-29_17:04:07.44816
> 2015-07-29_17:04:07.44817 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.44818 Connecting to Rabbit at 127.0.0.1:5672/chef 
> (http://127.0.0.1:5672/chef) (exchange: <<>>)
> 2015-07-29_17:04:07.48567
> 2015-07-29_17:04:07.48569 =ERROR REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.48569 Could not start the network driver: econnrefused
> 2015-07-29_17:04:07.48779
> 2015-07-29_17:04:07.48780 =ERROR REPORT==== 29-Jul-2015::10:04:07 ===
> 2015-07-29_17:04:07.48780 ** Generic server <0.252.0> terminating
> 2015-07-29_17:04:07.48780 ** Last message in was connect
> 2015-07-29_17:04:07.48780 ** When Server state == {state,<0.250.0>,
> 2015-07-29_17:04:07.48781 {amqp_params,<<"chef">>,
> 2015-07-29_17:04:07.48781 
> <<"050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10">>,
> 2015-07-29_17:04:07.48781 <<"/chef">>,"127.0.0.1",5672,0,0,0,none,[]},
> 2015-07-29_17:04:07.48782 undefined,undefined,undefined,undefined,undefined,
> 2015-07-29_17:04:07.48782 undefined,<0.251.0>,false,undefined,
> 2015-07-29_17:04:07.48782 {{0,nil},
> 2015-07-29_17:04:07.48782 {dict,0,16,16,8,80,48,
> 2015-07-29_17:04:07.48783 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> 2015-07-29_17:04:07.48783 [],[]},
> 2015-07-29_17:04:07.48783 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> 2015-07-29_17:04:07.48783 [],[]}}}},
> 2015-07-29_17:04:07.48784 undefined,#Fun<amqp_connection_sup.0.67288451>}
> 2015-07-29_17:04:07.48784 ** Reason for termination ==
> 2015-07-29_17:04:07.48784 ** econnrefused

This means rabbitmq is down (or just not up yet).

HTH,

--
Daniel DeLeo