[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Problem running chef-client, just from 2 days ago.

[Mike < >]

Was there ever resolution on this thread?

I just got Chef-Server 11.1.1-1 running on Ubuntu 12.04, and am seeing some of these types of problems when trying to run chef-client 11.12.8 concurrently across some 60 nodes.

Error Resolving Cookbooks for Run List:

Unknown Server Error: "The server had a fatal error attempting to load the node data."

Server Response: internal service error

Recipe Compile Error in /var/chef/cache/cookbooks/ourcookbook/recipe.rb

Net::HTTPFatalError: 500 "Internal Server Error"

Trace:... points to line where we load data_bag_item from server

Other failures point to server-side issues, as they fail on search queries.

Am open to debugging ideas, let me know what I should be hunting for.

-Mike

On Wed, May 28, 2014 at 5:23 PM, Lamont Granquist < " target="_blank"> > wrote:

The "End of File" is getting thrown because the socket connection gets torn down ("everything is a file" on unix (but not really, see plan9...)).

On 5/28/14, 11:35 AM, Adam Edwards wrote:

Hello Carlos -- do you have the output of c:/chef/cache/chef-stacktrace.out
from this last failure? And if it still repros, can you run with -l debug?

The error implies some sort of invalid file that is being accessed --
Lamont, can you tell the context from the output we have below?

-Adam

-----Original Message-----
From: Carlos Camacho [mailto: " target="_blank">carlos.camacho@frontiersin.org]
Sent: Tuesday, May 27, 2014 2:55 AM
To: Lamont Granquist
Cc: Adam Edwards; " target="_blank">
Subject: Re: [chef] Re: Re: Re: Re: Re: Re: Re: Re: Problem running
chef-client, just from 2 days ago.

Something really odd just happened.

I restarted the chef server and run the chef-client from Windows and it
worked fine... A few seconds later i executed the same command again and i
had the error.. o.O

Check this out....


PS C:\Users\Administrator> chef-client
Starting Chef Client, version 11.12.4
[2014-05-27T09:43:31+00:00] INFO: *** Chef 11.12.4 ***
[2014-05-27T09:43:31+00:00] INFO: Chef-client pid: 3952
[2014-05-27T09:43:39+00:00] INFO: Run List is [role[windows]]
[2014-05-27T09:43:39+00:00] INFO: Run List expands to []
[2014-05-27T09:43:39+00:00] INFO: Starting Chef Run for slwnone10
[2014-05-27T09:43:39+00:00] INFO: Running start handlers
[2014-05-27T09:43:39+00:00] INFO: Start handlers complete.
[2014-05-27T09:43:39+00:00] INFO: HTTP Request Returned 404 Object Not
Found:
resolving cookbooks for run list: []
[2014-05-27T09:43:39+00:00] INFO: Loading cookbooks [] Synchronizing
Cookbooks:
Compiling Cookbooks...
Converging 0 resources
[2014-05-27T09:43:39+00:00] INFO: Chef Run complete in 0.561614 seconds

Running handlers:
[2014-05-27T09:43:39+00:00] INFO: Running report handlers Running handlers
complete [2014-05-27T09:43:39+00:00] INFO: Report handlers complete Chef
Client finished, 0/0 resources updated in 11.029288 seconds




PS C:\Users\Administrator> chef-client
Starting Chef Client, version 11.12.4
[2014-05-27T09:43:51+00:00] INFO: *** Chef 11.12.4 ***
[2014-05-27T09:43:51+00:00] INFO: Chef-client pid: 3580
[2014-05-27T09:43:59+00:00] INFO: Run List is [role[windows]]
[2014-05-27T09:43:59+00:00] INFO: Run List expands to []
[2014-05-27T09:43:59+00:00] INFO: Starting Chef Run for slwnone10
[2014-05-27T09:43:59+00:00] INFO: Running start handlers
[2014-05-27T09:43:59+00:00] INFO: Start handlers complete.
[2014-05-27T09:43:59+00:00] INFO: HTTP Request Returned 404 Object Not
Found:
resolving cookbooks for run list: []
[2014-05-27T09:43:59+00:00] INFO: Loading cookbooks [] Synchronizing
Cookbooks:
Compiling Cookbooks...
Converging 0 resources

Running handlers:
[2014-05-27T09:43:59+00:00] ERROR: Running exception handlers Running
handlers complete [2014-05-27T09:43:59+00:00] ERROR: Exception handlers
complete [2014-05-27T09:43:59+00:00] FATAL: Stacktrace dumped to
c:/chef/cache/chef-stacktrace.out Chef Client failed. 0 resources updated in
10.670475 seconds [2014-05-27T09:43:59+00:00] FATAL: EOFError: end of file
reached

On Tue, May 27, 2014 at 10:54 AM, Carlos Camacho
< " target="_blank"> org> wrote:

Hello Lamont, Adam.

I'm finishing some tests and the PG error is documented here
(https://tickets.opscode.com/browse/CHEF-4390).

I have a new clue, so please if you can tell me what should be the
differences between these two types of connections.

If i connect the chef node (Windows) using pptpd and the server, it works
fine.
If i connect the chef node (Windows)  and the server using the
standard network, it fails.

Cheers!


On Wed, May 21, 2014 at 8:32 PM, Lamont Granquist < " target="_blank"> >
wrote:

The postgresql logs look interesting.  The failed unique constraints
might be the problem.  I would expect to see stack traces in the
erchef logs though.


On 5/20/14, 12:34 AM, Carlos Camacho wrote:

Hello,

Adam, Lamont good morning.

Ill give it a try to CHEF-5270 because this error is recent (I don't
loose any trying as i'm stuck right now) about 2 weeks ago it
suddenly stop working. Also I'm attaching the chef-client logs in
debug mode and the chef-server logs. But again i don't see any error
in the application layer.

Thanks for your help,

Cheers!


On Mon, May 19, 2014 at 9:15 PM, Lamont Granquist
< " target="_blank"> >
wrote:

On 5/19/14, 12:24 AM, Carlos Camacho wrote:

Hello Julian and Lamont,

I just check all logs in the server and there is not any apparent
error... It so confusing and frustrating this issue..

Well, the server is actually tearing down the connection normally
with a
FIN|ACK, but then the client is still sending data, which produces
FIN|a RST
from the server, then the client receives those and sends back
FIN|ACK and RST of its own.  So TCP is behaving normally and it
does look like this isn't a rackspace issue.  The question is why
the server seems to tear down the connection early while the client
is still attempting to send data.

There should be something either in the nginx or erchef logs on the
server on in the -l debug logs or the stacktrace on the client.