[chef] Re: Re: Re: Re: Re: Re: Re: Re: Re: Problem running chef-client, just from 2 days ago.

["Julian C. Dunn" < >]

Is there anything in the kernel messages (dmesg, /var/log/messages,
etc.) of the Chef server? Could it be that the kernel is sending the
TCP RST for some reason?

Windows nodes often send a LOT of data, so maybe that has something to
do with it...

- Julian

On Fri, May 16, 2014 at 2:38 AM, Carlos Camacho
<
 >
 wrote:
> Daniel, doing some tcpdump in the chef server...
>
>
> RST sent from server
> 07:30:35.013370 IP chef.server.https > mynode.49776: Flags [R], seq
> 2457752489, win 0, length 0
>
> RST from client
> 07:30:35.472340 IP mynode.49779 >chef.server.https: Flags [R], seq
> 3250774265, win 0, length 0
>
>
> So im a little bit blind here..
>
>
> This is the chef-server-ctl tail output when chef-client it is executed...
>
> ==> /var/log/chef-server/nginx/access.log <==
> mynode - - [16/May/2014:07:35:34 +0000]  "GET /nodes/mynode HTTP/1.1"
> 200 "0.007" 15767 "-" "Chef Client/11.12.4 (ruby-1.9.3-p484;
> ohai-7.0.4; i386-mingw32; +http://opscode.com)" "127.0.0.1:8000" "200"
> "0.007" "11.12.4" "algorithm=sha1;version=1.0;" "mynode"
> "2014-05-16T07:35:38Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 999
> mynode - - [16/May/2014:07:35:34 +0000]  "POST
> /reports/nodes/mynode/runs HTTP/1.1" 404 "0.044" 173 "-" "Chef
> Client/11.12.4 (ruby-1.9.3-p484; ohai-7.0.4; i386-mingw32;
> +http://opscode.com)" "127.0.0.1:8000" "404" "0.001" "11.12.4"
> "algorithm=sha1;version=1.0;" "mynode" "2014-05-16T07:35:38Z"
> "XemsHTvp8BHcocDMY1kiYRTf0dg=" 1214
> mynode - - [16/May/2014:07:35:34 +0000]  "POST
> /environments/_default/cookbook_versions HTTP/1.1" 200 "0.056" 864 "-"
> "Chef Client/11.12.4 (ruby-1.9.3-p484; ohai-7.0.4; i386-mingw32;
> +http://opscode.com)" "127.0.0.1:8000" "200" "0.009" "11.12.4"
> "algorithm=sha1;version=1.0;" "mynode" "2014-05-16T07:35:39Z"
> "oxopNq5HHhVWRiVIxCseXf3iHmU=" 1107
> mynode - - [16/May/2014:07:35:35 +0000]  "PUT /nodes/mynode HTTP/1.1"
> 200 "0.067" 15758 "-" "Chef Client/11.12.4 (ruby-1.9.3-p484;
> ohai-7.0.4; i386-mingw32; +http://opscode.com)" "127.0.0.1:8000" "200"
> "0.041" "11.12.4" "algorithm=sha1;version=1.0;" "mynode"
> "2014-05-16T07:35:39Z" "AI4nEjYvOw1tAkONEqnxBehhvu8=" 124188
>
> ==> /var/log/chef-server/chef-solr/current <==
> 2014-05-16_07:35:35.25015 May 16, 2014 7:35:35 AM
> org.apache.solr.core.SolrDeletionPolicy onInit
> 2014-05-16_07:35:35.25018 INFO: SolrDeletionPolicy.onInit: commits:num=1
> 2014-05-16_07:35:35.25018
> commit{dir=/var/opt/chef-server/chef-solr/data/index,segFN=segments_13,version=1399982839679,generation=39,filenames=[_9.tis,
> _7_1.del, _8.tis, _v.fdx, _q.prx, _d.tii, _9.tii, _3.fdt, _c.tii,
> _k.fnm, _7.fnm, _7.tis, _4.frq, _c.tis, _h.fdt, _10.tii, _y.nrm,
> _h.fdx, _k_1.del, _d.tis, _w.fnm, _u.tii, _10.tis, _a.prx, _g.fdt,
> _2.prx, _g.fdx, _u.tis, _5_2.del, _7.tii, _8.frq, _9.fdt, _2.frq,
> _h.frq, _9.fdx, _1.fdx, _10.prx, _8.tii, _g.nrm, _1.fdt, _0.frq,
> _6.fdt, _t.fdt, _a.fnm, _2.nrm, _0_4.del, _r.nrm, _y.tis, _6.fdx,
> _u.fdx, _t.frq, _y.tii, _r_1.del, _6.tis, _x_1.del, _r.fdt, _m.tis,
> _w_1.del, _r.fdx, _p.tii, _v.prx, _i.fdx, _m.tii, _y.fnm, _i.fdt,
> _2_2.del, _g.prx, _x.fnm, _p.tis, _c_3.del, _z.prx, _i.frq, _10.frq,
> _6.tii, _4.nrm, _l_1.del, _5.frq, _g.frq, _2.fnm, _3.fdx, _b.frq,
> _x.prx, _a.nrm, _f.tis, _0.tis, _g.tis, _p.fdx, _5.prx, _p.fdt,
> _m.prx, _4.prx, _c.fdx, _4.fdx, _c.fdt, _i.tis, _o.prx, _q_1.del,
> _p.nrm, _5.nrm, _t_1.del, _g.tii, _7.frq, _i.tii, _s.nrm, _f.tii,
> _m.fnm, _e.fnm, _o.fnm, _h.fnm, _j_1.del, _h.tis, _j.tis, _u.nrm,
> _1_2.del, _0.tii, _h.tii, _j.tii, _u.fdt, _q.fnm, _6.nrm, _h_1.del,
> _z.fdt, _x.nrm, _3.tii, _t.fnm, _z.fdx, _6.frq, _u.frq, _s.fnm,
> _a.fdt, _b.nrm, _e.tis, _w.prx, _t.prx, _a.fdx, _s.tii, _e.tii,
> _l.tis, _l.tii, _n.nrm, _10.fnm, _5.fnm, _n.prx, _b.prx, _x.frq,
> _4.fdt, _b.tis, _b.tii, _8.prx, _s.tis, _f_1.del, _6.prx, _b.fnm,
> _b.fdt, _8.fdx, _b.fdx, _8.fdt, _3.tis, _s.fdt, _z.frq, _1.frq,
> _a.tii, _m.frq, _e.frq, _2.tii, _y.fdt, _v_2.del, _7.nrm, _y.fdx,
> _n.frq, _e_1.del, _q.nrm, _c.nrm, _v.fnm, _g_1.del, _x.tis, _o.frq,
> _a.tis, _l.nrm, _d.frq, _w.fdt, _9.prx, _l.prx, _w.fdx, _q.frq,
> _c.prx, _2.fdt, _7.fdx, _x.tii, _6_1.del, _2.fdx, _z.nrm, _7.fdt,
> _f.fdt, _2.tis, _f.fdx, _l.fdx, segments_13, _l.fdt, _c.frq, _j.fdx,
> _x.fdt, _d_1.del, _j.fdt, _x.fdx, _k.prx, _l.frq, _3.fnm, _e.fdx,
> _y_1.del, _p.frq, _v.tis, _n.tis, _r.fnm, _o_1.del, _u.fnm, _k.nrm,
> _e.fdt, _j.nrm, _q.tis, _n.tii, _1.tii, _j.prx, _e.nrm, _o.fdt,
> _q.tii, _1.tis, _o.fdx, _s_1.del, _n.fdt, _9.fnm, _3_2.del, _n.fdx,
> _m.nrm, _y.prx, _v.nrm, _d.prx, _r.prx, _d.fdx, _8.nrm, _d.fdt,
> _o.nrm, _m.fdt, _v.tii, _m.fdx, _d.nrm, _6.fnm, _c.fnm, _j.frq,
> _f.frq, _p.prx, _h.prx, _5.fdt, _r.tii, _a_1.del, _q.fdx, _4.fnm,
> _z.fnm, _q.fdt, _o.tis, _1.fnm, _h.nrm, _8_1.del, _k.tii, _v.frq,
> _5.tis, _w.tii, _9_1.del, _t.tis, _4_1.del, _0.prx, _n_1.del, _n.fnm,
> _k.tis, _w.tis, _3.nrm, _f.nrm, _w.frq, _0.fnm, _t.tii, _a.frq,
> _s.prx, _d.fnm, _3.prx, _s.frq, _5.tii, _1.prx, _8.fnm, _f.prx,
> _u.prx, _o.tii, _t.nrm, _e.prx, _r.tis, _1.nrm, _i.prx, _i.nrm,
> _3.frq, _10.fdt, _z.tii, _u_1.del, _l.fnm, _p.fnm, _9.nrm, _k.fdt,
> _y.frq, _z.tis, _k.fdx, _m_1.del, _p_1.del, _10.fdx, _g.fnm, _z_1.del,
> _w.nrm, _r.frq, _4.tis, _0.nrm, _4.tii, _v.fdt, _9.frq, _7.prx,
> _s.fdx, _f.fnm, _j.fnm, _t.fdx, _5.fdx, _0.fdx, _10.nrm, _k.frq,
> _i.fnm, _0.fdt]
> 2014-05-16_07:35:35.25077 May 16, 2014 7:35:35 AM
> org.apache.solr.core.SolrDeletionPolicy updateCommits
> 2014-05-16_07:35:35.25078 INFO: newest commit = 1399982839679
> 2014-05-16_07:35:35.26603 May 16, 2014 7:35:35 AM
> org.apache.solr.update.processor.LogUpdateProcessor finish
> 2014-05-16_07:35:35.26606 INFO: {add=[0000000000007c1093a019810992da7a]} 0 
> 31
> 2014-05-16_07:35:35.26648 May 16, 2014 7:35:35 AM
> org.apache.solr.core.SolrCore execute
> 2014-05-16_07:35:35.26650 INFO: [] webapp=/solr path=/update params={}
> status=0 QTime=31
>
> ==> /var/log/chef-server/chef-expander/current <==
> 2014-05-16_07:35:35.26821 [Fri, 16 May 2014 07:35:35 +0000] INFO:
> indexed node[0000000000007c1093a019810992da7a]
> database[chef_00000000000000000000000000000000] transit,xml,solr-post
> | 0,0.10456681251525879,0.14017462730407715 |
>
> ==> /var/log/chef-server/erchef/erchef.log.1 <==
> 2014-05-16T07:35:34Z 
> 
 
>  INFO
> req_id=I1u0R4xTNbOQrNCbQ19eCg==; status=200; method=GET;
> path=/nodes/mynode; user=mynode; msg=[]; req_time=5; rdbms_time=0;
> rdbms_count=2
> 2014-05-16T07:35:34Z 
> 
 
>  INFO
> req_id=BUGbBDUTtkSFO5WwvomZGg==; status=200; method=POST;
> path=/environments/_default/cookbook_versions; user=mynode; msg=[];
> req_time=7; rdbms_time=2; rdbms_count=4
> 2014-05-16T07:35:35Z 
> 
 
>  INFO
> req_id=DiEmZOzphC/ivnnOmhpvwg==; status=200; method=PUT;
> path=/nodes/mynode; user=mynode; msg=[]; req_time=39; rdbms_time=13;
> rdbms_count=3
>
>
> On Fri, May 16, 2014 at 8:58 AM, Carlos Camacho
> <
 >
>  wrote:
>> Daniel, i have the default iptables rules from rackspace.
>>
>> It is a very long list and i jut run this to clean it a little bit.
>>
>> iptables --policy INPUT ACCEPT
>> iptables --policy OUTPUT ACCEPT
>> iptables --policy FORWARD ACCEPT
>> iptables -F INPUT
>> iptables -F OUTPUT
>> iptables -F FORWARD
>> iptables -F  RS-RackConnect-INBOUND
>> iptables -X  RS-RackConnect-INBOUND
>>
>> Now the output is this.
>>
>> Chain INPUT (policy ACCEPT)
>> target     prot opt source               destination
>>
>> Chain FORWARD (policy ACCEPT)
>> target     prot opt source               destination
>>
>> Chain OUTPUT (policy ACCEPT)
>> target     prot opt source               destination
>>
>>
>> Also in windows i disabled the windows firewall (It is important to
>> tell that this problem only happens with Windows nodes).
>>
>>
>> Now the chef-client output changed a little bit..
>>
>> This is now..
>>
>> [2014-05-16T06:53:30+00:00] INFO: *** Chef 11.12.4 ***
>> [2014-05-16T06:53:30+00:00] INFO: Chef-client pid: 3924
>> [2014-05-16T06:53:37+00:00] INFO: Run List is [recipe[frontiers_test]]
>> [2014-05-16T06:53:37+00:00] INFO: Run List expands to [frontiers_test]
>> [2014-05-16T06:53:37+00:00] INFO: Starting Chef Run for slwnone09
>> [2014-05-16T06:53:37+00:00] INFO: Running start handlers
>> [2014-05-16T06:53:37+00:00] INFO: Start handlers complete.
>> [2014-05-16T06:53:37+00:00] INFO: HTTP Request Returned 404 Object Not 
>> Found:
>> resolving cookbooks for run list: ["frontiers_test"]
>> [2014-05-16T06:53:38+00:00] INFO: Loading cookbooks 
>> 
 
>> Synchronizing Cookbooks:
>>   - cookbook_test
>> Compiling Cookbooks...
>> Converging 1 resources
>> Recipe:cookbook_test::default
>>   * directory[C:\hello_test] action create[2014-05-16T06:53:38+00:00]
>> INFO: Processing directory[C:\hello_test
>> ] action create (cookbook_test::default line 10)
>>  (up to date)
>>
>> Running handlers:
>> [2014-05-16T06:53:38+00:00] ERROR: Running exception handlers
>> Running handlers complete
>> [2014-05-16T06:53:38+00:00] ERROR: Exception handlers complete
>> [2014-05-16T06:53:38+00:00] FATAL: Stacktrace dumped to
>> c:/chef/cache/chef-stacktrace.out
>> Chef Client failed. 0 resources updated in 10.324711 seconds
>> [2014-05-16T06:53:38+00:00] FATAL: EOFError: end of file reached
>> PS C:\Users\Administrator>
>>
>>
>> And before the output was....
>>
>>
>> Starting Chef Client, version 11.12.4
>> [2014-05-14T14:02:26+00:00] INFO: *** Chef 11.12.4 ***
>> [2014-05-14T14:02:26+00:00] INFO: Chef-client pid: 2936
>>
>> ================================================================================
>> Chef encountered an error attempting to load the node data for "mynode"
>> ================================================================================
>>
>> Unexpected Error:
>> -----------------
>> EOFError: end of file reached
>>
>> [2014-05-14T14:02:34+00:00] FATAL: Stacktrace dumped to
>> c:/chef/cache/chef-stacktrace.out
>> Chef Client failed. 0 resources updated in 10.154057 seconds
>> [2014-05-14T14:02:34+00:00] FATAL: EOFError: end of file reached
>>
>>
>> So, it is really annoying not to have any help from rackspace, they
>> only tells me, it is a server problem, not network issue....
>>
>>
>> Daniel, if you have any idea im open minded!
>>
>>
>> Cheers!
>>
>>
>> On Thu, May 15, 2014 at 7:08 PM, Daniel DeLeo 
>> <
 >
>>  wrote:
>>>
>>>
>>> On Thursday, May 15, 2014 at 10:06 AM, Carlos Camacho wrote:
>>>
>>>> Daniel! Thanks for your answer,
>>>>
>>>> The RST package it is being send from the chef server, but i don't
>>>> know why, or the service which is failing.
>>>>
>>>> Can you recommend me things to test?!?!
>>>>
>>>>
>>>> Thanks!
>>>
>>> Is there a firewall on the server itself?
>>>
>>> --
>>> Daniel DeLeo
>>>



-- 
[ Julian C. Dunn 
<
 >
          * Sorry, I'm    ]
[ WWW: http://www.aquezada.com/staff/julian    * only Web 1.0  ;]
[ gopher://sdf.org/1/users/keymaker/           * compliant!    ;]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9       ]