[chef] Re: Re: Re: Re: Re: Re: Re: Re: Problem running chef-client, just from 2 days ago.

[Carlos Camacho < >]

Daniel, doing some tcpdump in the chef server...


RST sent from server
07:30:35.013370 IP chef.server.https > mynode.49776: Flags [R], seq
2457752489, win 0, length 0

RST from client
07:30:35.472340 IP mynode.49779 >chef.server.https: Flags [R], seq
3250774265, win 0, length 0


So im a little bit blind here..


This is the chef-server-ctl tail output when chef-client it is executed...

==> /var/log/chef-server/nginx/access.log <==
mynode - - [16/May/2014:07:35:34 +0000]  "GET /nodes/mynode HTTP/1.1"
200 "0.007" 15767 "-" "Chef Client/11.12.4 (ruby-1.9.3-p484;
ohai-7.0.4; i386-mingw32; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.007" "11.12.4" "algorithm=sha1;version=1.0;" "mynode"
"2014-05-16T07:35:38Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 999
mynode - - [16/May/2014:07:35:34 +0000]  "POST
/reports/nodes/mynode/runs HTTP/1.1" 404 "0.044" 173 "-" "Chef
Client/11.12.4 (ruby-1.9.3-p484; ohai-7.0.4; i386-mingw32;
+http://opscode.com)" "127.0.0.1:8000" "404" "0.001" "11.12.4"
"algorithm=sha1;version=1.0;" "mynode" "2014-05-16T07:35:38Z"
"XemsHTvp8BHcocDMY1kiYRTf0dg=" 1214
mynode - - [16/May/2014:07:35:34 +0000]  "POST
/environments/_default/cookbook_versions HTTP/1.1" 200 "0.056" 864 "-"
"Chef Client/11.12.4 (ruby-1.9.3-p484; ohai-7.0.4; i386-mingw32;
+http://opscode.com)" "127.0.0.1:8000" "200" "0.009" "11.12.4"
"algorithm=sha1;version=1.0;" "mynode" "2014-05-16T07:35:39Z"
"oxopNq5HHhVWRiVIxCseXf3iHmU=" 1107
mynode - - [16/May/2014:07:35:35 +0000]  "PUT /nodes/mynode HTTP/1.1"
200 "0.067" 15758 "-" "Chef Client/11.12.4 (ruby-1.9.3-p484;
ohai-7.0.4; i386-mingw32; +http://opscode.com)" "127.0.0.1:8000" "200"
"0.041" "11.12.4" "algorithm=sha1;version=1.0;" "mynode"
"2014-05-16T07:35:39Z" "AI4nEjYvOw1tAkONEqnxBehhvu8=" 124188

==> /var/log/chef-server/chef-solr/current <==
2014-05-16_07:35:35.25015 May 16, 2014 7:35:35 AM
org.apache.solr.core.SolrDeletionPolicy onInit
2014-05-16_07:35:35.25018 INFO: SolrDeletionPolicy.onInit: commits:num=1
2014-05-16_07:35:35.25018
commit{dir=/var/opt/chef-server/chef-solr/data/index,segFN=segments_13,version=1399982839679,generation=39,filenames=[_9.tis,
_7_1.del, _8.tis, _v.fdx, _q.prx, _d.tii, _9.tii, _3.fdt, _c.tii,
_k.fnm, _7.fnm, _7.tis, _4.frq, _c.tis, _h.fdt, _10.tii, _y.nrm,
_h.fdx, _k_1.del, _d.tis, _w.fnm, _u.tii, _10.tis, _a.prx, _g.fdt,
_2.prx, _g.fdx, _u.tis, _5_2.del, _7.tii, _8.frq, _9.fdt, _2.frq,
_h.frq, _9.fdx, _1.fdx, _10.prx, _8.tii, _g.nrm, _1.fdt, _0.frq,
_6.fdt, _t.fdt, _a.fnm, _2.nrm, _0_4.del, _r.nrm, _y.tis, _6.fdx,
_u.fdx, _t.frq, _y.tii, _r_1.del, _6.tis, _x_1.del, _r.fdt, _m.tis,
_w_1.del, _r.fdx, _p.tii, _v.prx, _i.fdx, _m.tii, _y.fnm, _i.fdt,
_2_2.del, _g.prx, _x.fnm, _p.tis, _c_3.del, _z.prx, _i.frq, _10.frq,
_6.tii, _4.nrm, _l_1.del, _5.frq, _g.frq, _2.fnm, _3.fdx, _b.frq,
_x.prx, _a.nrm, _f.tis, _0.tis, _g.tis, _p.fdx, _5.prx, _p.fdt,
_m.prx, _4.prx, _c.fdx, _4.fdx, _c.fdt, _i.tis, _o.prx, _q_1.del,
_p.nrm, _5.nrm, _t_1.del, _g.tii, _7.frq, _i.tii, _s.nrm, _f.tii,
_m.fnm, _e.fnm, _o.fnm, _h.fnm, _j_1.del, _h.tis, _j.tis, _u.nrm,
_1_2.del, _0.tii, _h.tii, _j.tii, _u.fdt, _q.fnm, _6.nrm, _h_1.del,
_z.fdt, _x.nrm, _3.tii, _t.fnm, _z.fdx, _6.frq, _u.frq, _s.fnm,
_a.fdt, _b.nrm, _e.tis, _w.prx, _t.prx, _a.fdx, _s.tii, _e.tii,
_l.tis, _l.tii, _n.nrm, _10.fnm, _5.fnm, _n.prx, _b.prx, _x.frq,
_4.fdt, _b.tis, _b.tii, _8.prx, _s.tis, _f_1.del, _6.prx, _b.fnm,
_b.fdt, _8.fdx, _b.fdx, _8.fdt, _3.tis, _s.fdt, _z.frq, _1.frq,
_a.tii, _m.frq, _e.frq, _2.tii, _y.fdt, _v_2.del, _7.nrm, _y.fdx,
_n.frq, _e_1.del, _q.nrm, _c.nrm, _v.fnm, _g_1.del, _x.tis, _o.frq,
_a.tis, _l.nrm, _d.frq, _w.fdt, _9.prx, _l.prx, _w.fdx, _q.frq,
_c.prx, _2.fdt, _7.fdx, _x.tii, _6_1.del, _2.fdx, _z.nrm, _7.fdt,
_f.fdt, _2.tis, _f.fdx, _l.fdx, segments_13, _l.fdt, _c.frq, _j.fdx,
_x.fdt, _d_1.del, _j.fdt, _x.fdx, _k.prx, _l.frq, _3.fnm, _e.fdx,
_y_1.del, _p.frq, _v.tis, _n.tis, _r.fnm, _o_1.del, _u.fnm, _k.nrm,
_e.fdt, _j.nrm, _q.tis, _n.tii, _1.tii, _j.prx, _e.nrm, _o.fdt,
_q.tii, _1.tis, _o.fdx, _s_1.del, _n.fdt, _9.fnm, _3_2.del, _n.fdx,
_m.nrm, _y.prx, _v.nrm, _d.prx, _r.prx, _d.fdx, _8.nrm, _d.fdt,
_o.nrm, _m.fdt, _v.tii, _m.fdx, _d.nrm, _6.fnm, _c.fnm, _j.frq,
_f.frq, _p.prx, _h.prx, _5.fdt, _r.tii, _a_1.del, _q.fdx, _4.fnm,
_z.fnm, _q.fdt, _o.tis, _1.fnm, _h.nrm, _8_1.del, _k.tii, _v.frq,
_5.tis, _w.tii, _9_1.del, _t.tis, _4_1.del, _0.prx, _n_1.del, _n.fnm,
_k.tis, _w.tis, _3.nrm, _f.nrm, _w.frq, _0.fnm, _t.tii, _a.frq,
_s.prx, _d.fnm, _3.prx, _s.frq, _5.tii, _1.prx, _8.fnm, _f.prx,
_u.prx, _o.tii, _t.nrm, _e.prx, _r.tis, _1.nrm, _i.prx, _i.nrm,
_3.frq, _10.fdt, _z.tii, _u_1.del, _l.fnm, _p.fnm, _9.nrm, _k.fdt,
_y.frq, _z.tis, _k.fdx, _m_1.del, _p_1.del, _10.fdx, _g.fnm, _z_1.del,
_w.nrm, _r.frq, _4.tis, _0.nrm, _4.tii, _v.fdt, _9.frq, _7.prx,
_s.fdx, _f.fnm, _j.fnm, _t.fdx, _5.fdx, _0.fdx, _10.nrm, _k.frq,
_i.fnm, _0.fdt]
2014-05-16_07:35:35.25077 May 16, 2014 7:35:35 AM
org.apache.solr.core.SolrDeletionPolicy updateCommits
2014-05-16_07:35:35.25078 INFO: newest commit = 1399982839679
2014-05-16_07:35:35.26603 May 16, 2014 7:35:35 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
2014-05-16_07:35:35.26606 INFO: {add=[0000000000007c1093a019810992da7a]} 0 31
2014-05-16_07:35:35.26648 May 16, 2014 7:35:35 AM
org.apache.solr.core.SolrCore execute
2014-05-16_07:35:35.26650 INFO: [] webapp=/solr path=/update params={}
status=0 QTime=31

==> /var/log/chef-server/chef-expander/current <==
2014-05-16_07:35:35.26821 [Fri, 16 May 2014 07:35:35 +0000] INFO:
indexed node[0000000000007c1093a019810992da7a]
database[chef_00000000000000000000000000000000] transit,xml,solr-post
| 0,0.10456681251525879,0.14017462730407715 |

==> /var/log/chef-server/erchef/erchef.log.1 <==
2014-05-16T07:35:34Z 

 
 INFO
req_id=I1u0R4xTNbOQrNCbQ19eCg==; status=200; method=GET;
path=/nodes/mynode; user=mynode; msg=[]; req_time=5; rdbms_time=0;
rdbms_count=2
2014-05-16T07:35:34Z 

 
 INFO
req_id=BUGbBDUTtkSFO5WwvomZGg==; status=200; method=POST;
path=/environments/_default/cookbook_versions; user=mynode; msg=[];
req_time=7; rdbms_time=2; rdbms_count=4
2014-05-16T07:35:35Z 

 
 INFO
req_id=DiEmZOzphC/ivnnOmhpvwg==; status=200; method=PUT;
path=/nodes/mynode; user=mynode; msg=[]; req_time=39; rdbms_time=13;
rdbms_count=3


On Fri, May 16, 2014 at 8:58 AM, Carlos Camacho
<
 >
 wrote:
> Daniel, i have the default iptables rules from rackspace.
>
> It is a very long list and i jut run this to clean it a little bit.
>
> iptables --policy INPUT ACCEPT
> iptables --policy OUTPUT ACCEPT
> iptables --policy FORWARD ACCEPT
> iptables -F INPUT
> iptables -F OUTPUT
> iptables -F FORWARD
> iptables -F  RS-RackConnect-INBOUND
> iptables -X  RS-RackConnect-INBOUND
>
> Now the output is this.
>
> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination
>
> Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination
>
> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination
>
>
> Also in windows i disabled the windows firewall (It is important to
> tell that this problem only happens with Windows nodes).
>
>
> Now the chef-client output changed a little bit..
>
> This is now..
>
> [2014-05-16T06:53:30+00:00] INFO: *** Chef 11.12.4 ***
> [2014-05-16T06:53:30+00:00] INFO: Chef-client pid: 3924
> [2014-05-16T06:53:37+00:00] INFO: Run List is [recipe[frontiers_test]]
> [2014-05-16T06:53:37+00:00] INFO: Run List expands to [frontiers_test]
> [2014-05-16T06:53:37+00:00] INFO: Starting Chef Run for slwnone09
> [2014-05-16T06:53:37+00:00] INFO: Running start handlers
> [2014-05-16T06:53:37+00:00] INFO: Start handlers complete.
> [2014-05-16T06:53:37+00:00] INFO: HTTP Request Returned 404 Object Not 
> Found:
> resolving cookbooks for run list: ["frontiers_test"]
> [2014-05-16T06:53:38+00:00] INFO: Loading cookbooks 
> 
 
> Synchronizing Cookbooks:
>   - cookbook_test
> Compiling Cookbooks...
> Converging 1 resources
> Recipe:cookbook_test::default
>   * directory[C:\hello_test] action create[2014-05-16T06:53:38+00:00]
> INFO: Processing directory[C:\hello_test
> ] action create (cookbook_test::default line 10)
>  (up to date)
>
> Running handlers:
> [2014-05-16T06:53:38+00:00] ERROR: Running exception handlers
> Running handlers complete
> [2014-05-16T06:53:38+00:00] ERROR: Exception handlers complete
> [2014-05-16T06:53:38+00:00] FATAL: Stacktrace dumped to
> c:/chef/cache/chef-stacktrace.out
> Chef Client failed. 0 resources updated in 10.324711 seconds
> [2014-05-16T06:53:38+00:00] FATAL: EOFError: end of file reached
> PS C:\Users\Administrator>
>
>
> And before the output was....
>
>
> Starting Chef Client, version 11.12.4
> [2014-05-14T14:02:26+00:00] INFO: *** Chef 11.12.4 ***
> [2014-05-14T14:02:26+00:00] INFO: Chef-client pid: 2936
>
> ================================================================================
> Chef encountered an error attempting to load the node data for "mynode"
> ================================================================================
>
> Unexpected Error:
> -----------------
> EOFError: end of file reached
>
> [2014-05-14T14:02:34+00:00] FATAL: Stacktrace dumped to
> c:/chef/cache/chef-stacktrace.out
> Chef Client failed. 0 resources updated in 10.154057 seconds
> [2014-05-14T14:02:34+00:00] FATAL: EOFError: end of file reached
>
>
> So, it is really annoying not to have any help from rackspace, they
> only tells me, it is a server problem, not network issue....
>
>
> Daniel, if you have any idea im open minded!
>
>
> Cheers!
>
>
> On Thu, May 15, 2014 at 7:08 PM, Daniel DeLeo 
> <
 >
>  wrote:
>>
>>
>> On Thursday, May 15, 2014 at 10:06 AM, Carlos Camacho wrote:
>>
>>> Daniel! Thanks for your answer,
>>>
>>> The RST package it is being send from the chef server, but i don't
>>> know why, or the service which is failing.
>>>
>>> Can you recommend me things to test?!?!
>>>
>>>
>>> Thanks!
>>
>> Is there a firewall on the server itself?
>>
>> --
>> Daniel DeLeo
>>