[chef] Bizarre EOFError on Google Compute Engine, behind NAT

[Jeff Goldschrafe < >]

Scenario: hybrid server environment running Chef 11.14.2/11.1.6 on a mix of
physical and virtual Ubuntu 12.04 systems. The virtual instances are a mix of
EC2 and Google Compute Engine. The instances are a combination of publicly
accessible (static/Elastic IP) and private (behind NAT). The systems on GCE
behind NAT receive this error when they attempt to request
/environments/<environment>/cookbook_versions:

2014-10-18T07:49:41+00:00] DEBUG: EOFError: end of file reached
/opt/chef/embedded/lib/ruby/1.9.1/openssl/buffering.rb:174:in
`sysread_nonblock'
/opt/chef/embedded/lib/ruby/1.9.1/openssl/buffering.rb:174:in `read_nonblock'
/opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
/opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:92:in `read'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2780:in `ensure in read_chunked'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2780:in `read_chunked'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2751:in `read_body_0'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2711:in `read_body'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:262:in
`block (2 levels) in send_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http/basic_client.rb:74:in
`block in request'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1323:in `block (2 levels) in
transport_request'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2672:in `reading_body'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1322:in `block in
transport_request'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1317:in `catch'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1317:in `transport_request'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1294:in `request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in
`request'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1287:in `block in request'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:746:in `start'
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1285:in `request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in
`request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http/basic_client.rb:65:in
`request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:262:in
`block in send_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:294:in
`block in retrying_http_errors'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:292:in
`loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:292:in
`retrying_http_errors'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:256:in
`send_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:143:in
`request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/http.rb:126:in
`post'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/policy_builder/expand_node_object.rb:168:in
`sync_cookbooks'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/policy_builder/expand_node_object.rb:66:in
`setup_run_context'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:265:in
`setup_run_context'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:429:in
`do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:213:in
`block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:207:in
`fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/client.rb:207:in
`run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application.rb:236:in
`run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application/client.rb:338:in
`block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application/client.rb:327:in
`loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application/client.rb:327:in
`run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/lib/chef/application.rb:55:in
`run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.16.4/bin/chef-client:26:in
`<top (required)>'
/usr/bin/chef-client:23:in `load'
/usr/bin/chef-client:23:in `<main>'

The issue is 100% tied to the NAT topology, or my implementation thereof, 
which
is vanilla iptables masquerading on Ubuntu 12.04. I can route the same 
instance
along a public IP and it works fine. The second I push the route back through
the NAT, I get this error again. The Chef Server indicates that a 200 was
served, both through Erchef and Nginx. The NAT gateway itself is also managed
through Chef, and the Chef cilent on this system works just fine without
generating the error above.

The error seems to be reproducible about 99% of the time; however, if it does
not fail here, it fails at another API call somewhere down the path. It's
frustrating that it occurs just slightly less than always.

I do not receive this error on SSL connections to other services, including
large file downloads. I can comfortably pull a 1 GB+ file from Amazon S3 or
Google Cloud Storage. I can clone the Linux kernel repository from GitHub 
using
HTTPS. This Chef server triggers the EOFError on this particular REST API 
call,
when the system is located behind a NAT gateway on Google Compute Engine but
not on Amazon.

Packet captures don't show anything tremendously out of the ordinary besides
some out-of-order packets that I'm blaming on GCE, but if anyone knows what's
special about this particular call that might lead me to what's up with this
networking configuration, it would be very much appreciated.