[chef] Re: Chef Upgrade issue

[Seth Falcon < >]

Hi there,

On Apr 30, 2013, at 7:44 , 

 
 wrote:

> Seems to be the problem in the bookshelf.
> Every uploaded cookbooks contents look's like this for example:
> 
> �
> wT    
> 
 
>  �#
> # Cookbook Name:: yumrepo
> # Recipe:: somerepo

That is likely correct and expected. When bookshelf writes cookbook content 
to its data directory, it prepends the file contents with a binary header. 
The header consists of a two byte magic number followed by 16 byte checksum 
of the content following the header.

I think what you are running into is related to bookshelf attempting to 
stream responses to the client using HTTP 1.1 chunked transfer coding and 
something in between bookshelf and the client not behaving well.


> After chef-client run it will become like this:

> 1b4^M

^^^ that bit looks like the chunked transfer protocol, specifying the chunk 
size in bytes written in hex.

> #
> # Cookbook Name:: yumrepo
> # Recipe:: somrepo
> #
> 
> remote_file "/etc/yum.repos.d/somerepo.repo" do
>  not_if {File.exists?("/etc/yum.repos.d/somerepo.repo")}
>  owner "root"
>  group "root"
>  mode 0644
>  source url
> end
> ^M
> 0^M
> ^M

^^^ And that looks like the last-chunk message.

Can you provide some detail about how you have installed and configured your 
Chef Server? Of particular interest would be detail about any load balancers 
or proxies that are between your clients and the server.

You can disable the streaming download behavior of bookshelf by putting the 
following in chef-server.rb and running a chef-server-ctl reconfigure:

    bookshelf['stream_download'] = false

Performance, especially in terms of memory use of bookshelf should be better 
with streaming.

We've seen this behavior before when we were first integrating bookshelf into 
the server. At the time, we were running an older version of nginx as the 
load balancer. The older version did not support proxying HTTP 1.1 servers 
and broke things.

+ seth