[chef] Re: Knife upload cookbook slowness

[Geoff Meakin Acid < >]

Hi all,

Just to clear things up in case anybody was interested, I finally got to the bottom of this. I thought you might like to know why although it's not really chef-related.

The problem was I installed my chef infrastructure on a 'example.local' domain.

By ruby-debugging most of chef to find the extremely slow code, it turned out to be Net::HTTP. I then realised that I was using a Mac (OS X Lion), and Net::HTTP uses the mDNSResolver BSD tools which were preferred over /etc/resolv.conf, /etc/hosts, and anything else since OS X Snow Leopard and above use it to do its resolving. (Tricky to debug as standard tools like 'telnet', 'dig', 'ping', and 'host' use completely different resolver pathways on Mac OS X)

Turning on lots of debug on mDNSResolver, it became clear that requests to example.local were sending out ipv6 dns multicast packets which weren't being returned as the domain was on ipv4. I came across this article: http://serverfault.com/questions/286185/multicast-hostname-lookups-on-osx  which explains the rest

Basically, when running spiceweasal and/or knife commands, each request is treated separately, each file (e.g. in cookbooks) is a separate upload request to the chef server, and each request had a +3-5seconds delay DNS lookup, just because its on the .local domain.

When I switched my infrastructure over to an 'example.other' domain, knife ran just as fast as a rake install on the chef server itself.

I guess chef could help this problem (and speed up generally) by uploading multiple assets in one request.. so there is not so much http wrapping to do , but i fixed the real problem anyhu

Cheers

Geoff

On 12 Sep 2011, at 17:48, Stephen Delano wrote:

Hi Geoff,

`knife cookbook upload` is definitely not the fastest kid on the block, but "HOURS" to upload 20 cookbooks seems quite a bit out of the ordinary.

First off, are you uploading to an open source Chef server or Opscode's Hosted Chef?

It seems to do a whole separate signed certificate-authenticated request for each little part of the cookbook it uploads looking at the debug..

If you're using the open source Chef server, you would be correct. You are uploading files directly to the server, so the requests must be signed. You wouldn't want someone dumping files onto your chef-server just by hitting the API would you?

If you're using Hosted Chef, the individual file uploads are not signed, but are instead sent to pre-signed, time-expired URLs that verify checksum integrity upon completion.

How many files do you have in this collection of 20 cookbooks?

What does your chef-server infrastructure look like? Is there a load-balancer, firewall, or some other appliance between you and your chef-server?

Cheers,

Stephen

On Mon, Sep 12, 2011 at 8:36 AM, Geoff Meakin Acid < "> > wrote:

Hi all,

Does anybody else find knife uploading cookbooks slow.. I mean really slow… (chef 0.10)

I have site installed about 20 cookbooks, but to upload them to my own chef server using the knife command takes >>HOURS

I have fallen back on rake installing from the server instead, (which takes a few seconds in comparison!) simply because knife uploading is taking so long it isnt really a viable option for me - but I cant think that that's what was intended. It seems to do a whole separate signed certificate-authenticated request for each little part of the cookbook it uploads looking at the debug..

Cheers
Geoff

--
Stephen Delano
Software Development Engineer
Opscode, Inc.
1008 Western Avenue
Suite 601
Seattle, WA 98104