[chef-dev] Re: Re: Re: Re: Error downloading cookbooks in Chef 11

[Ignasi < >]

Hi again,

Finally I've just been able to fix the issue. If I do not encode the
characters when performing the GET request, it just works.


Thank you all for the help!

ignasi

On 27 August 2013 09:32, Ignasi 
<
 >
 wrote:
> Hi Seth,
>
> That is exactly the case. The failures occur when requesting those files.
>
> I've been investigating a bit more and I've found a pattern for the
> failure. I've seen that the signature validation only fails when the
> URL of the resource has URL encoded characters in the "Signature"
> query parameter. For example, the following two URLs are failing:
>
> In Enterprise Chef:
> https://s3-external-1.amazonaws.com:443/opscode-platform-production-data/organization-<secret>/checksum-f111ba97dc5160e26eee6473dc8e86d5?AWSAccessKeyId=<secret>&Expires=1377590794&Signature=QPnT5XLRam%2Br8p09hXra5N3jd6A%3D
>
> In Chef 11:
> https://ubuntu64-1104.defaultdomain:443/bookshelf/organization-00000000000000000000000000000000/checksum-818a38d41bd95ebec58b6d430eabd7cb?AWSAccessKeyId=<secret>&Expires=1377588824&Signature=31OUx%2BUBdZSyzzA9xpHlFIOxjHc%3D
>
> Both have the '%2B' character in the signature. I've also seen URLs
> with a '%20' that are also failing. However, if the signature does not
> have those URL encoded characters, it just works fine (the padding
> '%3D' character is not causing any failure). Here is an example URL
> that is working:
>
> https://ubuntu64-1104.defaultdomain:443/bookshelf/organization-00000000000000000000000000000000/checksum-a7daae249eca97adcd4b3617c745940d?AWSAccessKeyId=<secret>&Expires=1377589454&Signature=Mp7TFuAyObhtsjgE7wLSv3Sm6WU%3D
>
>
> Is there something special that should be done to treat those encoded
> characters when signing? I've been taking a look at the knife and mix
> lib-authentication source code (a knife cookbook download works fine)
> but still haven't found anything relevant.
>
>
> Many thanks for your help,
>
>
> Ignasi
>
>
>
> On 27 August 2013 00:18, Seth Falcon 
> <
 >
>  wrote:
>>
>> Hi Ignasi,
>>
>> 
 
>>  writes:
>>> I am the maintainer of jclouds-chef [1], a Java library to access
>>> Chef, and just started testing it against Enterprise Chef and Chef 11.
>>> It used to work with Chef 10 and Hosted Chef, and I am just trying to
>>> figure out if there is something that has changed in the signature
>>> verification that is making it fail now with Chef 11 and Enterprise
>>> Chef.
>>>
>>> The signature generation is generic [2] and the same code is used to
>>> sign every request, so I don't understand why I only get failures when
>>> trying to download the contents of a file (and why it only fails in
>>> Chef 11 and Hosted Chef).
>>
>> There are a number of significant changes introduced in Chef 11 and more
>> recently deployed into Hosted Enterprise Chef (HEC).
>>
>> Here's my understanding of the problem you're seeing, please let me know
>> if I'm off:
>>
>> 1. You can use jclouds-chef to make some successful API requests against
>> Open Source Chef 11 Server (OSC) and against your org in Hosted
>> Enterprise Chef.
>>
>> 2. You are seeing intermittent 403 errors when doing GETs of cookbook
>> content.
>>
>>> I can provide the logs of the generated requests and the obtained
>>> responses, if it helps.
>>
>> It would help to have examples of the type of URL you are attempting to
>> access (you can put in placeholders to maintain privacy if you will be
>> posting this to the list).
>>
>> Similarly, associated responses would be helpful.
>>
>> Cookbook content, by which I mean the files in a cookbook stored by
>> checksum, are handled outside of the core server in both OSC 11 and HEC.
>>
>> OSC 11 uses a newly added component called "bookshelf" to serve the
>> cookbook content. It provides a think S3 API.
>>
>> HEC stores cookbook content in AWS S3.
>>
>> In both cases, when fetching a cookbook, the client receives a cookbook
>> version object containing pre-signed URLs that it can use to fetch the
>> cookbook content out of either bookshelf or S3. Although it is OK to
>> sign those requests like other API requests, both bookshelf and S3 use
>> S3-style authentication. If these are the requests that are failing,
>> inspecting the response body might give us a clue of where to look.
>>
>> + seth
>>
>>
>>
>> --
>> Seth Falcon | Development Lead | Opscode | @sfalcon
>>