[chef] Re: Re: Possible problem with parallel testing with test-kitchen

[AJ Christensen < >]

You can probably just chuck kb into your base containers too, no?

--AJ

On 17 March 2013 07:05, Scott M. Likens 
<
 >
 wrote:
>
> On Mar 13, 2013, at 2:24 PM, Fletcher Nichol 
> <
 >
>  wrote:
>
>> On Tuesday, 12 March, 2013 at 6:23 PM, Scott M. Likens wrote:
>>> Hi,
>>>
>>> No problem I created it https://github.com/opscode/test-kitchen/issues/68
>>>
>>>
>>> Thanks!
>>>
>>> On Mar 12, 2013, at 12:27 PM, AJ Christensen 
>>> <
 
>>>  
>>> (mailto:
 )>
>>>  wrote:
>>>
>>>> I mentioned this to Fletcher on Twitter, he is aware of the situation!
>>>> Tickets would be good.
>>>>
>>>> Cheers,
>>>>
>>>> AJ
>>>>
>>>> On 13 March 2013 06:48, Scott M. Likens 
>>>> <
 
>>>>  
>>>> (mailto:
 )>
>>>>  wrote:
>>>>> Dan,
>>>>>
>>>>> Unfortunately yes. I was able to reduce it to 1 GET (installation of 
>>>>> bats,
>>>>> minitest is installed via rubygems so it does not hit github) by 
>>>>> modifying
>>>>> neutering the installation script at
>>>>> https://github.com/opscode/kb/blob/go/install to
>>>>> https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
>>>>> lib/kitchen/busser.rb to point to a different INSTALL_URL. With that 
>>>>> small
>>>>> change this reduces the problem greatly.
>>>>>
>>>>> I imagine there are many ways to reduce this pain I'm just not sure of 
>>>>> the
>>>>> quickest and sanest path.
>>>>>
>>>>> If there's anyway I can help let me know
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>> On Mar 11, 2013, at 9:50 PM, Daniel DeLeo 
>>>>> <
 
>>>>>  
>>>>> (mailto:
 )>
>>>>>  wrote:
>>>>>
>>>>> Do conditional GETs or HEAD requests count against the limit?
>>>>>
>>>>> --
>>>>> Daniel DeLeo
>>>>>
>>>>> On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We've been having fun using the lxc driver for testing with test-kitchen
>>>>> (which makes testing really fast!). Unfortunately I forgot that GitHub 
>>>>> has
>>>>> rate limiting imposed now. (60 requests were used up in less than 15
>>>>> minutes (in part because LXC is so fast at creating and destroying 
>>>>> guests
>>>>> and booting new guests ))
>>>>>
>>>>> 
 :~/git/haproxy_lwrp#
>>>>>  curl https://api.github.com/rate_limit -u
>>>>> "damm"
>>>>> Enter host password for user 'damm':
>>>>> {
>>>>> "rate": {
>>>>> "limit": 5000,
>>>>> "remaining": 5000
>>>>> }
>>>>> }
>>>>> 
 :~/git/haproxy_lwrp#
>>>>>  curl https://api.github.com/rate_limit
>>>>> {
>>>>> "rate": {
>>>>> "limit": 60,
>>>>> "remaining": 60
>>>>> }
>>>>> }
>>>>>
>>>>> I believe a few api requests occur in
>>>>> https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100
>>>>>
>>>>> I don't think the right answer is 'can we ship Kitchen Busser with 
>>>>> Omnibus
>>>>> a/k/a Chef-full'.
>>>>>
>>>>> Could we move https://raw.github.com/opscode/kb/go to opscode.com ;
>>>>> (http://opscode.com) like we do
>>>>> with the full installer? curl -L https://www.opscode.com/chef/kb.sh? ;
>>>>> with
>>>>> that we could reduce the github api requests.
>>>>> the basic error seen when getting this can be found
>>>>> https://gist.github.com/damm/5130887
>>>>>
>>>>> P.S. I can create an issue for this, I just wanted to start a 
>>>>> conversation
>>>>> about it and get a general idea of where we wanted to steer this before
>>>>> hand.
>>>>>
>>>>> Thanks!
>>>>
>>>>
>
>> Personally I think it's awesome someone has bumped up against GitHub API 
>> throttling *because* their testing is going that fast. I was taking a look 
>> at this last night and there are 2 places where the git tag API calls are 
>> going to hurt us:
>>
>> * In the install script that installs Kitchen Busser, aka kb 
>> (https://github.com/opscode/kb/blob/e246b11c5761b9d681903d10d00c3c337311409e/install#L55-L63)
>> * In the `kb github-tags` subcommand 
>> (https://github.com/opscode/kb/blob/master/libexec/kb-github-tags)
>>
>> If we had a way to avoid these calls or ask some other endpoint (static or 
>> dynamic), we'd only be asking the GitHub file servers to download the 
>> correct tarballs. I'd like to avoid adding too many moving parts however 
>> it feels like at least one more *something* is called for here (barring 
>> some re-implementation of kb).
>
> In my first implementation at reducing the GitHub API Requests I found 
> reducing just the installation script was good enough to allow me to test 
> without interruptions.  However to scale this further beyond roughly 60 
> tests per hour (if you use bats, minitest does not incur an api request if 
> i recall, which hints at some flexibility)
>
>> Currently I'm toying with a very small web app that can answer a simple 
>> question: what is the latest version-tagging git tag on a github repo? 
>> This could make use of conditional GETs (which don't fall under 
>> throttling) and the responses could be cached using ETAG headers. The only 
>> real change to the kb code would be to call this API to get an answer and 
>> proceed to download off GitHub as usual. Essentially this app would 
>> mitigate the API throttling for everyone in one place.
>
> I would say if we are trying to cache the sources, basically then let's 
> host all the source files for testing (including chef-full, kb, etc) as 
> there might be someone who would want to do testing but cannot because 
> their Firewall blocks Github (yes this number is still shockingly high).
>
>> So why not do conditional GETs and ETAG caching on our workstations and 
>> remote nodes? The problem is that kb lives on those freshly created nodes 
>> which have no past history to draw from. Their first call to the GitHub 
>> API will always be a cache miss. Worse is that in certain environments 
>> (like LXC containers, NAT'ed subnets with VM instances) the source IP to 
>> GitHub could all be the same anyway.
>>
>> Anyway that is what I'd like to spend a bit more time on tonight to flesh 
>> it out. Longer term there could be a more clever solution, but for now 
>> let's get everyone back to fast, throttle-free tests :)
>>
>> Thoughts?
>>
>> Fletcher
>
> My initial thought of implementing this was to stop hardcoding the kb 
> installer and make it available in either .kitchen.local.yml or 
> .kitchen.yml.  Additionally from the evidence above, It's possible to ship 
> kb and it's counterparts up to rubygems to help (this may be enough for 
> people who no internet access to be able to test)
>
> I hope that helps answer your question.
>
> Thanks!
>
> Scott M. Likens
>