[chef] Re: Possible problem with parallel testing with test-kitchen

["Scott M. Likens" < >]

AJ,

Certainly but that basically locks you into a version of 'kb' for testing.  
Short term that may be do-able but that seems like a slippery crevice i'd 
rather avoid.

There is no reason you cannot host kb on s3 and have it use directory indexes 
to 'walk' and determine the version to install, and i'm certain there are 
other ways to do this as well.  So possibly making the INSTALL_URL exposed in 
.kitchen.local.yml could be a good beginning.

Thanks!

On Mar 16, 2013, at 10:27 PM, AJ Christensen 
<
 >
 wrote:

> You can probably just chuck kb into your base containers too, no?
> 
> --AJ
> 
> On 17 March 2013 07:05, Scott M. Likens 
> <
 >
>  wrote:
>> 
>> On Mar 13, 2013, at 2:24 PM, Fletcher Nichol 
>> <
 >
>>  wrote:
>> 
>>> On Tuesday, 12 March, 2013 at 6:23 PM, Scott M. Likens wrote:
>>>> Hi,
>>>> 
>>>> No problem I created it https://github.com/opscode/test-kitchen/issues/68
>>>> 
>>>> 
>>>> Thanks!
>>>> 
>>>> On Mar 12, 2013, at 12:27 PM, AJ Christensen 
>>>> <
 
>>>>  
>>>> (mailto:
 )>
>>>>  wrote:
>>>> 
>>>>> I mentioned this to Fletcher on Twitter, he is aware of the situation!
>>>>> Tickets would be good.
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> AJ
>>>>> 
>>>>> On 13 March 2013 06:48, Scott M. Likens 
>>>>> <
 
>>>>>  
>>>>> (mailto:
 )>
>>>>>  wrote:
>>>>>> Dan,
>>>>>> 
>>>>>> Unfortunately yes. I was able to reduce it to 1 GET (installation of 
>>>>>> bats,
>>>>>> minitest is installed via rubygems so it does not hit github) by 
>>>>>> modifying
>>>>>> neutering the installation script at
>>>>>> https://github.com/opscode/kb/blob/go/install to
>>>>>> https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
>>>>>> lib/kitchen/busser.rb to point to a different INSTALL_URL. With that 
>>>>>> small
>>>>>> change this reduces the problem greatly.
>>>>>> 
>>>>>> I imagine there are many ways to reduce this pain I'm just not sure of 
>>>>>> the
>>>>>> quickest and sanest path.
>>>>>> 
>>>>>> If there's anyway I can help let me know
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mar 11, 2013, at 9:50 PM, Daniel DeLeo 
>>>>>> <
 
>>>>>>  
>>>>>> (mailto:
 )>
>>>>>>  wrote:
>>>>>> 
>>>>>> Do conditional GETs or HEAD requests count against the limit?
>>>>>> 
>>>>>> --
>>>>>> Daniel DeLeo
>>>>>> 
>>>>>> On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> We've been having fun using the lxc driver for testing with 
>>>>>> test-kitchen
>>>>>> (which makes testing really fast!). Unfortunately I forgot that GitHub 
>>>>>> has
>>>>>> rate limiting imposed now. (60 requests were used up in less than 15
>>>>>> minutes (in part because LXC is so fast at creating and destroying 
>>>>>> guests
>>>>>> and booting new guests ))
>>>>>> 
>>>>>> 
 :~/git/haproxy_lwrp#
>>>>>>  curl https://api.github.com/rate_limit -u
>>>>>> "damm"
>>>>>> Enter host password for user 'damm':
>>>>>> {
>>>>>> "rate": {
>>>>>> "limit": 5000,
>>>>>> "remaining": 5000
>>>>>> }
>>>>>> }
>>>>>> 
 :~/git/haproxy_lwrp#
>>>>>>  curl https://api.github.com/rate_limit
>>>>>> {
>>>>>> "rate": {
>>>>>> "limit": 60,
>>>>>> "remaining": 60
>>>>>> }
>>>>>> }
>>>>>> 
>>>>>> I believe a few api requests occur in
>>>>>> https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100
>>>>>> 
>>>>>> I don't think the right answer is 'can we ship Kitchen Busser with 
>>>>>> Omnibus
>>>>>> a/k/a Chef-full'.
>>>>>> 
>>>>>> Could we move https://raw.github.com/opscode/kb/go to opscode.com ;
>>>>>> (http://opscode.com) like we do
>>>>>> with the full installer? curl -L https://www.opscode.com/chef/kb.sh? ;
>>>>>> with
>>>>>> that we could reduce the github api requests.
>>>>>> the basic error seen when getting this can be found
>>>>>> https://gist.github.com/damm/5130887
>>>>>> 
>>>>>> P.S. I can create an issue for this, I just wanted to start a 
>>>>>> conversation
>>>>>> about it and get a general idea of where we wanted to steer this before
>>>>>> hand.
>>>>>> 
>>>>>> Thanks!
>>>>> 
>>>>> 
>> 
>>> Personally I think it's awesome someone has bumped up against GitHub API 
>>> throttling *because* their testing is going that fast. I was taking a 
>>> look at this last night and there are 2 places where the git tag API 
>>> calls are going to hurt us:
>>> 
>>> * In the install script that installs Kitchen Busser, aka kb 
>>> (https://github.com/opscode/kb/blob/e246b11c5761b9d681903d10d00c3c337311409e/install#L55-L63)
>>> * In the `kb github-tags` subcommand 
>>> (https://github.com/opscode/kb/blob/master/libexec/kb-github-tags)
>>> 
>>> If we had a way to avoid these calls or ask some other endpoint (static 
>>> or dynamic), we'd only be asking the GitHub file servers to download the 
>>> correct tarballs. I'd like to avoid adding too many moving parts however 
>>> it feels like at least one more *something* is called for here (barring 
>>> some re-implementation of kb).
>> 
>> In my first implementation at reducing the GitHub API Requests I found 
>> reducing just the installation script was good enough to allow me to test 
>> without interruptions.  However to scale this further beyond roughly 60 
>> tests per hour (if you use bats, minitest does not incur an api request if 
>> i recall, which hints at some flexibility)
>> 
>>> Currently I'm toying with a very small web app that can answer a simple 
>>> question: what is the latest version-tagging git tag on a github repo? 
>>> This could make use of conditional GETs (which don't fall under 
>>> throttling) and the responses could be cached using ETAG headers. The 
>>> only real change to the kb code would be to call this API to get an 
>>> answer and proceed to download off GitHub as usual. Essentially this app 
>>> would mitigate the API throttling for everyone in one place.
>> 
>> I would say if we are trying to cache the sources, basically then let's 
>> host all the source files for testing (including chef-full, kb, etc) as 
>> there might be someone who would want to do testing but cannot because 
>> their Firewall blocks Github (yes this number is still shockingly high).
>> 
>>> So why not do conditional GETs and ETAG caching on our workstations and 
>>> remote nodes? The problem is that kb lives on those freshly created nodes 
>>> which have no past history to draw from. Their first call to the GitHub 
>>> API will always be a cache miss. Worse is that in certain environments 
>>> (like LXC containers, NAT'ed subnets with VM instances) the source IP to 
>>> GitHub could all be the same anyway.
>>> 
>>> Anyway that is what I'd like to spend a bit more time on tonight to flesh 
>>> it out. Longer term there could be a more clever solution, but for now 
>>> let's get everyone back to fast, throttle-free tests :)
>>> 
>>> Thoughts?
>>> 
>>> Fletcher
>> 
>> My initial thought of implementing this was to stop hardcoding the kb 
>> installer and make it available in either .kitchen.local.yml or 
>> .kitchen.yml.  Additionally from the evidence above, It's possible to ship 
>> kb and it's counterparts up to rubygems to help (this may be enough for 
>> people who no internet access to be able to test)
>> 
>> I hope that helps answer your question.
>> 
>> Thanks!
>> 
>> Scott M. Likens
>> 
> 
> !DSPAM:51455479147001804284693!
>