[chef] Re: Re: Possible problem with parallel testing with test-kitchen

[Fletcher Nichol < >]

On Tuesday, 12 March, 2013 at 6:23 PM, Scott M. Likens wrote:
> Hi,
> 
> No problem I created it https://github.com/opscode/test-kitchen/issues/68
> 
> 
> Thanks!
> 
> On Mar 12, 2013, at 12:27 PM, AJ Christensen 
> <
 
>  
> (mailto:
 )>
>  wrote:
> 
> > I mentioned this to Fletcher on Twitter, he is aware of the situation!
> > Tickets would be good.
> > 
> > Cheers,
> > 
> > AJ
> > 
> > On 13 March 2013 06:48, Scott M. Likens 
> > <
 
> >  
> > (mailto:
 )>
> >  wrote:
> > > Dan,
> > > 
> > > Unfortunately yes. I was able to reduce it to 1 GET (installation of 
> > > bats,
> > > minitest is installed via rubygems so it does not hit github) by 
> > > modifying
> > > neutering the installation script at
> > > https://github.com/opscode/kb/blob/go/install to
> > > https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
> > > lib/kitchen/busser.rb to point to a different INSTALL_URL. With that 
> > > small
> > > change this reduces the problem greatly.
> > > 
> > > I imagine there are many ways to reduce this pain I'm just not sure of 
> > > the
> > > quickest and sanest path.
> > > 
> > > If there's anyway I can help let me know
> > > 
> > > Thanks!
> > > 
> > > 
> > > 
> > > On Mar 11, 2013, at 9:50 PM, Daniel DeLeo 
> > > <
 
> > >  
> > > (mailto:
 )>
> > >  wrote:
> > > 
> > > Do conditional GETs or HEAD requests count against the limit?
> > > 
> > > --
> > > Daniel DeLeo
> > > 
> > > On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:
> > > 
> > > Hi,
> > > 
> > > We've been having fun using the lxc driver for testing with test-kitchen
> > > (which makes testing really fast!). Unfortunately I forgot that GitHub 
> > > has
> > > rate limiting imposed now. (60 requests were used up in less than 15
> > > minutes (in part because LXC is so fast at creating and destroying 
> > > guests
> > > and booting new guests ))
> > > 
> > > 
 :~/git/haproxy_lwrp#
> > >  curl https://api.github.com/rate_limit -u
> > > "damm"
> > > Enter host password for user 'damm':
> > > {
> > > "rate": {
> > > "limit": 5000,
> > > "remaining": 5000
> > > }
> > > }
> > > 
 :~/git/haproxy_lwrp#
> > >  curl https://api.github.com/rate_limit
> > > {
> > > "rate": {
> > > "limit": 60,
> > > "remaining": 60
> > > }
> > > }
> > > 
> > > I believe a few api requests occur in
> > > https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100
> > > 
> > > I don't think the right answer is 'can we ship Kitchen Busser with 
> > > Omnibus
> > > a/k/a Chef-full'.
> > > 
> > > Could we move https://raw.github.com/opscode/kb/go to opscode.com ;
> > > (http://opscode.com) like we do
> > > with the full installer? curl -L https://www.opscode.com/chef/kb.sh? ;
> > > with
> > > that we could reduce the github api requests.
> > > the basic error seen when getting this can be found
> > > https://gist.github.com/damm/5130887
> > > 
> > > P.S. I can create an issue for this, I just wanted to start a 
> > > conversation
> > > about it and get a general idea of where we wanted to steer this before
> > > hand.
> > > 
> > > Thanks!
> > 
> > !DSPAM:513f81af199541804284693! 

Personally I think it's awesome someone has bumped up against GitHub API 
throttling *because* their testing is going that fast. I was taking a look at 
this last night and there are 2 places where the git tag API calls are going 
to hurt us: 

* In the install script that installs Kitchen Busser, aka kb 
(https://github.com/opscode/kb/blob/e246b11c5761b9d681903d10d00c3c337311409e/install#L55-L63)
* In the `kb github-tags` subcommand 
(https://github.com/opscode/kb/blob/master/libexec/kb-github-tags)

If we had a way to avoid these calls or ask some other endpoint (static or 
dynamic), we'd only be asking the GitHub file servers to download the correct 
tarballs. I'd like to avoid adding too many moving parts however it feels 
like at least one more *something* is called for here (barring some 
re-implementation of kb).

Currently I'm toying with a very small web app that can answer a simple 
question: what is the latest version-tagging git tag on a github repo? This 
could make use of conditional GETs (which don't fall under throttling) and 
the responses could be cached using ETAG headers. The only real change to the 
kb code would be to call this API to get an answer and proceed to download 
off GitHub as usual. Essentially this app would mitigate the API throttling 
for everyone in one place.

So why not do conditional GETs and ETAG caching on our workstations and 
remote nodes? The problem is that kb lives on those freshly created nodes 
which have no past history to draw from. Their first call to the GitHub API 
will always be a cache miss. Worse is that in certain environments (like LXC 
containers, NAT'ed subnets with VM instances) the source IP to GitHub could 
all be the same anyway.

Anyway that is what I'd like to spend a bit more time on tonight to flesh it 
out. Longer term there could be a more clever solution, but for now let's get 
everyone back to fast, throttle-free tests :)

Thoughts?

Fletcher