[chef] How do I know if my application has really been "provisioned"? a suggestion

[Bryan Berry < >]

I have been struggling lately w/ the problem of how to know whether or
not my servers have been fully provisioned. This is a problem that
affects both cookbook testing and orchestration.

I have numerous java applications that have long startup times. I
believe Rails 3 apps also have long startup times. For example,
elasticsearch takes > 5s to start up fully. However Chef does not
natively have a way to determine when elasticsearch has fully started
up. All Chef can know is the exit code returned by `service
elasticsearch start` and its various platform equivalents. Why is this
an issue? Well, I don't want to run my minitest-handler integration
tests until elasticsearch is actually functional. Also, if I am
testing a multiple VM setup where another VM depends on elasticsearch,
I need to know when elasticsearch has completed startup.

Erik Hollensbe is doing some freaking awesome work on workflow
orchestration w/ chef-workflow and I think it illustrates the problem
here

require 'chef-workflow/helper'
class MyTest < MiniTest::Unit::VagrantTestCase
  def before
     @json_msg = '{ 'id': "dumb message json msg"}'
  end
  def setup
    provision('elasticsearch')
    provision('logstash')
    wait_for('elasticsearch')
    wait_for('logstash')
    inject_logstash_message(@json_msg)
  end

  def test_message_indexed_elasticsearch
    assert es_has_message?(@json_msg)
  end
end

If I understand Erik's code correctly, the `wait_for('elasticsearch')`
only waits for the vagrant provisioner to return. The vagrant
provisioner in turn only waits for `service elasticsearch start` to
return a non-zero exit-code.

We need an optional way to determine whether an server has been
complete provisioned, or that all the resources have entered a "done"
state.  The only way I know that elasticsearch has started
successfully is if I see in the log "Elasticsearch has started" w/ a
timestamp more recent than when I started the service.

Here is my idea, pls tell me if it is crazy.

I think that the service resource should have an additional attribute
"done" that takes two ruby blocks, named before and after. The first
block would return a hash with the results of the block.

# Warning much of the Ruby code here maybe horribly incorrect.

def get_current_offset(file_name)
    { :offset => ::File.seek(file_name, EOF).get_offset }
end

def es_started?(params)
    !::File.open(params[:log_file], params[:offset],
'r').grep(params[:started_text]).empty?
end

log_file = '/usr/local/var/log/elasticsearch.log'
started_test = "elasticsearch has started successfully"
timeout = 60

service "elasticsearch" do
    done(
              :before => {  get_current_offset(logfile)  },
              :after => {  es_started?(:log_file => log_file, :offset
=> @before_results[:offset]
              :timeout => 60
             )
end

The before block would run before the service is actually actioned.
Now Chef would need some additional machinery to collect all the done
:after blocks and the related @before_results. This could be done by
chef_handler but may be better as part of chef itself. Let's call it
the done_handler for now. This done_handler would mark the time before
it starts handling any done_after blocks, then loop through the
collected done_after blocks for the specified timeout. Once all blocks
are complete it would continue onto other handlers, such as the
minitest_handler.

I think this done_handler could be part of something called "wait_mode"