[chef] intermittent problems with default attributes

[Jonathan Dye < >]

something i'm doing with attributes is not quite right, and here's the proof:

opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null 1>/dev/null; if [[ "$?" != "0" ]]; then echo -n 'E'; elif grep 42559 /opt/moab/etc/moab.cfg >/dev/null && grep 42559 /opt/mws/etc/mws-config.groovy >/dev/null; then echo -n '.'; else echo -n '!'; fi; done

!!!!!!!.!!!...!!!!!..!.......!!!!!!.!!!!!..!!!!^CE^C

so, i'm running chef over and over without changing cookbooks, run lists, override attributes, or anything else in between.  if the run fails it should print 'E', if the expected attributes are in the config files it's writing out it should print '.' , and if the attributes are not in the file it should print '!'.  as you can see, the results are not the same in subsequent runs.  this is not because the chef runs are failing-- the value simply ends up being nil on some runs all else seemingly being equal.  i've confirmed this manually many times, it can succeed the client run but produce unexpected output.

here's what i'm doing, and it seems pretty straightforward.  there is a default attributes file that sets the 'sched_port' as a default attribute, and it is consumed by a template in that cookbook where it's set (mwm) and in a template from another cookbook which is also run on the same node (mws):

=====  mwm/attributes/default.rb =====

::Chef::Node.send(:include, Opscode::OpenSSL::Password)

set_unless['mcs']['mwm']['secret_key'] = secure_password

node.save

default['mcs']['mwm']['sched_port'] = '42559'

=======

====== mwm/recipes/default.rb =======

...

# write out the templated moab config

template "/opt/moab/etc/moab.cfg" do

        source "moab-cfg.erb"

        mode 0600

        owner "root"

        group "root"

        action :create

end

...

==========

===== mwm/templates/default/moab-cfg.erb =====

...

SCHEDCFG[Moab]                      SERVER=<%= node['hostname'] %>:<%= node['mcs']['mwm']['sched_port'] %>

...

=======

===== mws/recipes/default.rb =======

...

# write mws config file

template "/opt/mws/etc/mws-config.groovy" do

        source "mws-config-groovy.erb"

        mode 0600

        owner "tomcat"

        group "tomcat"

        variables(

                :moab_secretkey => node['mcs']['mwm']['secret_key'],

                :moab_server => node['hostname'],

                :moab_port => node['mcs']['mwm']['sched_port'],

                :mws_username => node['mcs']['mws']['username'],

                :mws_password => node['mcs']['mws']['password'],

                :mongo_host => mongo_node['hostname'],

                :mongo_port => mongo_node['mongodb']['port'],

                :mongo_database => node['mcs']['mws']['mongo_dbname']

        )

        action :create

end

=======

====== mws/templates/default/mws-config-groovy.erb ====

moab.secretKey = "<%= @moab_secretkey %>"

moab.server = "<%= @moab_server %>"

moab.port = <%= @moab_port %>

...

======

also, it disappears from either file.  every time i've checked it disappears from both files at once:

opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null 1>/dev/null; if [[ "$?" != "0" ]]; then echo -n 'E'; elif grep 42559 /opt/moab/etc/moab.cfg >/dev/null; then echo -n '.'; else echo -n '!'; fi; done

.!!!!!!..!!!.!^CE^C

opt]# while [[ 1 ]]; do chef-client -E chefenv1 2>/dev/null 1>/dev/null; if [[ "$?" != "0" ]]; then echo -n 'E'; elif grep 42559 /opt/mws/etc/mws-config.groovy >/dev/null; then echo -n '.'; else echo -n '!'; fi; done

!!.!!!!!!!!.!!...!!!..!.......!!!!

i've wrestled through several other attribute-related issues (one i detailed here http://www.devopstonoops.com/2012/07/04/what-ive-learned-from-using-chef-part-1/), but i'm baffled about this one.  any help would be appreciated.

- jonathan