[chef] Re: Re: Re: application builds/ci involving Chef

[Peter Donald < >]

Hi,

Our newer applications are all JEEish and thus have relatively low variability. We developed a "configuration template" in json for each application. As most of our cookbooks use an attribute driven design (See [1]) we actually could just merge parts of the json template into the node data. We just needed to make sure that the data conforms with the cookbooks attribute_driven schema.  ie We could generate a json blob something like

json =
{
  'config' =>
  {
    'max_memory' => '${CM:MAX_MEMORY|1548}',

    'max_perm_size' => 192,

    'port' => 80,
    'admin_port' => 8085,
    'max_stack_size' => 128,
    'username' => '${CM:ADMIN_USERNAME}',

    'password' => '${CM:ADMIN_PASS}'


  },
  'deployables' =>
  {
    'somapp' =>
    {
      'url' => 'http://repo.example.com/somapp-${CM:VERSION}.war',
      'context_root' => '/somapp'
    }
  },
  'extra_libraries' =>
  {
    'mydatabasedriver' =>
       'http://repo.example.com/mydatabasedriver-1.2.3.jar'
  },
  'jdbc_connection_pools' =>
    {
      'SomeappSQL' =>
      {
        'datasourceclassname' => 'net.sourceforge.jtds.jdbcx.JtdsDataSource',
        'restype' => 'javax.sql.DataSource',
        'isconnectvalidatereq' => 'true',
        'validationmethod' => 'auto-commit',
        'ping' => 'true',
        'description' => 'SomeappSQL Connection Pool',
        'properties' =>
        {
           'Instance' => 'Instance1',
           'ServerName' => 'db.example.com',
           'User' => '${CM:DB_WRITER_USER}',

           'Password' => '${CM:DB_WRITER_PASSWORD}'


           'PortNumber' => '1433',
           'DatabaseName' => 'SOMEAPP'
        },
        'resources' =>
        {
          'jdbc/SomeappDS' =>
            {'description' => 'SomeappSQL Connection Resource'}
        }
    },
    'custom_resources' =>
    {
      'MyServiceURL' => 'http://${CM:SOA_HOSTNAME}/MyService'

    }
}.to_json

You will notice that there is variables that need to be replaced as part of the template expansion and this is done via various mechanisms. Sometimes these values are extracted from data bags, sometimes derived etc.

These templates also declare other resources that the application either needs or provides. i.e. It defines the set of queues the queue server is meant to manage, the set of checks that our monitoring infrastructure should run etc. These are all deep merged into the required nodes as part of the release.

The application developers are responsible for writing the template, while the operations are responsible for validating and deploying. The developers are responsible for deploying their binaries and configuration templates into the same location (aka nexus) and chef picks it up from there. Locally devs run using vagrant, and then the changes are rolled out through different environments.

So what I wrote above is partially a lie. The above describes the model we are moving towards.

 At the moment there is no clear distinction between dev and ops so everyone has to modify chef artefacts to do a release. Currently our configuration template is not in nexus but is instead namespaced entries in a data bag that needs to be uploaded as part of a release. Currently some applications still need custom chef scripts, if they do exceptional things (like have a data directory, set up in a particular way in a particular location on host). We have also not yet figured out any reasonable approach to pairing the version of the configuration template with the version of the associated binary artefacts. We also have limited value interpolation.

However, said that we like what we have got now and plan to move further down this path in the future.

[1] http://realityforge.org/code/2012/11/12/reusable-cookbooks-revisited.html

--
Cheers,

Peter Donald