[chef] Re: app deployments w/ chef

[Clif Smith < >]

Here's how we handle deployments:
  - we pre-build EC2 images with all software that will need to be installed 
(but set to not start) to reduce our dependency on 3rd parties and time to 
production

  - EC2 nodes are provisioned via auto scale and bootstrapped with a user 
data script that (among other things) updates client.rb with the appropriate 
chef server, role and environment then kicks off a chef-client run

  - the deploy recipe searches a data bag to determine the correct build to 
deploy using the role as the key

  - builds are stored in S3 so we use s3_file to compare the remote build to 
the one already deployed, if one has been

  - if there's a new build, we have 2 scenarios
    - for public facing deploys, the recipe uses a db as a lock mechanism to 
determine if any other instances with the same role are deploying, if not, it 
inserts a row as a lock and starts the deploy.  If there is an existing 
row/lock, the deploy is delayed until the next chef-client run
    - for non-public facing deploys, the recipe deploys
  - deploys update the db once it's done so we can query what build the 
instance is running.  I have a ticket to monitor deploys and warn when/if 
we're in an inconsistent state after an acceptable amount of time.

  - chef-client runs every 30 minutes

cjs

On Oct 10, 2013, at 2:25 PM, Wes Morgan 
<
 >
 wrote:

> Ohai,
> 
> I know this is a perennial bugaboo, but I'm curious how folks are handling 
> internally-developed app deployments with Chef these days, and specifically 
> if anyone has found a good solution that achieves all of the following:
> 
> 1. When provisioning a new node, Chef can deploy the app(s) that should be 
> on that node. knife ec2 server create … should be the only command I need 
> to run to go from "no server" to "it's added to the LB and handling 
> requests." In other words, you don't need to provision with Chef, then 
> deploy with Capistrano, etc.
> 
> 2. When running chef-client on nodes that already have some version of the 
> app(s) running, make sure that they all run the same version and upgrade 
> simultaneously and atomically (or as close to that as possible). So, for 
> example, if you were storing a git rev in a data bag as the "current 
> version I want deployed" and then one node kicked off its regular 
> chef-client run and upgraded to that, the other nodes running that app 
> would then still be on an old version of that code. That shouldn't happen, 
> but it would with the simplest use of the Chef deploy resource and the 
> splayed, regular interval chef-client runs.
> 
> 3. Don't lose the ability to run chef-client automatically at regular 
> intervals on all nodes. This is important. I have better things to do than 
> running Chef code manually on servers when things change (and running that 
> detection algorithm in my brain) and then cleaning up whatever bit rot 
> breakage occurred since the last chef-client run hours/days/weeks/months 
> ago.
> 
> It seems the main tension here is #1 requires Chef to be able to kick off a 
> deploy to a single node, whereas #2 requires Chef to either never deploy 
> code to nodes that already have some version of it (and thus require a 
> separate manual process to do simultaneous, atomic upgrades) OR to kick off 
> an orchestrated run across all nodes that need to be upgraded (which really 
> isn't Chef's forte, AFAICT).
> 
> Something like Capistrano is handy because multi-server orchestration is 
> its bread and butter, but getting Chef and Capistrano to communicate and 
> share data is… tricky.
> 
> If the deploy resource could be configured to only deploy when no version 
> of the app was present (covering #1), or when a defaulted-false parameter 
> were set to true by a manual knife ssh run covering all nodes that run that 
> app, that could work. Does that seem reasonable?
> 
> - Wes

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail