[chef] Re: Re: Re: Re: Re: RabbitMQ and Logstash

[Harlan Barnes < >]

Messed up and only sent to James the first time.

On Tue, Jan 10, 2012 at 3:01 PM, James < "> > wrote:

We're currently working on a similar project. If you want to contribute your changes back / post them somewhere, we're happy to build on it and do the same. 

 

Hi James - Sure, here's where I am. (I've been sidetracked with another 
project, but I've solved what I think are hard problems. I've tested 
everything individually but not all together.)

My plan was two parts A) to have logstash act as an agent/shipper on of 
our nodes producing logs with a simple input from the local file to the 
rabbitmq cluster and B) have X number of logstash instances listening to
 the same queue in hopes of spreading the consuming out across multiple 
nodes. The logstash instances in B would do heavy filtering (grok'ing 
each event type to parse out interesting information to index in 
elasticsearch).

The log agent side ("A" above) is pretty easy. I modified Joshua Timerbman's cookbook to:

-
 Changed the agent startup scripts so that it will use Jordan's 
"directory" functionality instead of the explicit script. (The plan was 
to have other recipes dump their configurations into the directory as 
needed and let the agent glob them up.)
- To not install the agent.conf unless you tell it to (so it doesn't interfere with my own agent configuration above)
-
 I also changed the web startup scripts so that it didn't enforce 
'localhost' as the elasticsearch backend (it's an attribute now).
- Added Centos (init.d+daemonize) service support
- Here's the pull request I sent Joshua (and of course, it references back my version of it): https://github.com/jtimberman/logstash-cookbook/pull/1

On the server side ("B" above), my goal was have each one of the 
main components "clustered" (or at least stateless so they could be load
 balanced as is the case with the logstash web interface). I also wanted
 to be able to run a node having each "component" installed ... or run 
each component on a node by itself. Here's what I ended up with:

- Elasticsearch (clustered)
  - The original one I started with is this: http://community.opscode.com/cookbooks/elasticsearch
 It was only for ubuntu and debian. This is my first attempt at 
modifying a recipe so I ripped out all the stuff I didn't understand 
(the debian monit and ec2 stuff) so I could understand it easier. Then I
 added the RedHat / Centos back in by virtue of using the 
elasticsearch-wrapper here: https://github.com/elasticsearch/elasticsearch-servicewrapper (with a little modification to the wrapper file itself.)

- What I ended up with was this: https://github.com/harlanbarnes/elasticsearch-cookbook
-
 There are other Elasticsearch recipes on github. I don't think everyone
 has consolidated behind one maintainer. There's probably one that fits 
you between all the choices.

- RabbitMQ (clustered)
  - This cookbook from Opscode is pretty great https://github.com/opscode/cookbooks/tree/master/rabbitmq ... but I wanted it to auto cluster and support ssl connections. So I added that with this pull request: https://github.com/opscode/cookbooks/pull/280

  - One side note, to do clustering in RabbitMQ, you have to tell the 
recipe "up-front" which are your disk nodes (even if not all the disk 
nodes exist yet). As such, those disk nodes need to be resolvable by 
everyone in the cluster. In a traditional DNS setup, that's usually not a
 problem. On EC2 (and not using Route 53) I had to make a little recipe 
that made the "chef node name = the ephemeral private IP address".

- Logstash (consuming)
 - There's not much additional to do here above what I did for the log agent / shipper.
 -
 I did create a grok package to deploy the C grok commands that the 
version of logstash I was using relied on. However, I understand that 
the latest version of logstash has a pure ruby version of grok. So no 
need for that anymore.

There's more work to do (probably in a cookbook that implements the other cookbooks):
- Setup the users and queues on rabbitmq (There's a LWRP in the rabbitmq cookbook for that.)
- Write the logstash configuration into files or templates to drop in the configuration directory.

So that's it. I'm still relatively new to chef, so I'm sure there's lots of "wrong stuff." :-)