[chef] Re: Mind-boggling issue with Logstash

[Martin Smith < >]

Hi Fabien,

For what it's worth, we're likely to rewrite the elkstack cookbook to be a library cookbook very shortly.

Until then, did you see that you can provide the same data you're trying to provide to elkstack with examples found at:

https://github.com/rackspace-cookbooks/stack_commons/blob/master/attributes/logstash.rb

Alternatively, you could also just call the upstream logstash_config resource with your arguments (it's what elkstack does):

https://github.com/lusis/chef-logstash/blob/master/resources/config.rb

As one of the elkstack authors, I'd be glad to help you diagnose the issue further if you opened an issue on elkstack. Unfortunately, it's such a complex beast (until the library rewrite) that there's a number of things that can go wrong.

When you use chef to create your logstash config snippet, can you SSH into the server and see if the file is even being created?

Again, an issue on the Github repo is probably the best way for us to troubleshoot. Beyond that, a minimal reproducible example would also be the best way forward. Perhaps in the form of a test-kitchen suite and serverspec test?

Please also feel free to hit me up on #chef on Freenode for more realtime troubleshooting :)

- Martin B. Smith

---

From: Fabien Delpierre < >
Sent: Sunday, April 5, 2015 4:52 PM
To: chef
Subject: [chef] Mind-boggling issue with Logstash

 

Hi folks,

I'm trying to do a very basic implementation of an Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped someone had done the hard work of figuring out the mostly undocumented lusis/chef-logstash cookbooks and found the elkstack cookbook.

Ultimately, I just need Logstash to read an HAProxy log at /var/log/haproxy.log. 

I'm using the elkstack::recipe but removed the rsyslog::client include at the end because I don't need it. 

I also don't actually have HAProxy running, just a log file with a few sample entries from an actual HAProxy instance.

Here's what my recipe looks like. The kibana bits are disabled because I don't need them for troubleshooting and it just slows down the provisioning process.

include_recipe 'elkstack::_server'

include_recipe 'elkstack::elasticsearch'

include_recipe 'elkstack::logstash'

# node.run_state['elkstack_kibana_username'] = 'kibana'

# node.run_state['elkstack_kibana_password'] = 'kibana'

# include_recipe "elkstack::kibana"

instance_name = node['elkstack']['config']['logstash']['instance_name']

basedir = node['logstash']['instance'][instance_name]['basedir']

# Create a dummy HAProxy file containing a few entries

template node['cloudmine']['haproxy_log_file'] do

  source "haproxy.log.erb"

  owner "syslog"

  group "adm"

  mode 00644

  not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }

  notifies :restart, "logstash_service[#{instance_name}]", :delayed

end

At this point, there's nothing set up to tell Logstash to read my HAProxy log file. So I added this simple bit to the recipe:

cookbook_file "input_haproxy" do

  path "/opt/logstash/server/etc/conf.d/input_haproxy"

  owner "logstash"

  group "logstash"

  mode 00664

end

And this is the content of the input_haproxy file:

input {

  file {

    path => "/var/log/haproxy.log"

    start_position => "beginning"

    type => "haproxy-http"

  }

}

If this recipe runs with the cookbook_file block and I restart the Logstash service... I get nothing in Elasticsearch, like my HAProxy log file is not being read at all, ever.

However, if I run the recipe without the cookbook_file block, SSH into the instance, do a $ sudo su -l logstash, manually create the input_haproxy file in the correct location (I just copy-paste the contents from the file in my git repo) and assign it the right ownership/permissions, then restart the Logstash service, after about 30 seconds I can see that my HAProxy log was read and I can see it in Elasticsearch.

It's incredibly frustrating and I don't understand what I'm doing wrong. It's like Chef does things differently than me when I do it manually, resulting in this silent "failure". I realize Chef does do this a little bit differently, since I believe it would create the file as root and then set the correct permissions, whereas I'm doing it directly as the logstash user... But still, I don't see why this would matter since, in the end, the file is owned by the logstash user, with the rw-rw-r-- permissions I want.

Any ideas?