[chef] Re: Re: Mind-boggling issue with Logstash

[Fabien Delpierre < >]

Hi Martin,

Thanks for your answer. I wasn't expecting a reply from someone who actually maintains the elkstack cookbook!

I frankly do not think your cookbook is at fault here, it does everything I ask from it -- which isn't much -- just fine, it's the customization part I need that does not happen as expected, and that's outside of the scope of what your cookbook does for me in the current state of my recipe.

Yes, when I use cookbook_file, the input_haproxy file absolutely gets created. Whether I do it through the recipe or manually, the file ends up there with the same permissions, ownership and contents, which is why it's so frustrating that it doesn't work through the recipe. 

I had a faint glimmer of hope when you mentioned the custom_logstash attributes stack that's actually handled by your stack_commons cookbook -- I never figured out how all those input templates were being dropped into Logstash's /etc/conf.d directory, so knowing that allows me to overwrite them and only insert the one I need, which I just did. You're right that the elkstack cookbook is rather complex, but I'm not one to judge, I'm a beginner and actually reading your cookbook taught me a few things I never knew were possible. The cookbook is coded with a rather higher level of skill and complexity than I typically see in the Chef space.

Anyway, while overwriting the default['elkstack']['config']['custom_logstash']['name'] attribute to include just my HAProxy input config helped me remove all the other default inputs that were being set in Logstash, unfortunately it didn't change the issue I'm having. It's maddening! :(

I'll try opening an issue on GH to troubleshoot, but I'm running out of time, which of course has nothing to do with you.

On Sun, Apr 5, 2015 at 6:12 PM, Martin Smith < " target="_blank"> > wrote:

Hi Fabien,

For what it's worth, we're likely to rewrite the elkstack cookbook to be a library cookbook very shortly.

Until then, did you see that you can provide the same data you're trying to provide to elkstack with examples found at:

https://github.com/rackspace-cookbooks/stack_commons/blob/master/attributes/logstash.rb

Alternatively, you could also just call the upstream logstash_config resource with your arguments (it's what elkstack does):

https://github.com/lusis/chef-logstash/blob/master/resources/config.rb

As one of the elkstack authors, I'd be glad to help you diagnose the issue further if you opened an issue on elkstack. Unfortunately, it's such a complex beast (until the library rewrite) that there's a number of things that can go wrong.

When you use chef to create your logstash config snippet, can you SSH into the server and see if the file is even being created?

Again, an issue on the Github repo is probably the best way for us to troubleshoot. Beyond that, a minimal reproducible example would also be the best way forward. Perhaps in the form of a test-kitchen suite and serverspec test?

Please also feel free to hit me up on #chef on Freenode for more realtime troubleshooting :)

- Martin B. Smith

---

From: Fabien Delpierre < " target="_blank"> >
Sent: Sunday, April 5, 2015 4:52 PM
To: chef
Subject: [chef] Mind-boggling issue with Logstash

 

Hi folks,

I'm trying to do a very basic implementation of an Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped someone had done the hard work of figuring out the mostly undocumented lusis/chef-logstash cookbooks and found the elkstack cookbook.

Ultimately, I just need Logstash to read an HAProxy log at /var/log/haproxy.log. 

I'm using the elkstack::recipe but removed the rsyslog::client include at the end because I don't need it. 

I also don't actually have HAProxy running, just a log file with a few sample entries from an actual HAProxy instance.

Here's what my recipe looks like. The kibana bits are disabled because I don't need them for troubleshooting and it just slows down the provisioning process.

include_recipe 'elkstack::_server'

include_recipe 'elkstack::elasticsearch'

include_recipe 'elkstack::logstash'

# node.run_state['elkstack_kibana_username'] = 'kibana'

# node.run_state['elkstack_kibana_password'] = 'kibana'

# include_recipe "elkstack::kibana"

instance_name = node['elkstack']['config']['logstash']['instance_name']

basedir = node['logstash']['instance'][instance_name]['basedir']

# Create a dummy HAProxy file containing a few entries

template node['cloudmine']['haproxy_log_file'] do

  source "haproxy.log.erb"

  owner "syslog"

  group "adm"

  mode 00644

  not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }

  notifies :restart, "logstash_service[#{instance_name}]", :delayed

end

At this point, there's nothing set up to tell Logstash to read my HAProxy log file. So I added this simple bit to the recipe:

cookbook_file "input_haproxy" do

  path "/opt/logstash/server/etc/conf.d/input_haproxy"

  owner "logstash"

  group "logstash"

  mode 00664

end

And this is the content of the input_haproxy file:

input {

  file {

    path => "/var/log/haproxy.log"

    start_position => "beginning"

    type => "haproxy-http"

  }

}

If this recipe runs with the cookbook_file block and I restart the Logstash service... I get nothing in Elasticsearch, like my HAProxy log file is not being read at all, ever.

However, if I run the recipe without the cookbook_file block, SSH into the instance, do a $ sudo su -l logstash, manually create the input_haproxy file in the correct location (I just copy-paste the contents from the file in my git repo) and assign it the right ownership/permissions, then restart the Logstash service, after about 30 seconds I can see that my HAProxy log was read and I can see it in Elasticsearch.

It's incredibly frustrating and I don't understand what I'm doing wrong. It's like Chef does things differently than me when I do it manually, resulting in this silent "failure". I realize Chef does do this a little bit differently, since I believe it would create the file as root and then set the correct permissions, whereas I'm doing it directly as the logstash user... But still, I don't see why this would matter since, in the end, the file is owned by the logstash user, with the rw-rw-r-- permissions I want.

Any ideas?