Hi folks,
I'm trying to do a very basic implementation of an Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped someone had done the hard work of figuring out the mostly undocumented
lusis/chef-logstash cookbooks and found the
elkstack cookbook.
Ultimately, I just need Logstash to read an HAProxy log at /var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include at the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because I don't need them for troubleshooting and it just slows down the provisioning process.
include_recipe 'elkstack::_server'
include_recipe 'elkstack::elasticsearch'
include_recipe 'elkstack::logstash'
# node.run_state['elkstack_kibana_username'] = 'kibana'
# node.run_state['elkstack_kibana_password'] = 'kibana'
# include_recipe "elkstack::kibana"
instance_name = node['elkstack']['config']['logstash']['instance_name']
basedir = node['logstash']['instance'][instance_name]['basedir']
# Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my HAProxy log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the Logstash service... I get nothing in Elasticsearch, like my HAProxy log file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into the instance, do a $ sudo su -l logstash, manually create the input_haproxy file in the correct location (I just copy-paste the contents from the file in my git repo) and assign it the right ownership/permissions, then restart the Logstash service, after about 30 seconds I can see that my HAProxy log was read and I can see it in Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing wrong. It's like Chef does things differently than me when I do it manually, resulting in this silent "failure". I realize Chef does do this a little bit differently, since I believe it would create the file as root and then set the correct permissions, whereas I'm doing it directly as the logstash user... But still, I don't see why this would matter since, in the end, the file is owned by the logstash user, with the rw-rw-r-- permissions I want.
Any ideas?