[chef] Re: recurring bad perms on /var/log/chef, chef-solr returns 404

[Steffen Gebert < >]

Hi Peter,

that's a problem when you run chef-client and chef-server on the same host 
and was fixed with the latest release of the chef-client cookbook.

https://github.com/opscode-cookbooks/chef-client/commit/5c9bf4fcfb6578ffa033fc31a0b792dee55b5f38

However, as we haven't deployed our chef server through a chef recipe, the 
fix didn't help us and I removed the check for the chef-server recipe for us.

Yours
Steffen

On 10.04.2012, at 01:11, Peter Donald wrote:

> Hi,
> 
> I believe it is actually the lograte script that is causing the
> problem - I think it may be something you need to edit in
> /etc/logrotate.d/chef-solr but I don't have access to the config at
> the moment. For a long time I just ran a cron script that enforced the
> correct permissions and bounced the solr server when necessary
> 
> On Tue, Apr 10, 2012 at 5:21 AM,  
> <
 >
>  wrote:
>> 
>> hi. i thought i'd update subject in case it helps get more eyeballs
>> on this. the details are right here:
>> 
>> On Mon, 09 Apr 2012, 
>> 
 
>>  wrote:
>> 
>>> i suspect the following 2 dirs have the wrong permissions: /var/run/chef
>>> and /var/log/chef. wrong in that they are root:root owned. if you chown
>>> them to chef:chef, restart chef-solr, it should fix this.
>>> 
>>> i keep running into this problem where chef-solr dies for whatever reason,
>>> and for reasons i haven't figured out yet, these 2 dirs get root:root
>>> pwned.
>>> 
>>> anyone know why these bad perms keep getting laid down?
>>> 
>>> i'm about to cron a babysitter script to restart chef-solr and keep
>>> these dirs chowned properly :<
>>> 
>>> (yes, the chef-server recipe i run on my chef server managers proper
>>> perms on these dirs, but something else flips them back to root:root)
>>> 
>>> kallen
>>> 
>>> 
>>> On Mon, 09 Apr 2012, Paul Welch wrote:
>>> 
>>>> Output from my  /var/log/chef/solr.log:
>>>> 
>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: About to change privilege to 
>>>> chef:chef
>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: Daemonizing..
>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: Forked, in 895. Priveleges: 107 
>>>> 115
>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: Starting Solr with java
>>>> -Xmx256M -Xms256M -Dsolr.data.dir=/var/cache/chef/solr/data
>>>> -Dsolr.solr.home=/var/lib/chef/solr
>>>> -DSTART=/var/lib/chef/solr/solr-jetty/etc/start.config -jar
>>>> /var/lib/chef/solr/solr-jetty/start.jar
>>>> 1 [main] INFO org.mortbay.log - Logging to
>>>> org.slf4j.impl.SimpleLogger(org.mortbay.log) via
>>>> org.mortbay.log.Slf4jLog
>>>> 196 [main] INFO org.mortbay.log - jetty-6.1.22
>>>> SLF4J: Class path contains multiple SLF4J bindings.
>>>> SLF4J: Found binding in
>>>> [jar:file:/usr/share/java/slf4j-simple-1.5.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> SLF4J: Found binding in
>>>> [jar:file:/var/lib/chef/solr/solr-jetty/webapps/solr/WEB-INF/lib/slf4j-jdk14.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an ;
>>>> explanation.
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: No /solr/home in JNDI
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader <init>
>>>> INFO: Solr home set to '/var/lib/chef/solr/'
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.servlet.SolrDispatchFilter init
>>>> INFO: SolrDispatchFilter.init()
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: No /solr/home in JNDI
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.CoreContainer$Initializer
>>>> initialize
>>>> INFO: looking for solr.xml: /var/lib/chef/solr/solr.xml
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader <init>
>>>> INFO: Solr home set to '/var/lib/chef/solr/'
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrConfig <init>
>>>> INFO: Loaded SolrConfig: solrconfig.xml
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrCore <init>
>>>> INFO: Opening new SolrCore at /var/lib/chef/solr/,
>>>> dataDir=/var/cache/chef/solr/data/
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.schema.IndexSchema readSchema
>>>> INFO: Reading Solr Schema
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.schema.IndexSchema readSchema
>>>> INFO: Schema name=chef
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created string: org.apache.solr.schema.StrField
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created boolean: org.apache.solr.schema.BoolField
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created integer: org.apache.solr.schema.IntField
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created long: org.apache.solr.schema.LongField
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created float: org.apache.solr.schema.FloatField
>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created double: org.apache.solr.schema.DoubleField
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created sint: org.apache.solr.schema.SortableIntField
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created slong: org.apache.solr.schema.SortableLongField
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created sfloat: org.apache.solr.schema.SortableFloatField
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created sdouble: org.apache.solr.schema.SortableDoubleField
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created stuff: org.apache.solr.schema.TextField
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.util.plugin.AbstractPluginLoader 
>>>> load
>>>> INFO: created date: org.apache.solr.schema.DateField
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.schema.IndexSchema readSchema
>>>> INFO: default search field is content
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.schema.IndexSchema readSchema
>>>> INFO: query parser default operator is OR
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.schema.IndexSchema readSchema
>>>> INFO: unique key field: X_CHEF_id_CHEF_X
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.JmxMonitoredMap <init>
>>>> INFO: No JMX servers found, not exposing Solr information with JMX.
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created standard: solr.SearchHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /update: solr.XmlUpdateRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /update/javabin: solr.BinaryUpdateRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /analysis/document: solr.DocumentAnalysisRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /analysis/field: solr.FieldAnalysisRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: adding lazy requestHandler: solr.CSVRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /update/csv: solr.CSVRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /admin/ping: PingRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>> initHandlersFromConfig
>>>> INFO: created /debug/dump: solr.DumpRequestHandler
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.search.SolrIndexSearcher <init>
>>>> INFO: Opening 
>>>> 
 
>>>>  main
>>>> Apr 9, 2012 12:45:18 PM
>>>> org.apache.solr.update.DirectUpdateHandler2$CommitTracker <init>
>>>> INFO: AutoCommit: if uncommited for 10000ms; if 100 uncommited docs
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>> inform
>>>> INFO: Adding  
>>>> component:
 
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>> inform
>>>> INFO: Adding  
>>>> component:
 
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>> inform
>>>> INFO: Adding  
>>>> component:
 
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>> inform
>>>> INFO: Adding  
>>>> component:
 
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>> inform
>>>> INFO: Adding  
>>>> component:
 
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>> inform
>>>> INFO: Adding  debug
>>>> component:
 
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrCore registerSearcher
>>>> INFO: Registered new searcher 
>>>> 
 
>>>>  main
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.CoreContainer register
>>>> INFO: registering core:
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrDispatchFilter init
>>>> INFO: user.dir=/var/lib/chef/solr/solr-jetty
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrDispatchFilter init
>>>> INFO: SolrDispatchFilter.init() done
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrServlet init
>>>> INFO: SolrServlet.init()
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: No /solr/home in JNDI
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrServlet init
>>>> INFO: SolrServlet.init() done
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: No /solr/home in JNDI
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>> locateSolrHome
>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrUpdateServlet init
>>>> INFO: SolrUpdateServlet.init() done
>>>> 13144 [main] WARN org.mortbay.log - failed
>>>> 
 :
>>>>  java.io.IOException: Cannot
>>>> write log directory /var/log/chef
>>>> 13144 [main] WARN org.mortbay.log - failed 
>>>> 
 :
>>>> java.io.IOException: Cannot write log directory /var/log/chef
>>>> 13144 [main] WARN org.mortbay.log - failed 
>>>> 
 :
>>>> java.io.IOException: Cannot write log directory /var/log/chef
>>>> 13144 [main] WARN org.mortbay.log - Error starting handlers
>>>> java.io.IOException: Cannot write log directory /var/log/chef
>>>>     at 
>>>> org.mortbay.util.RolloverFileOutputStream.setFile(RolloverFileOutputStream.java:217)
>>>>     at 
>>>> org.mortbay.util.RolloverFileOutputStream.<init>(RolloverFileOutputStream.java:164)
>>>>     at org.mortbay.jetty.NCSARequestLog.doStart(NCSARequestLog.java:428)
>>>>     at 
>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>     at 
>>>> org.mortbay.jetty.handler.RequestLogHandler.doStart(RequestLogHandler.java:117)
>>>>     at 
>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>     at 
>>>> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
>>>>     at 
>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>     at 
>>>> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
>>>>     at org.mortbay.jetty.Server.doStart(Server.java:224)
>>>>     at 
>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>     at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>     at 
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>     at 
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>     at java.lang.reflect.Method.invoke(Method.java:616)
>>>>     at org.mortbay.start.Main.invokeMain(Main.java:194)
>>>>     at org.mortbay.start.Main.start(Main.java:534)
>>>>     at org.mortbay.start.Main.start(Main.java:441)
>>>>     at org.mortbay.start.Main.main(Main.java:119)
>>>> 13164 [main] INFO org.mortbay.log - Started 
>>>> 
 :8983
>>>> 
>>>> On Mon, Apr 9, 2012 at 2:04 PM, Bryan McLellan 
>>>> <
 >
>>>>  wrote:
>>>>> On Mon, Apr 9, 2012 at 12:36 PM, Paul Welch 
>>>>> <
 >
>>>>>  wrote:
>>>>>> If im not mistaken it looks like an issue talking to Solr. However, I
>>>>>> am not sure why it would work on initial install but stop working
>>>>>> later...
>>>>> 
>>>>> Anything interesting in the solr log? Usually /var/log/chef/solr.log.
>>>>> 
>>>>> Bryan
> 
> 
> 
> -- 
> Cheers,
> 
> Peter Donald