[chef] Re: Re: recurring bad perms on /var/log/chef, chef-solr returns 404

[Paul Welch < >]

Steffen,

Thank you for posting this. This seems to have been the issue on my system.

On Wed, Apr 11, 2012 at 12:50 PM, Steffen Gebert 
<
 >
 wrote:
> Hi Peter,
>
> that's a problem when you run chef-client and chef-server on the same host 
> and was fixed with the latest release of the chef-client cookbook.
>
> https://github.com/opscode-cookbooks/chef-client/commit/5c9bf4fcfb6578ffa033fc31a0b792dee55b5f38
>
> However, as we haven't deployed our chef server through a chef recipe, the 
> fix didn't help us and I removed the check for the chef-server recipe for 
> us.
>
> Yours
> Steffen
>
> On 10.04.2012, at 01:11, Peter Donald wrote:
>
>> Hi,
>>
>> I believe it is actually the lograte script that is causing the
>> problem - I think it may be something you need to edit in
>> /etc/logrotate.d/chef-solr but I don't have access to the config at
>> the moment. For a long time I just ran a cron script that enforced the
>> correct permissions and bounced the solr server when necessary
>>
>> On Tue, Apr 10, 2012 at 5:21 AM,  
>> <
 >
>>  wrote:
>>>
>>> hi. i thought i'd update subject in case it helps get more eyeballs
>>> on this. the details are right here:
>>>
>>> On Mon, 09 Apr 2012, 
>>> 
 
>>>  wrote:
>>>
>>>> i suspect the following 2 dirs have the wrong permissions: /var/run/chef
>>>> and /var/log/chef. wrong in that they are root:root owned. if you chown
>>>> them to chef:chef, restart chef-solr, it should fix this.
>>>>
>>>> i keep running into this problem where chef-solr dies for whatever 
>>>> reason,
>>>> and for reasons i haven't figured out yet, these 2 dirs get root:root
>>>> pwned.
>>>>
>>>> anyone know why these bad perms keep getting laid down?
>>>>
>>>> i'm about to cron a babysitter script to restart chef-solr and keep
>>>> these dirs chowned properly :<
>>>>
>>>> (yes, the chef-server recipe i run on my chef server managers proper
>>>> perms on these dirs, but something else flips them back to root:root)
>>>>
>>>> kallen
>>>>
>>>>
>>>> On Mon, 09 Apr 2012, Paul Welch wrote:
>>>>
>>>>> Output from my  /var/log/chef/solr.log:
>>>>>
>>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: About to change privilege to 
>>>>> chef:chef
>>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: Daemonizing..
>>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: Forked, in 895. Priveleges: 107 
>>>>> 115
>>>>> [Mon, 09 Apr 2012 12:45:04 -0400] INFO: Starting Solr with java
>>>>> -Xmx256M -Xms256M -Dsolr.data.dir=/var/cache/chef/solr/data
>>>>> -Dsolr.solr.home=/var/lib/chef/solr
>>>>> -DSTART=/var/lib/chef/solr/solr-jetty/etc/start.config -jar
>>>>> /var/lib/chef/solr/solr-jetty/start.jar
>>>>> 1 [main] INFO org.mortbay.log - Logging to
>>>>> org.slf4j.impl.SimpleLogger(org.mortbay.log) via
>>>>> org.mortbay.log.Slf4jLog
>>>>> 196 [main] INFO org.mortbay.log - jetty-6.1.22
>>>>> SLF4J: Class path contains multiple SLF4J bindings.
>>>>> SLF4J: Found binding in
>>>>> [jar:file:/usr/share/java/slf4j-simple-1.5.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>> SLF4J: Found binding in
>>>>> [jar:file:/var/lib/chef/solr/solr-jetty/webapps/solr/WEB-INF/lib/slf4j-jdk14.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an ;
>>>>> explanation.
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: No /solr/home in JNDI
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader <init>
>>>>> INFO: Solr home set to '/var/lib/chef/solr/'
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.servlet.SolrDispatchFilter init
>>>>> INFO: SolrDispatchFilter.init()
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: No /solr/home in JNDI
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.CoreContainer$Initializer
>>>>> initialize
>>>>> INFO: looking for solr.xml: /var/lib/chef/solr/solr.xml
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrResourceLoader <init>
>>>>> INFO: Solr home set to '/var/lib/chef/solr/'
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrConfig <init>
>>>>> INFO: Loaded SolrConfig: solrconfig.xml
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.core.SolrCore <init>
>>>>> INFO: Opening new SolrCore at /var/lib/chef/solr/,
>>>>> dataDir=/var/cache/chef/solr/data/
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.schema.IndexSchema readSchema
>>>>> INFO: Reading Solr Schema
>>>>> Apr 9, 2012 12:45:17 PM org.apache.solr.schema.IndexSchema readSchema
>>>>> INFO: Schema name=chef
>>>>> Apr 9, 2012 12:45:17 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created string: org.apache.solr.schema.StrField
>>>>> Apr 9, 2012 12:45:17 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created boolean: org.apache.solr.schema.BoolField
>>>>> Apr 9, 2012 12:45:17 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created integer: org.apache.solr.schema.IntField
>>>>> Apr 9, 2012 12:45:17 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created long: org.apache.solr.schema.LongField
>>>>> Apr 9, 2012 12:45:17 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created float: org.apache.solr.schema.FloatField
>>>>> Apr 9, 2012 12:45:17 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created double: org.apache.solr.schema.DoubleField
>>>>> Apr 9, 2012 12:45:18 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created sint: org.apache.solr.schema.SortableIntField
>>>>> Apr 9, 2012 12:45:18 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created slong: org.apache.solr.schema.SortableLongField
>>>>> Apr 9, 2012 12:45:18 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created sfloat: org.apache.solr.schema.SortableFloatField
>>>>> Apr 9, 2012 12:45:18 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created sdouble: org.apache.solr.schema.SortableDoubleField
>>>>> Apr 9, 2012 12:45:18 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created stuff: org.apache.solr.schema.TextField
>>>>> Apr 9, 2012 12:45:18 PM 
>>>>> org.apache.solr.util.plugin.AbstractPluginLoader load
>>>>> INFO: created date: org.apache.solr.schema.DateField
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.schema.IndexSchema readSchema
>>>>> INFO: default search field is content
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.schema.IndexSchema readSchema
>>>>> INFO: query parser default operator is OR
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.schema.IndexSchema readSchema
>>>>> INFO: unique key field: X_CHEF_id_CHEF_X
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.JmxMonitoredMap <init>
>>>>> INFO: No JMX servers found, not exposing Solr information with JMX.
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created standard: solr.SearchHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /update: solr.XmlUpdateRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /update/javabin: solr.BinaryUpdateRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /analysis/document: solr.DocumentAnalysisRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /analysis/field: solr.FieldAnalysisRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: adding lazy requestHandler: solr.CSVRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /update/csv: solr.CSVRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /admin/ping: PingRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.RequestHandlers
>>>>> initHandlersFromConfig
>>>>> INFO: created /debug/dump: solr.DumpRequestHandler
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.search.SolrIndexSearcher <init>
>>>>> INFO: Opening 
>>>>> 
 
>>>>>  main
>>>>> Apr 9, 2012 12:45:18 PM
>>>>> org.apache.solr.update.DirectUpdateHandler2$CommitTracker <init>
>>>>> INFO: AutoCommit: if uncommited for 10000ms; if 100 uncommited docs
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>>> inform
>>>>> INFO: Adding  
>>>>> component:
 
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>>> inform
>>>>> INFO: Adding  
>>>>> component:
 
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>>> inform
>>>>> INFO: Adding  
>>>>> component:
 
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>>> inform
>>>>> INFO: Adding  
>>>>> component:
 
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>>> inform
>>>>> INFO: Adding  
>>>>> component:
 
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.handler.component.SearchHandler 
>>>>> inform
>>>>> INFO: Adding  debug
>>>>> component:
 
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrCore registerSearcher
>>>>> INFO: Registered new searcher 
>>>>> 
 
>>>>>  main
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.CoreContainer register
>>>>> INFO: registering core:
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrDispatchFilter init
>>>>> INFO: user.dir=/var/lib/chef/solr/solr-jetty
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrDispatchFilter init
>>>>> INFO: SolrDispatchFilter.init() done
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrServlet init
>>>>> INFO: SolrServlet.init()
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: No /solr/home in JNDI
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrServlet init
>>>>> INFO: SolrServlet.init() done
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: No /solr/home in JNDI
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.core.SolrResourceLoader 
>>>>> locateSolrHome
>>>>> INFO: using system property solr.solr.home: /var/lib/chef/solr
>>>>> Apr 9, 2012 12:45:18 PM org.apache.solr.servlet.SolrUpdateServlet init
>>>>> INFO: SolrUpdateServlet.init() done
>>>>> 13144 [main] WARN org.mortbay.log - failed
>>>>> 
 :
>>>>>  java.io.IOException: Cannot
>>>>> write log directory /var/log/chef
>>>>> 13144 [main] WARN org.mortbay.log - failed 
>>>>> 
 :
>>>>> java.io.IOException: Cannot write log directory /var/log/chef
>>>>> 13144 [main] WARN org.mortbay.log - failed 
>>>>> 
 :
>>>>> java.io.IOException: Cannot write log directory /var/log/chef
>>>>> 13144 [main] WARN org.mortbay.log - Error starting handlers
>>>>> java.io.IOException: Cannot write log directory /var/log/chef
>>>>>     at 
>>>>> org.mortbay.util.RolloverFileOutputStream.setFile(RolloverFileOutputStream.java:217)
>>>>>     at 
>>>>> org.mortbay.util.RolloverFileOutputStream.<init>(RolloverFileOutputStream.java:164)
>>>>>     at org.mortbay.jetty.NCSARequestLog.doStart(NCSARequestLog.java:428)
>>>>>     at 
>>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>>     at 
>>>>> org.mortbay.jetty.handler.RequestLogHandler.doStart(RequestLogHandler.java:117)
>>>>>     at 
>>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>>     at 
>>>>> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
>>>>>     at 
>>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>>     at 
>>>>> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
>>>>>     at org.mortbay.jetty.Server.doStart(Server.java:224)
>>>>>     at 
>>>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>>>>>     at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>     at 
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>     at 
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>     at java.lang.reflect.Method.invoke(Method.java:616)
>>>>>     at org.mortbay.start.Main.invokeMain(Main.java:194)
>>>>>     at org.mortbay.start.Main.start(Main.java:534)
>>>>>     at org.mortbay.start.Main.start(Main.java:441)
>>>>>     at org.mortbay.start.Main.main(Main.java:119)
>>>>> 13164 [main] INFO org.mortbay.log - Started 
>>>>> 
 :8983
>>>>>
>>>>> On Mon, Apr 9, 2012 at 2:04 PM, Bryan McLellan 
>>>>> <
 >
>>>>>  wrote:
>>>>>> On Mon, Apr 9, 2012 at 12:36 PM, Paul Welch 
>>>>>> <
 >
>>>>>>  wrote:
>>>>>>> If im not mistaken it looks like an issue talking to Solr. However, I
>>>>>>> am not sure why it would work on initial install but stop working
>>>>>>> later...
>>>>>>
>>>>>> Anything interesting in the solr log? Usually /var/log/chef/solr.log.
>>>>>>
>>>>>> Bryan
>>
>>
>>
>> --
>> Cheers,
>>
>> Peter Donald
>



-- 
Paul Welch

 
+1.352.327.8087