[chef] Re: Re: erchef errors

[Seth Falcon < >]

Hi there,


 
 writes:
> Answering to myself ... hope will help somebody.

Sorry not to have been able to respond sooner.

> Symptoms on the chef server
>
> - higher load then usual, from 0.5 - 1 to  4 - 5
> - the beam.smp eating cpu constantly
> - all the errors that I sent before into the log files

The beam.smp for erchef pegging the CPU on 11.0.8 is a strong indication
that you're hitting a bug in the dependency solver [1].

[1] https://tickets.opscode.com/browse/CHEF-3921

> Spent some time looking at all components, read the erl files etc.
> Noticed as well exceptions for connection to the PostgreSQL from chef_wm
> however the db was doing nothing - has a max of 200 connections with a
> pool of 20 and never used it at full.
>
> The problem ... something as stupid as - somebody had written into
> /etc/hosts
> the wrong entry for chef server. After I corrected all went fine.

Chef Server needs to generate URLs to send to clients for cookbook
content upload/download. So it is possible that changes in /etc/hosts
could affect functionality there -- but unlikely to be related to high
CPU use of erchef beam process.

> ps: (/var/opt/chef-server/postgresql/data/pg_hba.conf has ONLY ip addresses
> however
> there most be something else that uses the hostname/ip from hosts
> files)

Yep, erchef treats the bookshelf service as an external
service. Cookbook upload and cookbook content download rely on erchef
being able to generate a URL to the bookshelf service (by default at
https://$SERVER_FQDN/bookshelf) that is routable by clients.

> On Tue, Nov 26, 2013 at 12:28 PM, Silviu Dicu 
> <
 >
>  wrote:
>
>> Hi,
>>
>> All the sudden my chef server version 11.4 started to act up - I get
>>
>> Postgresql
>> ----------------
>>
>> 2013-11-26_17:09:25.72986 STATEMENT:  INSERT INTO checksums(org_id,
>> checksum)
>> 2013-11-26_17:09:25.72986            VALUES ($1, $2)
>> 2013-11-26_17:09:26.51887 ERROR:  duplicate key value violates unique
>> constraint "checksums_pkey"
>> 2013-11-26_17:09:26.51891 DETAIL:  Key (org_id,
>> checksum)=(00000000000000000000000000000000,
>> 4dc4a7ba29b5ae8e48d6fe7745a5de0e) already exists.

These errors are (a bit sad to say) expected and happen under normal
operation. We use pg's foreign key constraints to do reference counting
on cookbook content (checksums). The checksummed cookbook content is
shared across cookbooks and cookbook versions. When we update a cookbook
version or delete one, the query we run attempts to insert or delete all
of the associated checksums where we expect to only succeed if the
constraints align -- side effect of this approach is errors in the
log. We have a ticket to address this using a stored procedure.

>> Erchef
>> ---------
>>
>> 2013-11-26_17:22:28.88338 webmachine error: path="/environments/
>> web_uat/cookbook_versions"
>> 2013-11-26_17:22:28.88338 {error,
>> 2013-11-26_17:22:28.88338     {error,
>> 2013-11-26_17:22:28.88339         {badrecord,chef_cookbook_version},
>> 2013-11-26_17:22:28.88339
>> [{chef_wm_depsolver,'-assemble_response/3-lc$^0/1-0-',1,
>> 2013-11-26_17:22:28.88339
>>  [{file,"src/chef_wm_depsolver.erl"},{line,266}]},
>> 2013-11-26_17:22:28.88340
>>  {chef_wm_depsolver,'-assemble_response/3-lc$^0/1-0-',1,

This one is not expected but difficult to say from this trace if there's
bad data in the db (I doubt it) and more likely that we encountered an
error fetching cookbook content.

>> Nginx
>> ---------
>>
>> ... truncated
>> "POST /environments/web_prod/cookbook_versions HTTP/1.1" 500 "201.387" 36
>> "-" "Chef Client/11.4.0

This is the call made by chef-client to resolve dependencies.

+ seth

-- 
Seth Falcon | Development Lead | Opscode | @sfalcon