[chef] Dependency solver overloaded issue

[Michael Hart < >]

I’m seeing occasional errors on chef-client runs which cause the runs to fail. I originally thought it was a resource issue on the chef server (open source chef server v11.1.1 on Ubuntu LTS) and resized the server, also set erchef['db_pool_size'] = 40 and postgresql['max_connections'] = 400. However that doesn’t resolve it. 

I suspect a hint in the erchef error below "msg=no_depsolver_workers”, but I don’t know how to further debug this one. I’d appreciate any hints/suggestions. Logs follow:

The error on the chef client run: (Note: hostnames have been changed to protect the innocent to a.b.com)

[2014-09-25T13:14:43+00:00] INFO: Starting Chef Run for a.b.com

[2014-09-25T13:14:43+00:00] INFO: Running start handlers

[2014-09-25T13:14:43+00:00] INFO: Start handlers complete.

[2014-09-25T13:14:43+00:00] INFO: HTTP Request Returned 404 Object Not Found: 

[2014-09-25T13:14:44+00:00] INFO: HTTP Request Returned 503 Service Unavailable: Dependency solver overloaded. Try again later.

================================================================================

Error Resolving Cookbooks for Run List:

================================================================================

Server Unavailable

------------------

The Chef Server is temporarily unavailable

Server Response:

----------------

Dependency solver overloaded. Try again later.

I can’t find much in the server logs except the following:

nginx/access.log:

10.10.10.100 - - [25/Sep/2014:13:14:44 +0000]  "POST /environments/production-0_5814/cookbook_versions HTTP/1.1" 503 "0.774" 60 "-" "Chef Client/11.12.8 (ruby-1.9.3-p484; ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:8000" "503" "0.733" "11.12.8" "algorithm=sha1;version=1.0;” “a.b.com" "2014-09-25T13:14:43Z" "O66i4b7dFTSNpuu9trSDnzWDklE=“ 1544

erchef/current

2014-09-25_13:14:44.64379 [error] {<<"method=POST; path=/environments/production-0_5814/cookbook_versions; status=503; ">>,"Service Unavailable"}

erchef/requests.log.4

2014-09-25T13:14:44Z "> method=POST; path=/environments/production-0_5814/cookbook_versions; status=503; user=a.b.com; req_id=tRvX5i45OFCH9a92qmPKGg==; msg=no_depsolver_workers; req_time=729; rdbms_time=709; rdbms_count=3; depsolver_time=0; depsolver_count=1;

thanks!

mike

--

Michael Hart

Arctic Wolf Networks

M: 226-388-4773