[chef] RE: Re: Chef server update went wrong - how to repair?


Chronological Thread 
  • From: Kevin Keane Subscription < >
  • To: < >
  • Subject: [chef] RE: Re: Chef server update went wrong - how to repair?
  • Date: Sun, 28 Jun 2015 13:00:38 -0700

Title: RE: [chef] Re: Chef server update went wrong - how to repair?

It does not seem to give me that option. If I tell it not to upgrade, it simply aborts. And if I tell it to upgrade, it seems to go right into the upgrade from chef 11 to chef 12 (even though chef11 does not even exist on that server any more, as far as I can tell)

# chef-server-ctl upgrade
Upgrading with options #<OpenStruct skip_confirmation=false, chef11_server_url="https://localhost", chef12_server_url="https://localhost", upload_threads=10, chef11_admin_client_name="admin", chef11_admin_client_key="/etc/chef-server/admin.pem">
Open source Chef 11 server detected.
Would you like to upgrade? [yN]
n
Aborting upgrade.


# chef-server-ctl upgrade
Upgrading with options #<OpenStruct skip_confirmation=false, chef11_server_url="https://localhost", chef12_server_url="https://localhost", upload_threads=10, chef11_admin_client_name="admin", chef11_admin_client_key="/etc/chef-server/admin.pem">
Open source Chef 11 server detected.
Would you like to upgrade? [yN]
y
Upgrading the open source Chef 11 server.
Chef 12 short organization name? <redacted>
Chef 12 full organization name? <redacted>
Creating /tmp/chef11-server-data20150628-5878-13zhtja as the location to save the open source Chef 11 server data
Ensuring Chef 12 server is stopped
ok: down: bookshelf: 0s, normally up
ok: down: nginx: 0s, normally up
ok: down: oc_bifrost: 1s, normally up
ok: down: oc_id: 0s, normally up
ok: down: opscode-chef-mover: 0s, normally up
ok: down: opscode-erchef: 1s, normally up
ok: down: opscode-expander: 0s, normally up
ok: down: opscode-expander-reindexer: 0s, normally up
ok: down: opscode-solr4: 0s, normally up
ok: down: postgresql: 1s, normally up
ok: down: rabbitmq: 0s, normally up
ok: down: redis_lb: 1s, normally up
sed: can't read /opt/chef-server/bin/wait-for-rabbit: No such file or directory
Failed to write fix to wait-for-rabbit script


Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html


-----Original message-----
From: Stephen Delano < >
Sent: Sunday 28th June 2015 12:02
To:
Subject: [chef] Re: Chef server update went wrong - how to repair?

Hi Kevin,

Running `chef-server-ctl upgrade` and choosing not to upgrade from the Chef Server 11 data will fix your issue. In Chef Server 12.1.0, there are schema changes to the database that make the cookbook versions endpoint cause less of a load on the database. It appears that the errors you are seeing are a result of this schema not getting applied.


Stephen Delano - Engineering Lead, Chef


On Sun, Jun 28, 2015 at 11:51 AM, Kevin Keane Subscription < " target="_blank" title="This external link will open in a new window"> > wrote:

A few weeks ago, I finally updated from chef 11 to chef 12 (I believe it was 12.0.3 at the time). The upgrade went smoothly and without a hitch that I could tell.

A few days ago, I updated to chef-server-core 12.1.0, and all hell broke loose. Well, not all hell, but some critical parts of chef aren't working.

The RPM told me that it detected chef 11, and to run chef-server-ctl update. I wasn't sure why it still saw chef 11 since I already upgraded to 12 a while ago, but I followed the instruction.

Now my chef client runs are failing with an "internal service error". knife node show works without a problem, so he node data actually exists.

I haven't found anything that seemed relevant in any log files, but then I'm ot sure what I'd be looking for. I did see error 500s in the nginx logs for the URL: POST /organizations/myorganization/environments/_default/cookbook_versions HTTP/1.1

How do I go about troubleshooting and fixing this?


Here is what happens on one of my clients:

Starting Chef Client, version 12.3.0
resolving cookbooks for run list: [ <redacted - the run list is correct> ]

================================================================================
Error Resolving Cookbooks for Run List:
================================================================================

Unknown Server Error:
---------------------
The server had a fatal error attempting to load the node data.

Server Response:
----------------
internal service error


Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 108.207621603 seconds



Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html






Archive powered by MHonArc 2.6.16.

§