General discussion about Chef

[chef] Re: Re: Re: Re: Re: Chef Server 12 upgrade failure


Chronological Thread 
  • From: Mark Mzyk < >
  • To:
  • Subject: [chef] Re: Re: Re: Re: Re: Chef Server 12 upgrade failure
  • Date: Fri, 06 Mar 2015 17:40:20 -0500
I'm not sure why the chef12-upgrade-download command would be taking so much cpu. It is, as you say, running a knife download as a first step. After that step it then accesses the db to grab some final data that it doesn't get through knife download.

You can try to run this process with your own local knife. You just need to replicate what the code is doing, specifically in this method: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb#L58

Note that you can inspect all the data it already downloaded. It's on disk in /tmp/chef11-<randomstring> unless you used the command line option to save it somewhere else. It's all in json and text files, so it's easy to see what is there.

As far as replicating what chef12-upgrade-download is doing, you'll want to make sure your config is set properly, like so: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb#L337

Then run knife download: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb#L351

The hardest part is that you'll need to make the key_dump.json file that will live with your chef11 data at the top level dir. That is created in the upgrade process here: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb#L39

but is populated later on here:

https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb#L379

Once you have the data downloaded and the key_dump.json in place, you've done all the steps that the chef12-upgrade-download command would do. You can then pick up with the chef12-upgrade-data-transform command and point it at your data location.

It's hard for me to offer any more advice than this without seeing stack traces and having more info. If you continue to have issues it's probably best to open an issue against the chef-server repo.

Hope this helps.

- Mark




" type="cite">
" photoname="Michael Hart" src="jpgFnnV5v5dej.jpg" name="compose-unknown-contact.jpg" height="25px" width="25px">
March 6, 2015 at 2:54 PM
The download failed with an EOF error shortly after I sent the email.

We did a huge ‘spring’ cleaning of all the roles/cookbooks before this last run, we had a lot of cruft that needed cleansing. 

What’s odd is that the "chef-server-ctl chef12-upgrade-download” command is just running a “knife download /“ behind the scenes, and that process used 100% of one CPU for the full 7 hours. It appeared to slow down over time, and I noticed that it opens/closes a new https session for every file it downloads. 

In comparison, for backups we’re using the knife backup plugin (I think from https://github.com/mdxp/knife-backup) and it can export the entire server in about 1.5 hours. We run a full backup every night and a —latest every few hours. We’re still trying to figure out what the difference is and why one works and the other fails. And if there’s a difference in the data that they provide.

Thanks
Mike

— 
Michael Hart
Arctic Wolf Networks
226.388.4773


From: Mark Mzyk
Reply-To: " "> "
Date: Friday, March 6, 2015 at 08:58
To: " "> "
Subject: [chef] Re: Re: Re: Chef Server 12 upgrade failure

Having it on the same node is fine; that's the default case the upgrade process was designed for. Hopefully it all goes well this time. If it does not, let us know and we'll help you get it sorted. The upgrade is taking as long as it is because the process is download all the data/transform it to Chef 12 format/upload all the data to a new server. The more data you have the longer it'll take (having many cookbook versions is a particular culprit here). It was done this way so that all your data moved through the API so we could ensure it would be in a good format on the Chef 12 server. We recognized that this might take a while, which is why the chef12-upgrade-* commands exist to give you finer control over the process.

- Mark
" photoname="Mark Mzyk" src="jpgFnnV5v5dej.jpg" name="compose-unknown-contact.jpg" height="25px" width="25px">
March 6, 2015 at 8:58 AM
Having it on the same node is fine; that's the default case the upgrade process was designed for. Hopefully it all goes well this time. If it does not, let us know and we'll help you get it sorted. The upgrade is taking as long as it is because the process is download all the data/transform it to Chef 12 format/upload all the data to a new server. The more data you have the longer it'll take (having many cookbook versions is a particular culprit here). It was done this way so that all your data moved through the API so we could ensure it would be in a good format on the Chef 12 server. We recognized that this might take a while, which is why the chef12-upgrade-* commands exist to give you finer control over the process.

- Mark

" photoname="Michael Hart" src="jpgFnnV5v5dej.jpg" name="compose-unknown-contact.jpg" height="25px" width="25px">
March 5, 2015 at 3:35 PM
Hi Mark,

Thanks for the response. We have chef server 11 and 12 installed on the same node, which is arguably not the best approach but so be it.

So versions:
Chef Server 11.1.1-1
Chef Server 12.0.5-1
Chef Client 11.16.4-1

We trimmed down the amount of data in chef significantly and am running another export again. I dare not say that it’s going well for fear of jinxing it, except that it’s been running for > 6 hours. 

Thanks
mike
— 
Michael Hart
Arctic Wolf Networks
226.388.4773


From: Mark Mzyk
Reply-To: " "> "
Date: Thursday, March 5, 2015 at 11:25
To: " "> "
Subject: [chef] Re: Chef Server 12 upgrade failure

Hi Michael,

You're correct that it's using knife download. To give you more insight into the process, all the upgrade code is here: https://github.com/chef/opscode-omnibus/tree/master/files/private-chef-ctl-commands

Look at the chef12_upgrade* files and the open_source_chef12_upgrade.rb file.

Specifically, the knife download is run here: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb

I don't know why it would fail during the upgrade but work for you locally. On the upgrade it is using the bundled in chef-client that is used to configure the chef server, so perhaps it is a difference in client version. You can, if you explore the upgrade code, replicate manually what the upgrade code is doing using your local chef-client and then you could proceed on with the other steps.

Can you tell me what version of the chef-client you're using that works? I think I've seen this error crop up for others but I'm not sure what causes it. Also what versions are your Chef 11 server and Chef 12 server package?

Thanks,

Mark

" photoname="Mark Mzyk" src="jpgFnnV5v5dej.jpg" name="compose-unknown-contact.jpg" height="25px" width="25px">
March 5, 2015 at 11:25 AM
Hi Michael,

You're correct that it's using knife download. To give you more insight into the process, all the upgrade code is here: https://github.com/chef/opscode-omnibus/tree/master/files/private-chef-ctl-commands

Look at the chef12_upgrade* files and the open_source_chef12_upgrade.rb file.

Specifically, the knife download is run here: https://github.com/chef/opscode-omnibus/blob/master/files/private-chef-ctl-commands/open_source_chef12_upgrade.rb

I don't know why it would fail during the upgrade but work for you locally. On the upgrade it is using the bundled in chef-client that is used to configure the chef server, so perhaps it is a difference in client version. You can, if you explore the upgrade code, replicate manually what the upgrade code is doing using your local chef-client and then you could proceed on with the other steps.

Can you tell me what version of the chef-client you're using that works? I think I've seen this error crop up for others but I'm not sure what causes it. Also what versions are your Chef 11 server and Chef 12 server package?

Thanks,

Mark

" photoname="Michael Hart" src="jpgFnnV5v5dej.jpg" name="compose-unknown-contact.jpg" height="25px" width="25px">
March 4, 2015 at 12:46 PM
We are trying to upgrade to Chef Server 12 (from 11.1.1), and ran into this error when running the "chef-server-ctl chef12-upgrade-download” command (which I think is just doing “knife download /“):

ERROR: Chef::ChefFS::FileSystem::NotFoundError: Chef::ChefFS::FileSystem::NotFoundError
knife download failed with pid 9244 exit 100

It was in the middle of downloading a few versions of a cookbook, which I can download successfully using knife cookbook download.

From what we can tell there’s no errors in the chef server log at all.

Any suggestions/ideas?

Thanks
mike
— 
Michael Hart
Arctic Wolf Networks
226.388.4773


Archive powered by MHonArc 2.6.16.

§