- From: "Vinayak, Gaurav" <
>
- To: "
" <
>, "
" <
>
- Subject: [chef] RE: RE: Issues with SQL Server 2008 R2 Recipe
- Date: Thu, 5 Jun 2014 17:40:47 +0000
- Accept-language: en-US
Thanks for responding Adam!
Please find attached the logs for chef client run & also the winrm
configuration parameter.
Regards,
Gaurav
________________________________________
From: Adam Edwards
<
>
Sent: Thursday, June 5, 2014 8:56 AM
To:
Subject: [chef] RE: Issues with SQL Server 2008 R2 Recipe
Gaurav, do you have a log from the Chef run? That would point to clues.
One issue I've seen with sql server is that it uses a lot of memory (it
will basically allocate as much memory as physical ram on your system for
performance reasons), and since knife-windows is executing over winrm, you
can run afoul of WinRM's memory restrictions. You could try setting
maxmemorypershellmb to something huge, say larger than physical ram on the
system.
Thanks.
-Adam
-----Original Message-----
From: Vinayak
[mailto:
Sent: Thursday, June 5, 2014 12:04 AM
To:
Subject: [chef] Issues with SQL Server 2008 R2 Recipe
Guys,
I am using the latest version of SQL Server Cookbook i.e. 2.2 on the chef
community & the chef-client version is 11.12.4.
Issue:
Even after successful execution of the recipe & i don't get to see the SQL
Server Installed on the target box whereas IIS got installed flawlessly.
I am using the knife windows plugin for bootstrapping.
Target Box OS - Win 2008 R2 64-Bit.
Anybody faced similar issue or can advise what's wrong here?
Regards,
Gaurav Vinayak
Waiting for remote response before
bootstrap.ec2-54-227-214-241.compute-1.amazonaws.com .
ec2-54-227-214-241.compute-1.amazonaws.com Response received.
Remote node responded after 0.0 minutes.
Bootstrapping Chef on ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 1"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 2"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 3"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 4"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 5"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 6"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 7"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 8"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk 9"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk
10"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk
11"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk
12"
ec2-54-227-214-241.compute-1.amazonaws.com "Rendering
"C:\Users\ADMINI~1\AppData\Local\Temp\bootstrap-25030-1401988557.bat" chunk
13"
ec2-54-227-214-241.compute-1.amazonaws.com Checking for existing directory
"C:\chef"...
ec2-54-227-214-241.compute-1.amazonaws.com Existing directory found, skipping
creation.
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>(
ec2-54-227-214-241.compute-1.amazonaws.com echo.url =
WScript.Arguments.Named("url")
ec2-54-227-214-241.compute-1.amazonaws.com echo.path =
WScript.Arguments.Named("path")
ec2-54-227-214-241.compute-1.amazonaws.com echo.proxy = null
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objXMLHTTP =
CreateObject("MSXML2.ServerXMLHTTP")
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set wshShell = CreateObject(
"WScript.Shell" )
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objUserVariables =
wshShell.Environment("USER")
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.rem http proxy is optional
ec2-54-227-214-241.compute-1.amazonaws.com echo.rem attempt to read from
HTTP_PROXY env var first
ec2-54-227-214-241.compute-1.amazonaws.com echo.On Error Resume Next
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.If NOT
(objUserVariables("HTTP_PROXY") = "") Then
ec2-54-227-214-241.compute-1.amazonaws.com echo.proxy =
objUserVariables("HTTP_PROXY")
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.rem fall back to named arg
ec2-54-227-214-241.compute-1.amazonaws.com echo.ElseIf NOT
(WScript.Arguments.Named("proxy") = "") Then
ec2-54-227-214-241.compute-1.amazonaws.com echo.proxy =
WScript.Arguments.Named("proxy")
ec2-54-227-214-241.compute-1.amazonaws.com echo.End If
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.If NOT isNull(proxy) Then
ec2-54-227-214-241.compute-1.amazonaws.com echo.rem setProxy method is only
available on ServerXMLHTTP 6.0+
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objXMLHTTP =
CreateObject("MSXML2.ServerXMLHTTP.6.0")
ec2-54-227-214-241.compute-1.amazonaws.com echo.objXMLHTTP.setProxy 2, proxy
ec2-54-227-214-241.compute-1.amazonaws.com echo.End If
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.On Error Goto 0
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.objXMLHTTP.open "GET", url,
false
ec2-54-227-214-241.compute-1.amazonaws.com echo.objXMLHTTP.send()
ec2-54-227-214-241.compute-1.amazonaws.com echo.If objXMLHTTP.Status = 200
Then
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objADOStream =
CreateObject("ADODB.Stream")
ec2-54-227-214-241.compute-1.amazonaws.com echo.objADOStream.Open
ec2-54-227-214-241.compute-1.amazonaws.com echo.objADOStream.Type = 1
ec2-54-227-214-241.compute-1.amazonaws.com echo.objADOStream.Write
objXMLHTTP.ResponseBody
ec2-54-227-214-241.compute-1.amazonaws.com echo.objADOStream.Position = 0
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objFSO =
Createobject("Scripting.FileSystemObject")
ec2-54-227-214-241.compute-1.amazonaws.com echo.If objFSO.Fileexists(path)
Then objFSO.DeleteFile path
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objFSO = Nothing
ec2-54-227-214-241.compute-1.amazonaws.com echo.objADOStream.SaveToFile path
ec2-54-227-214-241.compute-1.amazonaws.com echo.objADOStream.Close
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objADOStream = Nothing
ec2-54-227-214-241.compute-1.amazonaws.com echo.End if
ec2-54-227-214-241.compute-1.amazonaws.com echo.Set objXMLHTTP = Nothing
ec2-54-227-214-241.compute-1.amazonaws.com ) 1>C:\chef\wget.vbs
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>(
ec2-54-227-214-241.compute-1.amazonaws.com echo.param(
ec2-54-227-214-241.compute-1.amazonaws.com echo. [String] $remoteUrl,
ec2-54-227-214-241.compute-1.amazonaws.com echo. [String] $localPath
ec2-54-227-214-241.compute-1.amazonaws.com echo.)
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.$webClient = new-object
System.Net.WebClient;
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com
echo.$webClient.DownloadFile($remoteUrl, $localPath);
ec2-54-227-214-241.compute-1.amazonaws.com ) 1>C:\chef\wget.ps1
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>(
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com )
ec2-54-227-214-241.compute-1.amazonaws.com Detected Windows Version 6.1 Build
7601
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>goto
Version6.1
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>goto
architecture_select
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>goto
Architecture
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>goto
ArchitectureAMD64
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>goto install
ec2-54-227-214-241.compute-1.amazonaws.com Checking for existing downloaded
package at "C:\Users\ADMINI~1\AppData\Local\Temp\chef-client-latest.msi"
ec2-54-227-214-241.compute-1.amazonaws.com Found existing downloaded package,
deleting.
ec2-54-227-214-241.compute-1.amazonaws.com Attempting to download client
package using cscript...
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>cscript
/nologo C:\chef\wget.vbs
/url:"
https://www.opscode.com/chef/download?p=windows&pv=2008r2&m=x86_64"
/path:"C:\Users\ADMINI~1\AppData\Local\Temp\chef-client-latest.msi"
ec2-54-227-214-241.compute-1.amazonaws.com CScript Error: Execution of the
Windows Script Host failed. (0x800A0007)
ec2-54-227-214-241.compute-1.amazonaws.com Failed cscript download with
status code 1
ec2-54-227-214-241.compute-1.amazonaws.com Warning: Failed to download
"
https://www.opscode.com/chef/download?p=windows&pv=2008r2&m=x86_64" to
"C:\Users\ADMINI~1\AppData\Local\Temp\chef-client-latest.msi"
ec2-54-227-214-241.compute-1.amazonaws.com Warning: Retrying download with
PowerShell if available...
ec2-54-227-214-241.compute-1.amazonaws.com powershell.exe -ExecutionPolicy
Unrestricted -NoProfile -NonInteractive -File C:\chef\wget.ps1
"
https://www.opscode.com/chef/download?p=windows&pv=2008r2&m=x86_64&DownloadContext=PowerShell"
"C:\Users\ADMINI~1\AppData\Local\Temp\chef-client-latest.msi"
ec2-54-227-214-241.compute-1.amazonaws.com Download via PowerShell succeeded.
ec2-54-227-214-241.compute-1.amazonaws.com Installing downloaded client
package...
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>msiexec /qn
/log "C:\Users\ADMINI~1\AppData\Local\Temp\chef-client-msi25572.log" /i
"C:\Users\ADMINI~1\AppData\Local\Temp\chef-client-latest.msi"
ec2-54-227-214-241.compute-1.amazonaws.com Installation completed successfully
ec2-54-227-214-241.compute-1.amazonaws.com Writing validation key...
ec2-54-227-214-241.compute-1.amazonaws.com Validation key written.
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>(
ec2-54-227-214-241.compute-1.amazonaws.com echo.log_level :info
ec2-54-227-214-241.compute-1.amazonaws.com echo.log_location STDOUT
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.chef_server_url
"
https://api.opscode.com/organizations/onpremise"
ec2-54-227-214-241.compute-1.amazonaws.com echo.validation_client_name
"onpremise-validator"
ec2-54-227-214-241.compute-1.amazonaws.com echo.client_key
"c:/chef/client.pem"
ec2-54-227-214-241.compute-1.amazonaws.com echo.validation_key
"c:/chef/validation.pem"
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.file_cache_path
"c:/chef/cache"
ec2-54-227-214-241.compute-1.amazonaws.com echo.file_backup_path
"c:/chef/backup"
ec2-54-227-214-241.compute-1.amazonaws.com echo.cache_options ({:path =>
"c:/chef/cache/checksums", :skip_expires => true})
ec2-54-227-214-241.compute-1.amazonaws.com echo.
ec2-54-227-214-241.compute-1.amazonaws.com echo.node_name "win_node1"
ec2-54-227-214-241.compute-1.amazonaws.com ) 1>C:\chef\client.rb
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com
C:\Users\Administrator>(echo.{"run_list":["recipe[sql_server::server]"]})
1>C:\chef\first-boot.json
ec2-54-227-214-241.compute-1.amazonaws.com Starting chef to bootstrap the
node...
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>SET
"PATH=C:\Program Files\Microsoft SQL
Server\100\Tools\Binn;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
Files\Amazon\cfn-bootstrap\;C:\opscode\chef\bin;C:\opscode\chef\embedded\bin;c:\Program
Files\Microsoft SQL
Server\100\Tools\Binn\Redist\;C:\ruby\bin;C:\opscode\chef\bin;C:\opscode\chef\embedded\bin"
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com C:\Users\Administrator>chef-client
-c c:/chef/client.rb -j c:/chef/first-boot.json -E _default
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:30+00:00] WARN:
ec2-54-227-214-241.compute-1.amazonaws.com * * * * * * * * * * * * * * * * *
* * * * * * * * * * * * * * * * * * * * * * *
ec2-54-227-214-241.compute-1.amazonaws.com SSL validation of HTTPS requests
is disabled. HTTPS connections are still
ec2-54-227-214-241.compute-1.amazonaws.com encrypted, but chef is not able to
detect forged replies or man in the middle
ec2-54-227-214-241.compute-1.amazonaws.com attacks.
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com To fix this issue add an entry
like this to your configuration file:
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com ```
ec2-54-227-214-241.compute-1.amazonaws.com # Verify all HTTPS connections
(recommended)
ec2-54-227-214-241.compute-1.amazonaws.com ssl_verify_mode :verify_peer
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com # OR, Verify only connections to
chef-server
ec2-54-227-214-241.compute-1.amazonaws.com verify_api_cert true
ec2-54-227-214-241.compute-1.amazonaws.com ```
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com To check your SSL configuration,
or troubleshoot errors, you can use the
ec2-54-227-214-241.compute-1.amazonaws.com `knife ssl check` command like so:
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com ```
ec2-54-227-214-241.compute-1.amazonaws.com knife ssl check -c
c:/chef/client.rb
ec2-54-227-214-241.compute-1.amazonaws.com ```
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com * * * * * * * * * * * * * * * * *
* * * * * * * * * * * * * * * * * * * * * * *
ec2-54-227-214-241.compute-1.amazonaws.com
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:34+00:00] INFO:
*** Chef 11.12.4 ***
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:34+00:00] INFO:
Chef-client pid: 2688
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:42+00:00] INFO:
Setting the run_list to ["recipe[sql_server::server]"] from CLI options
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:42+00:00] INFO:
Run List is [recipe[sql_server::server]]
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:42+00:00] INFO:
Run List expands to [sql_server::server]
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:42+00:00] INFO:
Starting Chef Run for win_node1
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:42+00:00] INFO:
Running start handlers
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:42+00:00] INFO:
Start handlers complete.
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:43+00:00] INFO:
Loading cookbooks
,
,
,
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:43+00:00] INFO:
WindowsPackage light-weight resource already initialized -- overriding!
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:44+00:00] INFO:
Processing template[c:\chef\cache\ConfigurationFile.ini] action create
(sql_server::server line 47)
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:44+00:00] INFO:
Processing windows_package[Microsoft SQL Server 2008 R2 (64-bit)] action
install (sql_server::server line 51)
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:44+00:00] INFO:
Installing windows_package[Microsoft SQL Server 2008 R2 (64-bit)] version
latest
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:44+00:00] INFO:
Processing remote_file[c:/chef/cache/SQLEXPR_x64_ENU.exe] action create
(dynamically defined)
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:17:46+00:00] INFO:
Starting installation...this could take awhile.
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:18:26+00:00] INFO:
Processing service[MSSQL$SQLEXPRESS] action nothing (sql_server::server line
60)
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:18:26+00:00] INFO:
Processing registry_key[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL
Server\MSSQL10_50.SQLEXPRESS\MSSQLServer\SuperSocketNetLib\Tcp\IPAll] action
create_if_missing (sql_server::server line 65)
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:18:26+00:00] INFO:
Processing gem_package[tiny_tds] action install (sql_server::client line 45)
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:18:27+00:00] INFO:
Chef Run complete in 44.9592 seconds
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:18:27+00:00] INFO:
Running report handlers
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:18:27+00:00] INFO:
Report handlers complete
ec2-54-227-214-241.compute-1.amazonaws.com [2014-06-05T17:18:27+00:00] INFO:
Sending resource update report (run-id: 19a3572e-0e3f-4fbe-84e6-547276669877)
[Config
MaxEnvelopeSizekb = 500
MaxTimeoutms = 1800000
MaxBatchItems = 32000
MaxProviderRequests = 4294967295
Client
NetworkDelayms = 5000
URLPrefix = wsman
AllowUnencrypted = false
Auth
Basic = true
Digest = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = false
DefaultPorts
HTTP = 5985
HTTPS = 5986
TrustedHosts = *
Service
RootSDDL =
O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
MaxConcurrentOperations = 4294967295
MaxConcurrentOperationsPerUser = 1500
EnumerationTimeoutms = 240000
MaxConnections = 300
MaxPacketRetrievalTimeSeconds = 120
AllowUnencrypted = true
Auth
Basic = true
Kerberos = true
Negotiate = true
Certificate = false
CredSSP = false
CbtHardeningLevel = Relaxed
DefaultPorts
HTTP = 5985
HTTPS = 5986
IPv4Filter = *
IPv6Filter = *
EnableCompatibilityHttpListener = false
EnableCompatibilityHttpsListener = false
CertificateThumbprint
AllowRemoteAccess = true
Winrs
AllowRemoteShellAccess = true
IdleTimeout = 7200000
MaxConcurrentUsers = 10
MaxShellRunTime = 2147483647
MaxProcessesPerShell = 25
MaxMemoryPerShellMB = 2000
MaxShellsPerUser = 30
Archive powered by MHonArc 2.6.16.