Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1577805 - 4.5.0 -> 4.5.4 upgrade breaks in ipa-server-upgrade: No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
Summary: 4.5.0 -> 4.5.4 upgrade breaks in ipa-server-upgrade: No such file or director...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.5
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On: 1550555
Blocks: 1579203
TreeView+ depends on / blocked
 
Reported: 2018-05-14 07:37 UTC by Alexander Bokovoy
Modified: 2018-10-30 10:59 UTC (History)
15 users (show)

Fixed In Version: ipa-4.6.4-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1550555
: 1579203 (view as bug list)
Environment:
Last Closed: 2018-10-30 10:58:39 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
FedoraHosted FreeIPA 7409 0 None None None 2018-05-14 07:37:30 UTC
Red Hat Product Errata RHBA-2018:3187 0 None None None 2018-10-30 10:59:53 UTC

Description Alexander Bokovoy 2018-05-14 07:37:30 UTC 
This is a clone of Fedora 27 bug but we are getting the same error from CentOS 7.5 users too.

ipa-server-upgrade fails with "Upgrade failed with [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'"

CA-less + Let's Encrypt certificate actually


+++ This bug was initially created as a clone of Bug #1550555 +++

Description of problem:
Upgrading from FreeIPA 4.6.1 to 4.6.3, on F27. This installation with external CA. ipa-server-upgrade fails at:

ipaserver.install.ipa_server_upgrade: ERROR: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
ipapython.admintool: DEBUG:   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run
    server.upgrade()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1999, in upgrade
    upgrade_configuration()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1686, in upgrade_configuration
    ca.backup_config()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/dogtaginstance.py", line 475, in backup_config
    shutil.copy(path, path + '.ipabkp')
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

ipapython.admintool: DEBUG: The ipa-server-upgrade command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
ipapython.admintool: ERROR: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

--- Additional comment from James Ettle on 2018-03-01 15:05:36 EET ---

Had to roll back to 4.6.1, now back in action. So at least ipa-server-upgrade didn't hose the database...

--- Additional comment from Florence Blanc-Renaud on 2018-03-01 19:17:24 EET ---

Upstream ticket:
https://pagure.io/freeipa/issue/7409

--- Additional comment from Florence Blanc-Renaud on 2018-03-07 14:49:45 EET ---

Fixed upstream
master:
https://pagure.io/freeipa/c/95a45a2b0942a9ac38d5418b23821f7da1ce28a3
ipa-4-6:
https://pagure.io/freeipa/c/f24a3aeb1f39a790b61bd362718cb2fd16cf9f43

--- Additional comment from James Ettle on 2018-03-07 21:22:25 EET ---

Hold on, hold on. What do I have to do to test this without risking a broken database and having to start all over again?

--- Additional comment from Rob Crittenden on 2018-03-07 23:57:04 EET ---

Just update to the fixed packages and that should do it.

Even if the upgrade failed it wouldn't corrupt the database.

--- Additional comment from James Ettle on 2018-03-08 00:08:05 EET ---

Reopening. This is not fixed. Downgrading again.

# rpm -q freeipa-server
freeipa-server-4.6.3-2.fc27.x86_64

# systemctl status ipa
● ipa.service - Identity, Policy, Audit
   Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2018-03-07 22:06:01 GMT; 39s ago
  Process: 20965 ExecStart=/usr/sbin/ipactl start (code=exited, status=1/FAILURE)
 Main PID: 20965 (code=exited, status=1/FAILURE)

Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: [Verifying that root certificate is published]
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-serve
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more informat
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: See the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: Aborting ipactl
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: ipa.service: Main process exited, code=exited, status=1/FAILURE
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: Failed to start Identity, Policy, Audit.
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: ipa.service: Unit entered failed state.
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: ipa.service: Failed with result 'exit-code'.


End of /var/log/ipaupgrade.log:


2018-03-07T22:06:00Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2018-03-07T22:06:00Z DEBUG   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run
    server.upgrade()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1999, in upgrade
    upgrade_configuration()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1686, in upgrade_configuration
    ca.backup_config()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/dogtaginstance.py", line 475, in backup_config
    shutil.copy(path, path + '.ipabkp')
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

2018-03-07T22:06:00Z DEBUG The ipa-server-upgrade command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
2018-03-07T22:06:00Z ERROR [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
2018-03-07T22:06:00Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

--- Additional comment from James Ettle on 2018-03-09 01:06:00 EET ---

Looking at the SRPM those commits from Comment 3 simply aren't in 4.6.3-2 (ipaserver/install/server/upgrade.py). Patch0001 only deals with KRA-related stuff, but seems to be matching against code from that commit...

--- Additional comment from Rob Crittenden on 2018-03-12 20:52:25 EET ---

You're right, I missed a patch. There were two issues, one hiding the other. I'll spin up a new build.

--- Additional comment from Fedora Update System on 2018-03-13 16:15:06 EET ---

freeipa-4.6.3-3.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-0a4399f314

--- Additional comment from James Ettle on 2018-03-13 21:19:12 EET ---

OK - 4.6.3-3.fc27.x86_64 updated cleanly. Server rebooted, confirmed login with OTP, NFS and web interface work. Thanks, Rob!

--- Additional comment from Fedora Update System on 2018-03-14 03:39:18 EET ---

freeipa-4.6.3-3.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-0a4399f314
Comment 4 Florence Blanc-Renaud 2018-05-15 07:13:09 UTC 
Hi,

I don't think this is the same issue as Bug #1550555 because the problem happens at a different moment.
Following a discussion on IRC with joko, he provided the following logs:

--------------------
2018-05-14T19:53:07Z DEBUG Executing upgrade plugin: update_ca_renewal_master
2018-05-14T19:53:07Z DEBUG raw: update_ca_renewal_master
2018-05-14T19:53:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:07Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2018-05-14T19:53:07Z DEBUG certmonger request for RA cert not found
2018-05-14T19:53:07Z DEBUG Destroyed connection context.ldap2_140692280089040
2018-05-14T19:53:07Z ERROR Upgrade failed with [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
2018-05-14T19:53:07Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 220, in __upgrade
    self.modified = (ld.update(self.files) or self.modified)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 911, in update
    self._run_updates(all_updates)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 883, in _run_updates
    self._run_update_plugin(update['plugin'])
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 859, in _run_update_plugin
    restart_ds, updates = self.api.Updater[plugin_name]()
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1470, in __call__
    return self.execute(**options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/plugins/ca_renewal_master.py", line 106, in execute
    paths.CA_CS_CFG_PATH, 'subsystem.select', '=')
  File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 578, in get_directive
    fd = open(filename, "r")
IOError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'

2018-05-14T19:53:07Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 228, in __upgrade
    raise RuntimeError(e)
RuntimeError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'

2018-05-14T19:53:07Z DEBUG   [error] RuntimeError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
2018-05-14T19:53:07Z DEBUG   [cleanup]: stopping directory server
2018-05-14T19:53:07Z DEBUG Destroyed connection context.ldap2_140692311895056
2018-05-14T19:53:07Z DEBUG Starting external process
2018-05-14T19:53:07Z DEBUG args=/bin/systemctl stop dirsrv
2018-05-14T19:53:08Z DEBUG Process finished, return code=0
2018-05-14T19:53:08Z DEBUG stdout=
2018-05-14T19:53:08Z DEBUG stderr=
2018-05-14T19:53:08Z DEBUG   duration: 1 seconds
2018-05-14T19:53:08Z DEBUG   [cleanup]: restoring configuration
2018-05-14T19:53:08Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:08Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:08Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:08Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:08Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:08Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:08Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-14T19:53:08Z DEBUG   duration: 0 seconds
2018-05-14T19:53:08Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2018-05-14T19:53:08Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run
    raise admintool.ScriptError(str(e))

2018-05-14T19:53:08Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: ('IPA upgrade failed.', 1)
2018-05-14T19:53:08Z ERROR ('IPA upgrade failed.', 1)
2018-05-14T19:53:08Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

----------

It looks like the upgrade plugin update_ca_renewal_master is executed even though there is no CA installed. One of the first steps of the plugin is to check if ca.is_configured(), which is equivalent to looking for a [pki-tomcatd] section in /var/lib/ipa/sysrestore/sysrestore.state. The plugin should be skipped when the CA is not configured, but the logs show that it gets executed.

I asked the user to provide the sysrestore.state content, we'll be able to know more when we have this information available. Note: the logs confirm he's using IPA version 4.5.4-10.el7.centos.
Comment 5 Florence Blanc-Renaud 2018-05-15 11:32:41 UTC 
sysrestore.state contains
[pki-tomcatd]
installed = true

which means that the CA was installed on the host. It's probable that CA installation went wrong (or some files were manually deleted) and this is a different issue from 1550555.

After discussion with ab, we agreed to backport the fix for 1550555 to 4.5 anyway because the upgrade would show a WARNING that may be misleading.
Comment 7 Florence Blanc-Renaud 2018-05-15 15:12:46 UTC 
Fixed upstream
ipa-4-5:
https://pagure.io/freeipa/c/035f1cb24a228ba40b3e124d78a507be22aa52bd
Comment 12 Nikhil Dehadrai 2018-07-19 07:14:25 UTC 
IPA version :
ipa-server-4.6.4-2.el7.x86_64

Verified the bug using following steps:
1: Install CA-less ipa-server.
2: Upgrade to RHEL 7.6. 
3: tail /var/log/ipaupgrade.log, no errors mentioned in the bug are observed.

Tested for following paths:
1. RHEL 75z > 7.6
2. RHEL 75-0day > 7.6
3. RHEL 74z > 7.6


Thus on the basis of above observations marking the status of bug to 'VERIFIED'.
Comment 15 errata-xmlrpc 2018-10-30 10:58:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3187
Note You need to log in before you can comment on or make changes to this bug.