Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1473756
| Summary: | Fail to start server: incompatible OpenSSL version | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Anton Guda <atu> |
| Component: | mariadb | Assignee: | Michal Schorm <mschorm> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 27 | CC: | atu, dciabrin, hhorak, jstanek, mbayer, mmuzila, mschorm, praiskup, skpgkp1 |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | mariadb-10.2.13-2.fc27 mariadb-10.2.13-2.fc28 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-03-13 23:15:08 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1544986 | ||
| Bug Blocks: | |||
|
Description
Anton Guda
2017-07-21 14:39:00 UTC
After update to mariadb-10.2.7-5.fc27.x86_64 - the same result. Hello, I tried to reproduce, but so far I wasn't succesful. I have the exact set of the packages - mariadb, openssl, compat-openssl. -- Do you know from which version did you updated? Could you try to uninstall and re-install mariadb completely? Which Fedora version do you use? (I suppose you use 25 or 26 with Rawhide repository) last working was mariadb-10.2.6-4.fc27 now used is mariadb-10.2.7-6.fc27 - not work too. rpm -V mariadb shows nothing. Used rawhide - rolling for long time. Some packages was rebuilded myself - but seems not closely realted. I suppose to wait untill FC27 mass rebuild finish before investigating. After update (the same result):
rpm -q mariadb
mariadb-10.2.7-8.fc27.x86_64
rpm -q openssl
openssl-1.1.0f-9.fc27.x86_64
ldd /usr/libexec/mysqld
linux-vdso.so.1 (0x00007ffcea6c6000)
libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f8a840cc000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a83ead000)
libaio.so.1 => /lib64/libaio.so.1 (0x00007f8a83cab000)
libz.so.1 => /lib64/libz.so.1 (0x00007f8a83a94000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8a83821000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f8a835eb000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f8a8337f000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8a82ef8000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a82cf4000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f8a8296e000)
libm.so.6 => /lib64/libm.so.6 (0x00007f8a82619000)
libc.so.6 => /lib64/libc.so.6 (0x00007f8a82234000)
libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f8a81f27000)
librt.so.1 => /lib64/librt.so.1 (0x00007f8a81d1f000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8a81af9000)
liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f8a818e5000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8a816bc000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a814a5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8a86048000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f8a812a2000)
libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f8a8108d000)
I can also send strace and ltrace logs, or other debug info,
but I failed to detect come criminal behaviour for now.
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle. Changing version to '27'. gdb trace shows, that error occurs on function CRYPTO_set_mem_functions called from check_openssl_compatibility () ltrace part: CRYPTO_set_mem_functions(0x916f6ba4f0, 0x7f7817924aa0, 0x7f7817924a30, 0x9170ca8698) = 0 Anton, do you by any chance use some server-side plugins? Especially not coming from mariadb Fedora build? I carefully checked directories /etc/mysql/ fabric.cfg - mysql-utilities-1.5.6-5.fc27.noarch /etc/my.cnf.d/ auth_gssapi.cnf cracklib_password_check.cnf enable_encryption.preset mariadb-server.cnf mysql-clients.cnf rocksdb.cnf tokudb.cnf (all from mariadb-server-10.2.7-8.fc27.x86_64, mariadb-config-10.2.7-8.fc27.x86_64, no difference) local config have only password - removed too. /usr/lib64/mysql (9 files) (+plugin: 45 files) Nothing external Sometimes i used percona-toolkit, but 1. Was removed during investigation 2. It have no plugins - only executables. Now I have suspicion about OpneSSL usage, hope to verify soon. (In reply to Anton Guda from comment #8) > fabric.cfg - mysql-utilities-1.5.6-5.fc27.noarch mysql-utilities uses mysql-connector-python, which is Python-only implementation of the MySQL protocol, so shouldn't cause troubles here. I rebuilt the mariadb package, with patch, where chech_openssl_compatibility always returns 0. All works for me. It seems, that CRYPTO_set_mem_functions sometimes fails w/o known reasons. I was able to reproduce mariadb startup failure issue. Apparently it's caused by dracut-fips-046-5.fc27.x86_64 package. Reproduce steps... ##### Start mariadb check status # systemctl start mariadb # systemctl status mariadb ● mariadb.service - MariaDB 10.2 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-11-21 10:01:01 PST; 8s ago # systemctl stop mariadb ##### Install dracut-fips-046-5.fc27.x86_64 package. # dnf install dracut-fips-046-5.fc27.x86_64 Last metadata expiration check: 0:06:00 ago on Tue 21 Nov 2017 09:56:48 AM PST. Dependencies resolved. ============================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================= Installing: dracut-fips x86_64 046-5.fc27 fedora 48 k Installing dependencies: hmaccalc x86_64 0.9.14-8.fc27 fedora 32 k Transaction Summary ============================================================================================================================================= Install 2 Packages Total download size: 81 k Installed size: 104 k Is this ok [y/N]: y Downloading Packages: (1/2): hmaccalc-0.9.14-8.fc27.x86_64.rpm 113 kB/s | 32 kB 00:00 (2/2): dracut-fips-046-5.fc27.x86_64.rpm 168 kB/s | 48 kB 00:00 --------------------------------------------------------------------------------------------------------------------------------------------- Total 114 kB/s | 81 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : hmaccalc-0.9.14-8.fc27.x86_64 1/2 Installing : dracut-fips-046-5.fc27.x86_64 2/2 Running scriptlet: dracut-fips-046-5.fc27.x86_64 2/2 Verifying : dracut-fips-046-5.fc27.x86_64 1/2 Verifying : hmaccalc-0.9.14-8.fc27.x86_64 2/2 Installed: dracut-fips.x86_64 046-5.fc27 hmaccalc.x86_64 0.9.14-8.fc27 Complete! ##### Start Mariadb again. It fail with Incompatible OpenSSL version. Cannot continue... # systemctl start mariadb Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details. # systemctl status mariadb ● mariadb.service - MariaDB 10.2 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2017-11-21 10:04:34 PST; 4min 11s ago Process: 20748 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS) Process: 21077 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=exited, status=1/FAILURE) Process: 21040 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS) Process: 21016 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS) Main PID: 21077 (code=exited, status=1/FAILURE) Nov 21 10:04:33 systemd[1]: Starting MariaDB 10.2 database server... Nov 21 10:04:34 mysql-prepare-db-dir[21040]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done. Nov 21 10:04:34 mysql-prepare-db-dir[21040]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir. Nov 21 10:04:34 mysqld[21077]: 2017-11-21 10:04:34 140737353538816 [ERROR] Incompatible OpenSSL version. Cannot continue... ##### Remove dracut-fips-046-5.fc27.x86_64, mariadb start successfully again. # dnf remove dracut-fips-046-5.fc27.x86_64 # systemctl start mariadb # systemctl status mariadb ● mariadb.service - MariaDB 10.2 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-11-21 10:11:57 PST; 6s ago Confirm, removing dracut-fips really helps. May be add this package to conflicts? The package conflict is used only for packages that can't be *installed* together. I went through this issue today, and I found the cause and reported it to the upstream. https://jira.mariadb.org/browse/MDEV-14567 I can't help you much, because the 'dracut-fips' package actually alter the behaviour of openssl and the MariaDB won't start as expected when such behavious occurs. However MariaDB should be able to run in FIPS mode, so I think upstream will be interested in that issue. mariadb-10.2.13-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4a4dc47afe mariadb-10.2.13-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-83bbd0c22f mariadb-10.2.13-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-00647ae0d5 mariadb-10.2.13-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-83bbd0c22f mariadb-10.2.13-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-00647ae0d5 mariadb-10.2.13-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report. mariadb-10.2.13-2.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. |