After update to mariadb-10.2.7-5.fc27.x86_64 - the same result.

Hello,
I tried to reproduce, but so far I wasn't succesful.

I have the exact set of the packages - mariadb, openssl, compat-openssl.

--

Do you know from which version did you updated?
Could you try to uninstall and re-install mariadb completely?
Which Fedora version do you use? (I suppose you use 25 or 26 with Rawhide repository)

last working was mariadb-10.2.6-4.fc27

now used is mariadb-10.2.7-6.fc27 - not work too.
rpm -V mariadb shows nothing.
Used rawhide - rolling for long time.
Some packages was rebuilded myself - but seems not closely realted.

I suppose to wait untill FC27 mass rebuild finish
before investigating.

After update (the same result):
rpm -q mariadb
mariadb-10.2.7-8.fc27.x86_64
rpm -q openssl
openssl-1.1.0f-9.fc27.x86_64

ldd /usr/libexec/mysqld
        linux-vdso.so.1 (0x00007ffcea6c6000)
        libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f8a840cc000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a83ead000)
        libaio.so.1 => /lib64/libaio.so.1 (0x00007f8a83cab000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f8a83a94000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8a83821000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f8a835eb000)
        libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f8a8337f000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8a82ef8000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a82cf4000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f8a8296e000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f8a82619000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f8a82234000)
        libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f8a81f27000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f8a81d1f000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8a81af9000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f8a818e5000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8a816bc000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a814a5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8a86048000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007f8a812a2000)
        libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f8a8108d000)

I can also send strace and ltrace logs, or other debug info,
but I failed to detect come criminal behaviour for now.

This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

gdb trace shows, that error occurs on function
CRYPTO_set_mem_functions called from
check_openssl_compatibility ()

ltrace part:
CRYPTO_set_mem_functions(0x916f6ba4f0, 0x7f7817924aa0, 0x7f7817924a30, 0x9170ca8698) = 0

Anton, do you by any chance use some server-side plugins? Especially not coming from mariadb Fedora build?

I carefully checked directories
/etc/mysql/
  fabric.cfg - mysql-utilities-1.5.6-5.fc27.noarch

/etc/my.cnf.d/
  auth_gssapi.cnf
  cracklib_password_check.cnf
  enable_encryption.preset
  mariadb-server.cnf
  mysql-clients.cnf
  rocksdb.cnf
  tokudb.cnf

(all from mariadb-server-10.2.7-8.fc27.x86_64, mariadb-config-10.2.7-8.fc27.x86_64, no difference)

local config have only password - removed too.

/usr/lib64/mysql (9 files) (+plugin: 45 files)

Nothing external

Sometimes i used percona-toolkit, but
 1. Was removed during investigation
 2. It have no plugins - only executables.

Now I have suspicion about OpneSSL usage, hope to verify soon.

(In reply to Anton Guda from comment #8)
>   fabric.cfg - mysql-utilities-1.5.6-5.fc27.noarch

mysql-utilities uses mysql-connector-python, which is Python-only implementation of the MySQL protocol, so shouldn't cause troubles here.

I rebuilt the mariadb package, with patch, where chech_openssl_compatibility always returns 0.
All works for me. It seems, that CRYPTO_set_mem_functions
sometimes fails w/o known reasons.

I was able to reproduce mariadb startup failure issue. Apparently it's caused by dracut-fips-046-5.fc27.x86_64 package. Reproduce steps...

##### Start mariadb check status
# systemctl start mariadb
# systemctl status mariadb
● mariadb.service - MariaDB 10.2 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-21 10:01:01 PST; 8s ago
# systemctl stop mariadb

##### Install dracut-fips-046-5.fc27.x86_64 package.

# dnf install dracut-fips-046-5.fc27.x86_64
Last metadata expiration check: 0:06:00 ago on Tue 21 Nov 2017 09:56:48 AM PST.
Dependencies resolved.
=============================================================================================================================================
 Package                            Arch                          Version                                Repository                     Size
=============================================================================================================================================
Installing:
 dracut-fips                        x86_64                        046-5.fc27                             fedora                         48 k
Installing dependencies:
 hmaccalc                           x86_64                        0.9.14-8.fc27                          fedora                         32 k

Transaction Summary
=============================================================================================================================================
Install  2 Packages

Total download size: 81 k
Installed size: 104 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): hmaccalc-0.9.14-8.fc27.x86_64.rpm                                                                     113 kB/s |  32 kB     00:00
(2/2): dracut-fips-046-5.fc27.x86_64.rpm                                                                     168 kB/s |  48 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                        114 kB/s |  81 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                     1/1
  Installing       : hmaccalc-0.9.14-8.fc27.x86_64                                                                                       1/2
  Installing       : dracut-fips-046-5.fc27.x86_64                                                                                       2/2
  Running scriptlet: dracut-fips-046-5.fc27.x86_64                                                                                       2/2
  Verifying        : dracut-fips-046-5.fc27.x86_64                                                                                       1/2
  Verifying        : hmaccalc-0.9.14-8.fc27.x86_64                                                                                       2/2

Installed:
  dracut-fips.x86_64 046-5.fc27                                         hmaccalc.x86_64 0.9.14-8.fc27

Complete!

##### Start Mariadb again. It fail with Incompatible OpenSSL version. Cannot continue...

# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl  status mariadb.service" and "journalctl  -xe" for details.

# systemctl status mariadb
● mariadb.service - MariaDB 10.2 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-11-21 10:04:34 PST; 4min 11s ago
  Process: 20748 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
  Process: 21077 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=exited, status=1/FAILURE)
  Process: 21040 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
  Process: 21016 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 21077 (code=exited, status=1/FAILURE)

Nov 21 10:04:33 systemd[1]: Starting MariaDB 10.2 database server...
Nov 21 10:04:34 mysql-prepare-db-dir[21040]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Nov 21 10:04:34 mysql-prepare-db-dir[21040]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.
Nov 21 10:04:34 mysqld[21077]: 2017-11-21 10:04:34 140737353538816 [ERROR] Incompatible OpenSSL version. Cannot continue...

##### Remove dracut-fips-046-5.fc27.x86_64, mariadb start successfully again.

# dnf remove dracut-fips-046-5.fc27.x86_64
# systemctl start mariadb
# systemctl status mariadb
● mariadb.service - MariaDB 10.2 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-21 10:11:57 PST; 6s ago

Confirm, removing dracut-fips really helps.
May be add this package to conflicts?

The package conflict is used only for packages that can't be *installed* together.

I went through this issue today, and I found the cause and reported it to the upstream.
https://jira.mariadb.org/browse/MDEV-14567

I can't help you much, because the 'dracut-fips' package actually alter the behaviour of openssl and the MariaDB won't start as expected when such behavious occurs.
However MariaDB should be able to run in FIPS mode, so I think upstream will be interested in that issue.

mariadb-10.2.13-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4a4dc47afe

mariadb-10.2.13-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-83bbd0c22f

mariadb-10.2.13-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-00647ae0d5

mariadb-10.2.13-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-83bbd0c22f

mariadb-10.2.13-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-00647ae0d5

mariadb-10.2.13-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

mariadb-10.2.13-2.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.