Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1607635 - Segfault in ldif2db during ipa-server-install on current Rawhide
Summary: Segfault in ldif2db during ipa-server-install on current Rawhide
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: softhsm
Version: rawhide
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Paul Wouters
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1570306 (view as bug list)
Depends On:
Blocks: F29BetaBlocker 1608690 1615751
TreeView+ depends on / blocked
 
Reported: 2018-07-23 21:54 UTC by Adam Williamson
Modified: 2018-08-14 07:31 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1608690 1615751 (view as bug list)
Environment:
Last Closed: 2018-08-01 18:32:45 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1570306 0 unspecified CLOSED softhsm crashes applications using p11-kit-proxy 2022-05-16 11:32:56 UTC
Red Hat Bugzilla 1606541 0 unspecified CLOSED freeipa-server missing dependency on 389-ds-base-legacy-tools, results in "No such file or directory: '/usr/sbin/setup-d... 2022-05-16 11:32:56 UTC

Internal Links: 1570306 1606541

Description Adam Williamson 2018-07-23 21:54:13 UTC 
As confirmed by multiple manual and automated test systems, attempting to deploy a FreeIPA server in current Rawhide fails due to a segfault in ldif2db - assuming you install 389-ds-base-legacy-tools manually to avoid running into https://bugzilla.redhat.com/show_bug.cgi?id=1606541 first.

The ipa-server-install log shows:

2018-07-23T21:16:42Z DEBUG calling setup-ds.pl
2018-07-23T21:16:42Z DEBUG Starting external process
2018-07-23T21:16:42Z DEBUG args=['/usr/sbin/setup-ds.pl', '--silent', '--logfile', '-', '-f', '/tmp/tmp6fgfe2a7']
2018-07-23T21:16:44Z DEBUG Process finished, return code=1
2018-07-23T21:16:44Z DEBUG stdout=[18/07/23:17:16:44] - [Setup] Info Could not import LDIF file '/var/lib/dirsrv/boot.ldif'.  Error: 35584.  Output: importing data ...
[23/Jul/2018:17:16:43.162647633 -0400] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[23/Jul/2018:17:16:43.175129006 -0400] - INFO - dblayer_instance_start - Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[23/Jul/2018:17:16:43.176112798 -0400] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 1530028032, process usage 23871488 
[23/Jul/2018:17:16:43.176620045 -0400] - INFO - check_and_set_import_cache - Import allocates 597667KB import cache.
[23/Jul/2018:17:16:43.231174115 -0400] - INFO - import_main_offline - import userRoot: Beginning import job...
[23/Jul/2018:17:16:43.232812822 -0400] - INFO - import_main_offline - import userRoot: Index buffering enabled with bucket size 100
[23/Jul/2018:17:16:43.434301363 -0400] - INFO - import_producer - import userRoot: Processing file "/var/lib/dirsrv/boot.ldif"
[23/Jul/2018:17:16:43.436276590 -0400] - INFO - import_producer - import userRoot: Finished scanning file "/var/lib/dirsrv/boot.ldif" (1 entries)
[23/Jul/2018:17:16:43.936857575 -0400] - INFO - import_monitor_threads - import userRoot: Workers finished; cleaning up...
[23/Jul/2018:17:16:44.039242916 -0400] - INFO - import_monitor_threads - import userRoot: Workers cleaned up.
[23/Jul/2018:17:16:44.041202683 -0400] - INFO - import_main_offline - import userRoot: Cleaning up producer thread...
[23/Jul/2018:17:16:44.043048675 -0400] - INFO - import_main_offline - import userRoot: Indexing complete.  Post-processing...
[23/Jul/2018:17:16:44.044870701 -0400] - INFO - import_main_offline - import userRoot: Generating numsubordinates (this may take several minutes to complete)...
[23/Jul/2018:17:16:44.049397446 -0400] - INFO - import_main_offline - import userRoot: Generating numSubordinates complete.
[23/Jul/2018:17:16:44.050721261 -0400] - INFO - ldbm_get_nonleaf_ids - import userRoot: Gathering ancestorid non-leaf IDs...
[23/Jul/2018:17:16:44.051367366 -0400] - INFO - ldbm_get_nonleaf_ids - import userRoot: Finished gathering ancestorid non-leaf IDs.
[23/Jul/2018:17:16:44.052816619 -0400] - ERR - ldbm_ancestorid_new_idl_create_index - Nothing to do to build ancestorid index
[23/Jul/2018:17:16:44.053291974 -0400] - INFO - ldbm_ancestorid_new_idl_create_index - import userRoot: Created ancestorid index (new idl).
[23/Jul/2018:17:16:44.053725299 -0400] - INFO - import_main_offline - import userRoot: Flushing caches...
[23/Jul/2018:17:16:44.054201783 -0400] - INFO - import_main_offline - import userRoot: Closing files...
[23/Jul/2018:17:16:44.068410788 -0400] - INFO - dblayer_pre_close - All database threads now stopped
[23/Jul/2018:17:16:44.069178807 -0400] - INFO - import_main_offline - import userRoot: Import complete.  Processed 1 entries in 1 seconds. (1.00 entries/sec)
/usr/sbin/ldif2db: line 116: 28878 Segmentation fault      /usr/sbin/ns-slapd ldif2db -D /etc/dirsrv/slapd-DOMAIN-LOCAL -n "userRoot" -i "/var/lib/dirsrv/boot.ldif"

Could not import LDIF file '/var/lib/dirsrv/boot.ldif'.  Error: 35584.  Output: importing data ...

This is 100% reproducible during IPA server install, it seems. mreynolds is looking into it now, I am filing this bug just so we have somewhere to track it.
Comment 1 mreynolds 2018-07-23 22:25:39 UTC 
The crash is happening in NSS:


main (argc=8, argv=0x7fffffffdfd8) at ldap/servers/slapd/main.c:939
939	        goto cleanup;
(gdb) 
1197	    compute_terminate();
(gdb) 
1198	    SSL_ShutdownServerSessionIDCache();
(gdb) 
1199	    SSL_ClearSessionCache();
(gdb) 
1200	    ndn_cache_destroy();
(gdb) 
1201	    NSS_Shutdown();
(gdb) 

Thread 1 "ns-slapd" received signal SIGSEGV, Segmentation fault.
0x00007ffff3c1e680 in ?? ()
(gdb) where
#0  0x00007ffff3c1e680 in None ()
#1  0x00007ffff3b75805 in Mutex::~Mutex() () at /usr/lib64/pkcs11/libsofthsm2.so
#2  0x00007ffff3b7c5bd in OSSLCryptoFactory::~OSSLCryptoFactory() () at /usr/lib64/pkcs11/libsofthsm2.so
#3  0x00007ffff3b7c5fd in OSSLCryptoFactory::~OSSLCryptoFactory() () at /usr/lib64/pkcs11/libsofthsm2.so
#4  0x00007ffff77aee67 in __cxa_finalize () at /lib64/libc.so.6
#5  0x00007ffff3b360a7 in __do_global_dtors_aux () at /usr/lib64/pkcs11/libsofthsm2.so
#6  0x00007fffffffd8c0 in None ()
#7  0x00007ffff7fe88ed in _dl_close_worker () at /lib64/ld-linux-x86-64.so.2


NSS Version:

nss-3.38.0-4
Comment 3 Alexander Bokovoy 2018-07-24 08:27:48 UTC 
So the actual crash is within SoftHSM2 when it is loaded as a PKCS#11 module (as part of processing of slapd_nss_init()) and is later discharged. Because softhsm internally uses static objects to represent mutex-based locks, C++ constructor should be kicked in at dlopen() and C++ destructor is kicked at dlclose().


During OSSLCryptoFactory's destructor, softhsm recycles mutexes that were created during the module load:

        for (unsigned i = 0; i < nlocks; i++)
        {
                MutexFactory::i()->recycleMutex(locks[i]);
        }
        delete[] locks;

recycleMutex() is simple:

        if (mutex != NULL) delete mutex;

The mutex destructor is also simple:
        if (isValid)
        {
                MutexFactory::i()->DestroyMutex(handle);
        }

where DestroyMutex() calls OS-specific mutex handler if mutex support is enabled. In case of Linux it is pthread_mutex_destroy(mutex) and then free(mutex) because this memory was allocated with malloc() when mutex was initialized.

When I reproduced it locally, I can see that we somehow got called twice in the destructor:

Thread 1 (Thread 0x7ffff6a69d80 (LWP 16374)):
#0  0x00007ffff3c2e680 in None ()
#1  0x00007ffff39861d5 in Mutex::~Mutex (this=0x555555889270, __in_chrg=<optimized out>) at MutexFactory.cpp:52
#2  0x00007ffff39861d5 in Mutex::~Mutex (this=0x555555889270, __in_chrg=<optimized out>) at MutexFactory.cpp:56
#3  0x00007ffff398cf8d in OSSLCryptoFactory::~OSSLCryptoFactory (this=<optimized out>, __in_chrg=<optimized out>) at OSSLCryptoFactory.cpp:222
        i = 0
#4  0x00007ffff398cfcd in OSSLCryptoFactory::~OSSLCryptoFactory (this=0x555555889230, __in_chrg=<optimized out>) at OSSLCryptoFactory.cpp:200
#5  0x00007ffff77bbe67 in __cxa_finalize () at /lib64/libc.so.6
#6  0x00007ffff3946a87 in __do_global_dtors_aux () at SoftHSM.cpp:10976
#7  0x00007fffffffdcb0 in None ()
#8  0x00007ffff7fe88ed in _dl_close_worker () at /lib64/ld-linux-x86-64.so.2
(gdb) print *(Mutex*)0x555555889270
$1 = {_vptr.Mutex = 0x7ffff3be2240 <vtable for Mutex+16>, handle = 0x555555889290, isValid = true}
(gdb) print *(pthread_mutex_t*)0x555555889290
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 1, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
  __size = '\000' <repeats 16 times>, "\001", '\000' <repeats 22 times>, __align = 0}


Nikos, do you have any ideas why this destructor may crash?
Comment 4 Nikos Mavrogiannopoulos 2018-07-24 10:56:56 UTC 
Deja vu (added other bug). If you can replicate this backtrace could you do when the app is compiled with libasan or (easier) under valgrind? If it is a double free we could "easily" address that with a fix to softhsm. There is also the possibility of the initialization happening with the glibc pthread stubs while the deinitialization with the real pthread functions. In either case we should move that to softhsm.
Comment 5 Alexander Bokovoy 2018-07-24 14:00:34 UTC 
Nikos, I rebuilt 389-ds-base and softhsm with asan but unfortunately, I do not get any reasonable output. The only thing I get is this:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==19906==ERROR: AddressSanitizer: SEGV on unknown address 0x7f64e3171680 (pc 0x7f64e3171680 bp 0x7f64e3124110 sp 0x7ffc6aa16df8 T0)
==19906==The signal is caused by a READ memory access.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

as result, there is no coredump at all.

The report is reproducible with re-runs of ns-slapd too:
# ns-slapd ldif2db -D /etc/dirsrv/slapd-RAWHIDE-VDA-LI -n userroot -i /var/lib/dirsrv/boot.ldif 
[24/Jul/2018:13:57:04.345397209 +0000] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[24/Jul/2018:13:57:04.400488288 +0000] - INFO - dblayer_instance_start - Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[24/Jul/2018:13:57:04.406659729 +0000] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 1154424832, process usage 81641472 
[24/Jul/2018:13:57:04.411744194 +0000] - INFO - check_and_set_import_cache - Import allocates 450947KB import cache.
[24/Jul/2018:13:57:04.477867116 +0000] - INFO - import_main_offline - import userRoot: Beginning import job...
[24/Jul/2018:13:57:04.484277573 +0000] - INFO - import_main_offline - import userRoot: Index buffering enabled with bucket size 100
[24/Jul/2018:13:57:04.686274612 +0000] - INFO - import_producer - import userRoot: Processing file "/var/lib/dirsrv/boot.ldif"
[24/Jul/2018:13:57:04.692068745 +0000] - INFO - import_producer - import userRoot: Finished scanning file "/var/lib/dirsrv/boot.ldif" (1 entries)
[24/Jul/2018:13:57:05.150203427 +0000] - INFO - import_monitor_threads - import userRoot: Workers finished; cleaning up...
[24/Jul/2018:13:57:05.354830511 +0000] - INFO - import_monitor_threads - import userRoot: Workers cleaned up.
[24/Jul/2018:13:57:05.358724536 +0000] - INFO - import_main_offline - import userRoot: Cleaning up producer thread...
[24/Jul/2018:13:57:05.362410427 +0000] - INFO - import_main_offline - import userRoot: Indexing complete.  Post-processing...
[24/Jul/2018:13:57:05.365930650 +0000] - INFO - import_main_offline - import userRoot: Generating numsubordinates (this may take several minutes to complete)...
[24/Jul/2018:13:57:05.374086900 +0000] - INFO - import_main_offline - import userRoot: Generating numSubordinates complete.
[24/Jul/2018:13:57:05.378344991 +0000] - INFO - ldbm_get_nonleaf_ids - import userRoot: Gathering ancestorid non-leaf IDs...
[24/Jul/2018:13:57:05.383022605 +0000] - INFO - ldbm_get_nonleaf_ids - import userRoot: Finished gathering ancestorid non-leaf IDs.
[24/Jul/2018:13:57:05.396543510 +0000] - ERR - ldbm_ancestorid_new_idl_create_index - Nothing to do to build ancestorid index
[24/Jul/2018:13:57:05.401228460 +0000] - INFO - ldbm_ancestorid_new_idl_create_index - import userRoot: Created ancestorid index (new idl).
[24/Jul/2018:13:57:05.405959856 +0000] - INFO - import_main_offline - import userRoot: Flushing caches...
[24/Jul/2018:13:57:05.409773423 +0000] - INFO - import_main_offline - import userRoot: Closing files...
[24/Jul/2018:13:57:05.437290858 +0000] - INFO - dblayer_pre_close - All database threads now stopped
[24/Jul/2018:13:57:05.441592570 +0000] - INFO - import_main_offline - import userRoot: Import complete.  Processed 1 entries in 1 seconds. (1.00 entries/sec)
AddressSanitizer:DEADLYSIGNAL
=================================================================
==20204==ERROR: AddressSanitizer: SEGV on unknown address 0x7ff4f9e4e680 (pc 0x7ff4f9e4e680 bp 0x7ff4f9e01110 sp 0x7ffc488693f8 T0)
==20204==The signal is caused by a READ memory access.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

I cannot now debug in gdb because gdb crashes when loading asan-enabled ns-slapd. ;)

If you want to reproduce it, I have scratch builds of 389-ds-base and softhsm with asan:
 - https://koji.fedoraproject.org/koji/taskinfo?taskID=28562719 is 389-ds-base
 - https://koji.fedoraproject.org/koji/taskinfo?taskID=28563263 is softhsm
Comment 6 Nikos Mavrogiannopoulos 2018-07-24 14:05:33 UTC 
What about valgrind? Any useful info with that (you'd need to recompile without asan)
Comment 7 Alexander Bokovoy 2018-07-24 14:34:01 UTC 
Valgrind helps but not much as it points to the same place within the Mutex() destructor:

==13553== Jump to the invalid address stated on the next line
==13553==    at 0xAAA6680: ???
==13553==    by 0xAC0A804: UnknownInlinedFun (MutexFactory.cpp:56)
==13553==    by 0xAC0A804: Mutex::~Mutex() (MutexFactory.cpp:56)
==13553==    by 0xAC115BC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:222)
==13553==    by 0xAC115FC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:225)
==13553==    by 0x4F05E66: __cxa_finalize (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0xABCB0A6: ??? (in /usr/lib64/pkcs11/libsofthsm2.so)
==13553==    by 0x40148EC: _dl_close_worker (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4014FA1: _dl_close (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x5004676: _dl_catch_exception (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x5004712: _dl_catch_error (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x4D3A9D8: _dlerror_run (in /usr/lib64/libdl-2.27.9000.so)
==13553==    by 0x4D3A3A7: dlclose (in /usr/lib64/libdl-2.27.9000.so)
==13553==  Address 0xaaa6680 is not stack'd, malloc'd or (recently) free'd
==13553== 
==13553== 
==13553== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==13553==  Access not within mapped region at address 0xAAA6680
==13553==    at 0xAAA6680: ???
==13553==    by 0xAC0A804: UnknownInlinedFun (MutexFactory.cpp:56)
==13553==    by 0xAC0A804: Mutex::~Mutex() (MutexFactory.cpp:56)
==13553==    by 0xAC115BC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:222)
==13553==    by 0xAC115FC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:225)
==13553==    by 0x4F05E66: __cxa_finalize (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0xABCB0A6: ??? (in /usr/lib64/pkcs11/libsofthsm2.so)
==13553==    by 0x40148EC: _dl_close_worker (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4014FA1: _dl_close (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x5004676: _dl_catch_exception (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x5004712: _dl_catch_error (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x4D3A9D8: _dlerror_run (in /usr/lib64/libdl-2.27.9000.so)
==13553==    by 0x4D3A3A7: dlclose (in /usr/lib64/libdl-2.27.9000.so)
==13553==  If you believe this happened as a result of a stack
==13553==  overflow in your program's main thread (unlikely but
==13553==  possible), you can try to increase the size of the
==13553==  main thread stack using the --main-stacksize= flag.
==13553==  The main thread stack size used in this run was 8388608.



# valgrind --leak-check=full --track-origins=yes  --trace-children=yes ./ns-slapd ldif2db -D /etc/dirsrv/slapd-RAWHIDE-VDA-LI -n userroot -i /var/lib/dirsrv/boot.ldif
==13553== Memcheck, a memory error detector
==13553== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==13553== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==13553== Command: ./ns-slapd ldif2db -D /etc/dirsrv/slapd-RAWHIDE-VDA-LI -n userroot -i /var/lib/dirsrv/boot.ldif
==13553== 
[24/Jul/2018:14:31:07.501360235 +0000] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[24/Jul/2018:14:31:08.340493416 +0000] - INFO - dblayer_instance_start - Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[24/Jul/2018:14:31:08.354434264 +0000] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 929873920, process usage 313126912 
[24/Jul/2018:14:31:08.360860219 +0000] - INFO - check_and_set_import_cache - Import allocates 363232KB import cache.
[24/Jul/2018:14:31:09.561244172 +0000] - INFO - import_main_offline - import userRoot: Beginning import job...
[24/Jul/2018:14:31:09.802942300 +0000] - INFO - import_main_offline - import userRoot: Index buffering enabled with bucket size 100
[24/Jul/2018:14:31:14.342338111 +0000] - INFO - import_producer - import userRoot: Processing file "/var/lib/dirsrv/boot.ldif"
[24/Jul/2018:14:31:14.388393814 +0000] - INFO - import_producer - import userRoot: Finished scanning file "/var/lib/dirsrv/boot.ldif" (1 entries)
[24/Jul/2018:14:31:14.834134427 +0000] - INFO - import_monitor_threads - import userRoot: Workers finished; cleaning up...
[24/Jul/2018:14:31:15.692826382 +0000] - INFO - import_monitor_threads - import userRoot: Workers cleaned up.
[24/Jul/2018:14:31:15.700811588 +0000] - INFO - import_main_offline - import userRoot: Cleaning up producer thread...
[24/Jul/2018:14:31:15.708651464 +0000] - INFO - import_main_offline - import userRoot: Indexing complete.  Post-processing...
[24/Jul/2018:14:31:15.714598023 +0000] - INFO - import_main_offline - import userRoot: Generating numsubordinates (this may take several minutes to complete)...
[24/Jul/2018:14:31:15.744416137 +0000] - INFO - import_main_offline - import userRoot: Generating numSubordinates complete.
[24/Jul/2018:14:31:15.769885338 +0000] - INFO - ldbm_get_nonleaf_ids - import userRoot: Gathering ancestorid non-leaf IDs...
[24/Jul/2018:14:31:15.776064760 +0000] - INFO - ldbm_get_nonleaf_ids - import userRoot: Finished gathering ancestorid non-leaf IDs.
[24/Jul/2018:14:31:15.800247789 +0000] - ERR - ldbm_ancestorid_new_idl_create_index - Nothing to do to build ancestorid index
[24/Jul/2018:14:31:15.809940910 +0000] - INFO - ldbm_ancestorid_new_idl_create_index - import userRoot: Created ancestorid index (new idl).
[24/Jul/2018:14:31:15.820298269 +0000] - INFO - import_main_offline - import userRoot: Flushing caches...
[24/Jul/2018:14:31:15.828432536 +0000] - INFO - import_main_offline - import userRoot: Closing files...
==13553== Syscall param pwrite64(buf) points to uninitialised byte(s)
==13553==    at 0x4DBF0C7: pwrite (in /usr/lib64/libpthread-2.27.9000.so)
==13553==    by 0x88F4F04: __os_io (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x88E0732: ??? (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x88E0ACB: __memp_bhwrite (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x88F0C36: __memp_sync_int (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x88872CE: __db_sync (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x8884887: __db_refresh (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x8884E99: __db_close (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x8896B3C: __db_close_pp (in /usr/lib64/libdb-5.3.so)
==13553==    by 0xA3E2EC5: dblayer_close_indexes (dblayer.c:2338)
==13553==    by 0xA3E3006: dblayer_instance_close (dblayer.c:2391)
==13553==    by 0xA3F7F7E: import_main_offline (import.c:1500)
==13553==  Address 0x1566ce83 is 8,259 bytes inside a block of size 8,288 alloc'd
==13553==    at 0x483678B: malloc (vg_replace_malloc.c:299)
==13553==    by 0x88F20F4: __os_malloc (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x88BC693: __env_alloc (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x88DE34A: __memp_alloc (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x88E32E4: __memp_fget (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x87E319D: __bam_get_root (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x87E39DB: __bam_search (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x87CF13A: ??? (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x87D3554: ??? (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x888B800: __dbc_iput (in /usr/lib64/libdb-5.3.so)
==13553==    by 0x889A25A: __dbc_put_pp (in /usr/lib64/libdb-5.3.so)
==13553==    by 0xA4191EA: _entryrdn_put_data (ldbm_entryrdn.c:1910)
==13553==  Uninitialised value was created by a heap allocation
==13553==    at 0x483678B: malloc (vg_replace_malloc.c:299)
==13553==    by 0x48A39A6: slapi_ch_malloc (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0xA419527: _entryrdn_new_rdn_elem (ldbm_entryrdn.c:1528)
==13553==    by 0xA41BE85: _entryrdn_insert_key (ldbm_entryrdn.c:2403)
==13553==    by 0xA41DF4A: entryrdn_index_entry (ldbm_entryrdn.c:269)
==13553==    by 0xA3FC24D: foreman_do_entryrdn.isra.3 (import-threads.c:2442)
==13553==    by 0xA400850: import_foreman (import-threads.c:2590)
==13553==    by 0x4D74AC7: ??? (in /usr/lib64/libnspr4.so)
==13553==    by 0x4DB4FDB: start_thread (in /usr/lib64/libpthread-2.27.9000.so)
==13553==    by 0x4FC7CBE: clone (in /usr/lib64/libc-2.27.9000.so)
==13553== 
[24/Jul/2018:14:31:16.343616639 +0000] - INFO - dblayer_pre_close - All database threads now stopped
[24/Jul/2018:14:31:16.400965993 +0000] - INFO - import_main_offline - import userRoot: Import complete.  Processed 1 entries in 7 seconds. (0.14 entries/sec)
==13553== Jump to the invalid address stated on the next line
==13553==    at 0xAAA6680: ???
==13553==    by 0xAC0A804: UnknownInlinedFun (MutexFactory.cpp:56)
==13553==    by 0xAC0A804: Mutex::~Mutex() (MutexFactory.cpp:56)
==13553==    by 0xAC115BC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:222)
==13553==    by 0xAC115FC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:225)
==13553==    by 0x4F05E66: __cxa_finalize (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0xABCB0A6: ??? (in /usr/lib64/pkcs11/libsofthsm2.so)
==13553==    by 0x40148EC: _dl_close_worker (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4014FA1: _dl_close (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x5004676: _dl_catch_exception (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x5004712: _dl_catch_error (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x4D3A9D8: _dlerror_run (in /usr/lib64/libdl-2.27.9000.so)
==13553==    by 0x4D3A3A7: dlclose (in /usr/lib64/libdl-2.27.9000.so)
==13553==  Address 0xaaa6680 is not stack'd, malloc'd or (recently) free'd
==13553== 
==13553== 
==13553== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==13553==  Access not within mapped region at address 0xAAA6680
==13553==    at 0xAAA6680: ???
==13553==    by 0xAC0A804: UnknownInlinedFun (MutexFactory.cpp:56)
==13553==    by 0xAC0A804: Mutex::~Mutex() (MutexFactory.cpp:56)
==13553==    by 0xAC115BC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:222)
==13553==    by 0xAC115FC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:225)
==13553==    by 0x4F05E66: __cxa_finalize (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0xABCB0A6: ??? (in /usr/lib64/pkcs11/libsofthsm2.so)
==13553==    by 0x40148EC: _dl_close_worker (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4014FA1: _dl_close (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x5004676: _dl_catch_exception (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x5004712: _dl_catch_error (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x4D3A9D8: _dlerror_run (in /usr/lib64/libdl-2.27.9000.so)
==13553==    by 0x4D3A3A7: dlclose (in /usr/lib64/libdl-2.27.9000.so)
==13553==  If you believe this happened as a result of a stack
==13553==  overflow in your program's main thread (unlikely but
==13553==  possible), you can try to increase the size of the
==13553==  main thread stack using the --main-stacksize= flag.
==13553==  The main thread stack size used in this run was 8388608.
==13553== 
==13553== HEAP SUMMARY:
==13553==     in use at exit: 2,627,160 bytes in 38,237 blocks
==13553==   total heap usage: 258,113 allocs, 219,876 frees, 57,615,462 bytes allocated
==13553== 
==13553== 72 bytes in 1 blocks are possibly lost in loss record 1,828 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x57D6598: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x57D66DF: PORT_NewArena_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5A9F1: secmod_NewModule (pk11pars.c:32)
==13553==    by 0x4C5A9F1: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C275F2: nss_InitModules (nssinit.c:464)
==13553==    by 0x4C275F2: nss_Init (nssinit.c:689)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 72 bytes in 1 blocks are possibly lost in loss record 1,829 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x57D6598: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x57D66DF: PORT_NewArena_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5A9F1: secmod_NewModule (pk11pars.c:32)
==13553==    by 0x4C5A9F1: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 72 bytes in 1 blocks are possibly lost in loss record 1,830 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x57D6598: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x57D66DF: PORT_NewArena_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5A9F1: secmod_NewModule (pk11pars.c:32)
==13553==    by 0x4C5A9F1: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C5C667: SECMOD_LoadModule (pk11pars.c:1862)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 128 bytes in 2 blocks are definitely lost in loss record 2,026 of 2,516
==13553==    at 0x483678B: malloc (vg_replace_malloc.c:299)
==13553==    by 0x4013CED: dl_open_worker (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x5004676: _dl_catch_exception (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x401319D: _dl_open (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4D3A279: dlopen_doit (in /usr/lib64/libdl-2.27.9000.so)
==13553==    by 0x5004676: _dl_catch_exception (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x5004712: _dl_catch_error (in /usr/lib64/libc-2.27.9000.so)
==13553==    by 0x4D3A9D8: _dlerror_run (in /usr/lib64/libdl-2.27.9000.so)
==13553==    by 0x4D3A319: dlopen@@GLIBC_2.2.5 (in /usr/lib64/libdl-2.27.9000.so)
==13553==    by 0x4D5F82A: PR_LoadLibraryWithFlags (in /usr/lib64/libnspr4.so)
==13553==    by 0x48B3269: sym_load_with_flags (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x48FC27D: plugin_setup (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553== 
==13553== 168 bytes in 1 blocks are possibly lost in loss record 2,137 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4D6E9E0: PR_NewLock (in /usr/lib64/libnspr4.so)
==13553==    by 0x57D66F3: PORT_NewArena_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5A9F1: secmod_NewModule (pk11pars.c:32)
==13553==    by 0x4C5A9F1: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C275F2: nss_InitModules (nssinit.c:464)
==13553==    by 0x4C275F2: nss_Init (nssinit.c:689)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 168 bytes in 1 blocks are possibly lost in loss record 2,138 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4D6E9E0: PR_NewLock (in /usr/lib64/libnspr4.so)
==13553==    by 0x4C5AACD: secmod_NewModule (pk11pars.c:70)
==13553==    by 0x4C5AACD: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C275F2: nss_InitModules (nssinit.c:464)
==13553==    by 0x4C275F2: nss_Init (nssinit.c:689)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 168 bytes in 1 blocks are possibly lost in loss record 2,139 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4D6E9E0: PR_NewLock (in /usr/lib64/libnspr4.so)
==13553==    by 0x57D66F3: PORT_NewArena_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5A9F1: secmod_NewModule (pk11pars.c:32)
==13553==    by 0x4C5A9F1: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 168 bytes in 1 blocks are possibly lost in loss record 2,140 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4D6E9E0: PR_NewLock (in /usr/lib64/libnspr4.so)
==13553==    by 0x4C5AACD: secmod_NewModule (pk11pars.c:70)
==13553==    by 0x4C5AACD: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 168 bytes in 1 blocks are possibly lost in loss record 2,141 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4D6E9E0: PR_NewLock (in /usr/lib64/libnspr4.so)
==13553==    by 0x57D66F3: PORT_NewArena_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5A9F1: secmod_NewModule (pk11pars.c:32)
==13553==    by 0x4C5A9F1: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C5C667: SECMOD_LoadModule (pk11pars.c:1862)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 168 bytes in 1 blocks are possibly lost in loss record 2,142 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4D6E9E0: PR_NewLock (in /usr/lib64/libnspr4.so)
==13553==    by 0x4C5AACD: secmod_NewModule (pk11pars.c:70)
==13553==    by 0x4C5AACD: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C5C667: SECMOD_LoadModule (pk11pars.c:1862)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 400 bytes in 1 blocks are possibly lost in loss record 2,245 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4011CB1: allocate_dtv (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4012651: _dl_allocate_tls (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4DB5BE5: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.27.9000.so)
==13553==    by 0x4847E61: ns_thrpool_new (in /usr/lib64/dirsrv/libnunc-stans.so.0.1.0)
==13553==    by 0x11926C: main_create_ns (main.c:223)
==13553==    by 0x11926C: main (main.c:930)
==13553== 
==13553== 512 bytes in 1 blocks are possibly lost in loss record 2,265 of 2,516
==13553==    at 0x483678B: malloc (vg_replace_malloc.c:299)
==13553==    by 0x4D46306: PL_ArenaAllocate (in /usr/lib64/libplds4.so)
==13553==    by 0x57D6889: PORT_ArenaAlloc_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5AA06: secmod_NewModule (pk11pars.c:37)
==13553==    by 0x4C5AA06: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C275F2: nss_InitModules (nssinit.c:464)
==13553==    by 0x4C275F2: nss_Init (nssinit.c:689)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 512 bytes in 1 blocks are possibly lost in loss record 2,266 of 2,516
==13553==    at 0x483678B: malloc (vg_replace_malloc.c:299)
==13553==    by 0x4D46306: PL_ArenaAllocate (in /usr/lib64/libplds4.so)
==13553==    by 0x57D6889: PORT_ArenaAlloc_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x57D6BF6: PORT_ArenaStrdup_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5AB1B: SECMOD_CreateModuleEx (pk11pars.c:805)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C275F2: nss_InitModules (nssinit.c:464)
==13553==    by 0x4C275F2: nss_Init (nssinit.c:689)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 512 bytes in 1 blocks are possibly lost in loss record 2,267 of 2,516
==13553==    at 0x483678B: malloc (vg_replace_malloc.c:299)
==13553==    by 0x4D46306: PL_ArenaAllocate (in /usr/lib64/libplds4.so)
==13553==    by 0x57D6889: PORT_ArenaAlloc_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5AA06: secmod_NewModule (pk11pars.c:37)
==13553==    by 0x4C5AA06: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 512 bytes in 1 blocks are possibly lost in loss record 2,268 of 2,516
==13553==    at 0x483678B: malloc (vg_replace_malloc.c:299)
==13553==    by 0x4D46306: PL_ArenaAllocate (in /usr/lib64/libplds4.so)
==13553==    by 0x57D6889: PORT_ArenaAlloc_Util (in /usr/lib64/libnssutil3.so)
==13553==    by 0x4C5AA06: secmod_NewModule (pk11pars.c:37)
==13553==    by 0x4C5AA06: SECMOD_CreateModuleEx (pk11pars.c:795)
==13553==    by 0x4C5C46C: SECMOD_LoadModule (pk11pars.c:1797)
==13553==    by 0x4C5C667: SECMOD_LoadModule (pk11pars.c:1862)
==13553==    by 0x4C277DC: nss_Init (nssinit.c:712)
==13553==    by 0x4C27CA5: NSS_Initialize (nssinit.c:889)
==13553==    by 0x4922337: slapd_nss_init (in /usr/lib64/dirsrv/libslapd.so.0.1.0)
==13553==    by 0x12F249: slapd_do_all_nss_ssl_init (main.c:2986)
==13553==    by 0x12A464: detach (detach.c:177)
==13553==    by 0x118E0E: main (main.c:922)
==13553== 
==13553== 6,400 bytes in 16 blocks are possibly lost in loss record 2,450 of 2,516
==13553==    at 0x48386E6: calloc (vg_replace_malloc.c:711)
==13553==    by 0x4011CB1: allocate_dtv (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4012651: _dl_allocate_tls (in /usr/lib64/ld-2.27.9000.so)
==13553==    by 0x4DB5BE5: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.27.9000.so)
==13553==    by 0x4847E2A: ns_thrpool_new (in /usr/lib64/dirsrv/libnunc-stans.so.0.1.0)
==13553==    by 0x11926C: main_create_ns (main.c:223)
==13553==    by 0x11926C: main (main.c:930)
==13553== 
==13553== LEAK SUMMARY:
==13553==    definitely lost: 128 bytes in 2 blocks
==13553==    indirectly lost: 0 bytes in 0 blocks
==13553==      possibly lost: 10,072 bytes in 30 blocks
==13553==    still reachable: 2,616,960 bytes in 38,205 blocks
==13553==         suppressed: 0 bytes in 0 blocks
==13553== Reachable blocks (those to which a pointer was found) are not shown.
==13553== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==13553== 
==13553== For counts of detected and suppressed errors, rerun with: -v
==13553== ERROR SUMMARY: 18 errors from 18 contexts (suppressed: 0 from 0)
==13553== could not unlink /tmp/vgdb-pipe-from-vgdb-to-13553-by-root-on-ho-2.rawhide.vda.li
==13553== could not unlink /tmp/vgdb-pipe-to-vgdb-from-13553-by-root-on-ho-2.rawhide.vda.li
==13553== could not unlink /tmp/vgdb-pipe-shared-mem-vgdb-13553-by-root-on-ho-2.rawhide.vda.li
Segmentation fault
[root@ho-2 ~]#
Comment 8 Alexander Bokovoy 2018-07-24 14:47:07 UTC 
I think we have to treat this as a blocker as it doesn't allow FreeIPA DC to be installed at all in rawhide.
Comment 9 Alexander Bokovoy 2018-07-24 15:33:08 UTC 
Talking to lev@freebsd, he suggested that it might be an issue with a corrupted stack/heap by soemthing else by the time we see a call to OSSLCrfyptoFactory::~OSSCryptoFactory() destructor. 

Moving to softhsm as that is the place where a bug is really happening.
Comment 10 Alexander Bokovoy 2018-07-24 15:46:55 UTC 
Another datapoint is that softhsm is built with gcc 8.0 in F28 and gcc 8.1.1 in rawhide. While gcc 8.1.1 is in F28 now, softhsm wasn't rebuilt since F28 mass rebuild in February 2018.

From koji: https://koji.fedoraproject.org/koji/packageinfo?packageID=12764

softhsm-2.3.0-3.fc29.2 	releng 	2018-07-15 20:21:38 	complete
softhsm-2.3.0-3.fc28.1 	releng 	2018-02-11 00:23:47 	complete
Comment 11 Adam Williamson 2018-07-24 16:10:36 UTC 
Yes, this is obviously a Beta blocker, as a violation of https://fedoraproject.org/wiki/Basic_Release_Criteria#FreeIPA_server_requirements - "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages."

A data point that ties in well with ab's is that this was working as recently as 20180710.n.0. On 20180711.n.0 deployment was broken by a different bug (firewalld), then we didn't get another compose till 20180719.n.0. So that softhsm build definitely falls in the window where this bug showed up.
Comment 12 Adam Williamson 2018-07-24 19:13:30 UTC 
There were a few compiler warnings during the build, not sure if any would be relevant, but just in case, here they are:

SymmetricAlgorithm.cpp: In member function 'virtual bool SymmetricAlgorithm::isStreamCipher()':
SymmetricAlgorithm.cpp:190:9: warning: enumeration value 'Unknown' not handled in switch [-Wswitch]
  switch (currentCipherMode)
         ^
SymmetricAlgorithm.cpp:190:9: warning: enumeration value 'CBC' not handled in switch [-Wswitch]
SymmetricAlgorithm.cpp:190:9: warning: enumeration value 'ECB' not handled in switch [-Wswitch]
SymmetricAlgorithm.cpp: In member function 'virtual bool SymmetricAlgorithm::isBlockCipher()':
SymmetricAlgorithm.cpp:203:9: warning: enumeration value 'Unknown' not handled in switch [-Wswitch]
  switch (currentCipherMode)
         ^
SymmetricAlgorithm.cpp:203:9: warning: enumeration value 'CFB' not handled in switch [-Wswitch]
SymmetricAlgorithm.cpp:203:9: warning: enumeration value 'CTR' not handled in switch [-Wswitch]
SymmetricAlgorithm.cpp:203:9: warning: enumeration value 'OFB' not handled in switch [-Wswitch]

...

File.cpp: In member function 'bool File::writeString(const string&)':
File.cpp:421:54: warning: type qualifiers ignored on cast result type [-Wignored-qualifiers]
  ByteString toWrite((const unsigned long) value.size());
                                                      ^

...

base64.c:46:19: warning: 'rcsid' defined but not used [-Wunused-const-variable=]
 static const char rcsid[] = "$ISC: base64.c,v 8.6 1999/01/08 19:25:18 vixie Exp $";
                   ^~~~~
Comment 13 Nikos Mavrogiannopoulos 2018-07-25 08:05:00 UTC 
*** Bug 1570306 has been marked as a duplicate of this bug. ***
Comment 14 Lukas Slebodnik 2018-07-26 17:51:24 UTC 
(In reply to Nikos Mavrogiannopoulos from comment #6)
> What about valgrind? Any useful info with that (you'd need to recompile
> without asan)

Nikos here is a super trivial reproducer on rawhide.

sh-4.4$ rm -rf p11_nssdb/
sh-4.4$ mkdir p11_nssdb
sh-4.4$ /usr/bin/certutil -d sql:./p11_nssdb -N --empty-password
Segmentation fault (core dumped)

sh-4.4$ rm -rf p11_nssdb/
sh-4.4$ mkdir p11_nssdb
sh-4.4$ valgrind /usr/bin/certutil -d sql:./p11_nssdb -N --empty-password
==18432== Memcheck, a memory error detector
==18432== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==18432== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==18432== Command: /usr/bin/certutil -d sql:./p11_nssdb -N --empty-password
==18432== 
==18432== Jump to the invalid address stated on the next line
==18432==    at 0x5437680: ???
==18432==    by 0x559B804: UnknownInlinedFun (MutexFactory.cpp:56)
==18432==    by 0x559B804: Mutex::~Mutex() (MutexFactory.cpp:56)
==18432==    by 0x55A25BC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:222)
==18432==    by 0x55A25FC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:225)
==18432==    by 0x4B3092B: __run_exit_handlers (in /usr/lib64/libc-2.27.9000.so)
==18432==    by 0x4B30A5F: exit (in /usr/lib64/libc-2.27.9000.so)
==18432==    by 0x4B1A0E9: (below main) (in /usr/lib64/libc-2.27.9000.so)
==18432==  Address 0x5437680 is not stack'd, malloc'd or (recently) free'd
==18432== 
==18432== 
==18432== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==18432==  Access not within mapped region at address 0x5437680
==18432==    at 0x5437680: ???
==18432==    by 0x559B804: UnknownInlinedFun (MutexFactory.cpp:56)
==18432==    by 0x559B804: Mutex::~Mutex() (MutexFactory.cpp:56)
==18432==    by 0x55A25BC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:222)
==18432==    by 0x55A25FC: OSSLCryptoFactory::~OSSLCryptoFactory() (OSSLCryptoFactory.cpp:225)
==18432==    by 0x4B3092B: __run_exit_handlers (in /usr/lib64/libc-2.27.9000.so)
==18432==    by 0x4B30A5F: exit (in /usr/lib64/libc-2.27.9000.so)
==18432==    by 0x4B1A0E9: (below main) (in /usr/lib64/libc-2.27.9000.so)
==18432==  If you believe this happened as a result of a stack
==18432==  overflow in your program's main thread (unlikely but
==18432==  possible), you can try to increase the size of the
==18432==  main thread stack using the --main-stacksize= flag.
==18432==  The main thread stack size used in this run was 8388608.
==18432== 
==18432== HEAP SUMMARY:
==18432==     in use at exit: 86,045 bytes in 39 blocks
==18432==   total heap usage: 10,804 allocs, 10,765 frees, 2,647,294 bytes allocated
==18432== 
==18432== LEAK SUMMARY:
==18432==    definitely lost: 40 bytes in 1 blocks
==18432==    indirectly lost: 72 bytes in 3 blocks
==18432==      possibly lost: 0 bytes in 0 blocks
==18432==    still reachable: 85,933 bytes in 35 blocks
==18432==         suppressed: 0 bytes in 0 blocks
==18432== Rerun with --leak-check=full to see details of leaked memory
==18432== 
==18432== For counts of detected and suppressed errors, rerun with: -v
==18432== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)
Comment 15 Lukas Slebodnik 2018-07-26 17:52:25 UTC 
sh-4.4$ rpm -qf /usr/bin/certutil /usr/lib64/pkcs11/libsofthsm2.so
nss-tools-3.38.0-4.fc29.x86_64
softhsm-2.3.0-3.fc29.2.x86_64
Comment 16 Lukas Slebodnik 2018-07-26 19:25:04 UTC 
https://github.com/opendnssec/SoftHSMv2/issues/408
Comment 17 Alexander Bokovoy 2018-07-31 12:05:12 UTC 
I submitted a pull request upstream that should fix the problem: https://github.com/opendnssec/SoftHSMv2/pull/409
Comment 18 Alexander Bokovoy 2018-07-31 12:41:58 UTC 
A scratch build for rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=28735420
Comment 19 Alexander Bokovoy 2018-07-31 13:24:12 UTC 
A test deployment of FreeIPA 4.7.0 with the scratch build on F29 succeeded (almost, there is unrelated issue with kdcproxy startup). There is a number of AVCs for certmonger. I also had to remove softhsm token from older broken attempts (in /var/lib/softhsm/tokens/*) before installing.
Comment 20 Francisco de la Peña 2018-07-31 16:22:12 UTC 
I confirm the softhsm scratch build fixes the test case in bug 1570306.
Comment 21 Adam Williamson 2018-07-31 20:18:10 UTC 
Crash fix also confirmed in openQA, like ab I found the deployment still runs into problems later on, though - https://openqa.stg.fedoraproject.org/tests/335261# . I'll file bugs for the other issues.

ab, can we get an official build with the fix? Thanks!
Comment 22 Alexander Bokovoy 2018-08-01 07:16:20 UTC 
(In reply to Adam Williamson from comment #21)
> Crash fix also confirmed in openQA, like ab I found the deployment still
> runs into problems later on, though -
> https://openqa.stg.fedoraproject.org/tests/335261# . I'll file bugs for the
> other issues.
> 
> ab, can we get an official build with the fix? Thanks!

I cannot do any official build for softhsm package as I'm not a maintainer there. I did submit a pull request https://src.fedoraproject.org/rpms/softhsm/pull-request/3 yesterday and asked Nikos to merge it. Hopefully, that happens today.
Comment 23 Nikos Mavrogiannopoulos 2018-08-01 08:45:26 UTC 
Thanks. I sent a build for rawhide and will backport for f28 if easy.
Comment 24 Fedora Update System 2018-08-01 09:20:55 UTC 
softhsm-2.3.0-4.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-f28f355bd8
Comment 25 Fedora Update System 2018-08-01 18:26:05 UTC 
softhsm-2.3.0-4.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-f28f355bd8
Comment 26 Adam Williamson 2018-08-01 18:32:45 UTC 
As the bug is reported against Rawhide we should just close it, it shouldn't be associated with the update. I've edited the update to uncheck the 'close bug when update is stable' box, and will just close this now.
Comment 27 Fedora Update System 2018-08-09 17:40:55 UTC 
softhsm-2.3.0-4.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.