Created attachment 1252049 [details]
build.log

Created attachment 1252050 [details]
root.log

Created attachment 1252051 [details]
state.log

The problem appears to be the use of CRYPTO_malloc_init in connsec.c.

connsec.c: In function 'nsdb_connsec_crypto_startup':
connsec.c:47:2: warning: implicit declaration of function 'CRYPTO_malloc_init'; did you mean 'CRYPTO_malloc'? [-Wimplicit-function-declaration]
  CRYPTO_malloc_init();
  ^~~~~~~~~~~~~~~~~~
  CRYPTO_malloc
connsec.c: In function 'nsdb_connsec_crypto_shutdown':
connsec.c:65:2: warning: 'ERR_remove_thread_state' is deprecated [-Wdeprecated-declarations]
  ERR_remove_thread_state(NULL);
  ^~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/opensslconf.h:27:0,
                 from /usr/include/openssl/ui.h:13,
                 from /usr/include/openssl/engine.h:29,
                 from connsec.c:30:
/usr/include/openssl/err.h:246:1: note: declared here
 DEPRECATEDIN_1_1_0(void ERR_remove_thread_state(void *))
 ^

A call to CRYPTO_malloc_init is open-coded in nsdb_connsec_crypto_startup(). It appears that OpenSSL 1.1.0 and later do not provide CRYPTO_malloc_init.

The OpenSSL FAQ states (PROG #2) that CRYPTO_malloc_init() must be invoked before the first use of the OpenSSL API. Is this call no longer necessary? Or is there another call that should be made instead? Or has that call been moved to another header/library?

I haven't found any clear documentation of this change. The common element to the advice I found was "downgrade your OpenSSL library".

This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

(In reply to Chuck Lever from comment #4)
> The problem appears to be the use of CRYPTO_malloc_init in connsec.c.

Yep, and that might be the only fedfs-utils problem with the
change from openssl-1.0 to openssl-1.1, not sure yet.

snip ...

> 
> A call to CRYPTO_malloc_init is open-coded in nsdb_connsec_crypto_startup().
> It appears that OpenSSL 1.1.0 and later do not provide CRYPTO_malloc_init.
> 
> The OpenSSL FAQ states (PROG #2) that CRYPTO_malloc_init() must be invoked
> before the first use of the OpenSSL API. Is this call no longer necessary?
> Or is there another call that should be made instead? Or has that call been
> moved to another header/library?
> 
> I haven't found any clear documentation of this change. The common element
> to the advice I found was "downgrade your OpenSSL library".

Documentation on the changes is hard to find but I saw this:
http://openssl.6102.n7.nabble.com/Cleanup-and-changing-the-malloc-routines-td59913.html

which implies that at some point (possibly at openssl-1.0)
CRYPTO_malloc_init() should be changed to OPENSSL_malloc_init().

So there's probably a need for some autoconf to obtain OpenSSL
version info. and conditional call to the appropriate function
based on the version obtained.

We could add a patch that just changes the call to get it to build
to give us time to put together a more complete fix.

But now I think I know what's going on I'll need to ask around a
bit and see if anyone knows the details about the changes.

Ian

(In reply to Ian Kent from comment #6)
> 
> which implies that at some point (possibly at openssl-1.0)

Umm ... that should be openssl-1.1.

Oops!

iankent's fedfs-utils-0.10.5-6.fc27 completed http://koji.fedoraproject.org/koji/buildinfo?buildID=871043

As far as I can tell CRYPTO_malloc_init() and ERR_remove_thread_state()
did nothing for quite a while.

CRYPTO_malloc_init() has been removed and ERR_remove_thread_state()
reamins but is deprecated.

I've added a simple patch to check the version of OpenSSL and not
call these functions if OpenSSL is at version 1.1 or above. The
fedfs-utils package has now been buildt in Rawhide.

I'll post the patch upstream for further discussion.

Ian