2196395 – GDB crashes when legacy provider settings for openssl are enabled

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 2196395 - GDB crashes when legacy provider settings for openssl are enabled

Summary: GDB crashes when legacy provider settings for openssl are enabled

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	gdb
Sub Component:
Version:	38
Hardware:	Unspecified
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Assignee:	amerey
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-05-09 00:41 UTC by Kevin Buettner
Modified:	2023-08-10 00:41 UTC (History)
CC List:	25 users (show)
Fixed In Version:	gdb-13.2-3.fc38
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-08-10 00:41:33 UTC
Type:	---
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	FC-830	0	None	None	None	2023-05-09 14:53:32 UTC

Description Kevin Buettner 2023-05-09 00:41:39 UTC

I'm seeing a GDB crash on Fedora 37, Fedora 38, and rawhide (f39).

I think that the bug is actually in one of the openssl packages.  When the default
/etc/ssl/openssl.cnf file is used, the bug does not occur.  That said, I'm going to start with the component set to "gdb" instead of "openssl".

The file /etc/ssl/openssl.cnf contains the following lines:

[provider_sect]
default = default_sect
##legacy = legacy_sect
##
[default_sect]
activate = 1

##[legacy_sect]
##activate = 1

The bug will occur when the '##' characters are removed so that the lines in question look like this:

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

With that change in place, do the following to reproduce the bug:

[kev@f38-2 e]$ rm -rf $HOME/.cache/debuginfod_client
[kev@f38-2 e]$ gdb -q /bin/ls
Reading symbols from /bin/ls...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/bin/ls
Reading symbols from /home/kev/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo...
Downloading separate debug info for /home/kev/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo
(gdb) start                                                                     
Downloading source file /usr/src/debug/coreutils-9.1-11.fc38.x86_64/separate/../src/ls.c
Temporary breakpoint 1 at 0x4e50: file ../src/ls.c, line 1646.                  
Starting program: /usr/bin/ls 
Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000
Downloading separate debug info for /lib64/libselinux.so.1                      
Downloading separate debug info for /home/kev/.cache/debuginfod_client/7adfc95d3eb203b795825f82ab41639cb0bb9c4c/debuginfo
Downloading separate debug info for /lib64/libcap.so.2                          
Downloading separate debug info for /home/kev/.cache/debuginfod_client/2fe8ebb5811811046e04d8f4c000696cb9b24e24/debuginfo
[Thread debugging using libthread_db enabled]                                   
Using host libthread_db library "/lib64/libthread_db.so.1".
Downloading separate debug info for /lib64/libpcre2-8.so.0
Downloading separate debug info for /home/kev/.cache/debuginfod_client/1ef098ba40305d86ec0cdae1e6aac188b1656550/debuginfo
                                                                                
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffda68) at ../src/ls.c:1646
1646	{
(gdb) quit
A debugging session is active.

	Inferior 1 [process 59528] will be killed.

Quit anyway? (y or n) y


Fatal signal: Segmentation fault
----- Backtrace -----
0x55a251ed9810 gdb_internal_backtrace_1
	../../gdb/bt-utils.c:122
0x55a251ed9810 _Z22gdb_internal_backtracev
	../../gdb/bt-utils.c:168
0x55a2520140dd handle_fatal_signal
	../../gdb/event-top.c:962
0x55a2520142a9 handle_sigsegv
	../../gdb/event-top.c:1035
0x7f542b23bb6f ???
	/usr/src/debug/glibc-2.37-4.fc38.x86_64/signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x7f542b2902b5 ???
	/usr/src/debug/glibc-2.37-4.fc38.x86_64/nptl/pthread_rwlock_common.c:298
0x7f54283c0f8c ???
0x7f54283b901e ???
0x7f54283c659c ???
0x7f54283b9f46 ???
0x7f54180e7bed ???
0x7f54283c69fa ???
0x7f54283904bb ???
0x7f5428b96abc ???
0x7f542af5c306 ???
	../../lib/vtls/openssl.c:1895
0x7f542af661ce ???
	../../lib/vtls/vtls.c:1423
0x7f542af02203 ???
	../../lib/cfilters.c:165
0x7f542af5ea92 ???
	../../lib/cfilters.c:186
0x7f542af0369b ???
	../../lib/conncache.c:532
0x7f542af3c9b2 ???
	../../lib/multi.c:2801
0x7f542b73c54d ???
	/usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1892
0x55a251f7d9c4 _ZN25debuginfod_client_deleterclEP17debuginfod_client
	../../gdb/debuginfod-support.c:105
0x55a251f7d9c4 _ZNSt10unique_ptrI17debuginfod_client25debuginfod_client_deleterED2Ev
	/usr/include/c++/13/bits/unique_ptr.h:404
0x7f542b23e1a5 ???
	/usr/src/debug/glibc-2.37-4.fc38.x86_64/stdlib/exit.c:111
0x7f542b23e2ed __GI_exit
	/usr/src/debug/glibc-2.37-4.fc38.x86_64/stdlib/exit.c:141
0x55a252313099 _Z10quit_forcePii
	../../gdb/top.c:1894
0x55a251f0c14b _Z12quit_commandPKci
	../../gdb/cli/cli-cmds.c:506
0x55a251f14064 _Z8cmd_funcP16cmd_list_elementPKci
	../../gdb/cli/cli-decode.c:2543
0x55a2523142a4 _Z15execute_commandPKci
	../../gdb/top.c:688
0x55a2520148de _Z15command_handlerPKc
	../../gdb/event-top.c:619
0x55a252016398 _Z20command_line_handlerOSt10unique_ptrIcN3gdb13xfree_deleterIcEEE
	../../gdb/event-top.c:855
0x55a252014f10 gdb_rl_callback_handler
	../../gdb/event-top.c:247
0x7f542bf0a0cc ???
0x55a252015075 gdb_rl_callback_read_char_wrapper_noexcept
	../../gdb/event-top.c:189
0x55a252015227 gdb_rl_callback_read_char_wrapper
	../../gdb/event-top.c:222
0x55a252013f8f stdin_event_handler
	../../gdb/event-top.c:544
0x55a25257397d gdb_wait_for_event
	../../gdbsupport/event-loop.cc:716
0x55a252574531 _Z16gdb_do_one_eventi
	../../gdbsupport/event-loop.cc:264
0x55a2520fb481 start_event_loop
	../../gdb/main.c:411
0x55a2520fb481 captured_command_loop
	../../gdb/main.c:471
0x55a2520fd5f4 captured_main
	../../gdb/main.c:1330
0x55a2520fd5f4 _Z8gdb_mainP18captured_main_args
	../../gdb/main.c:1345
0x55a251dea9ee main
	../../gdb/gdb.c:40
---------------------
A fatal error internal to GDB has been detected, further
debugging is not possible.  GDB will now terminate.

This is a bug, please report it.  For instructions, see:
<https://www.gnu.org/software/gdb/bugs/>.

Segmentation fault (core dumped)


Reproducible: Always




The backtrace for the core dump (on Fedora 38) looks like this:

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f542b28c8b3 in __pthread_kill_internal (signo=11, 
    threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f542b23babe in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#3  0x000055a25201408a in handle_fatal_signal (sig=sig@entry=11)
    at ../../gdb/event-top.c:985
#4  0x000055a2520142aa in handle_sigsegv (sig=11) at ../../gdb/event-top.c:1035
#5  <signal handler called>
#6  __pthread_rwlock_rdlock_full64 (abstime=0x0, clockid=0, rwlock=0x0)
    at /usr/src/debug/glibc-2.37-4.fc38.x86_64/nptl/pthread_rwlock_common.c:298
#7  ___pthread_rwlock_rdlock (rwlock=0x0) at pthread_rwlock_rdlock.c:26
#8  0x00007f54283c0f8d in CRYPTO_THREAD_read_lock (lock=<optimized out>)
    at crypto/threads_pthread.c:97
#9  0x00007f54283b901f in ossl_lib_ctx_get_data (ctx=<optimized out>, index=1, 
    meth=0x7f54285ddaa0 <provider_store_method.lto_priv>)
    at crypto/context.c:413
#10 0x00007f54283c659d in get_provider_store (libctx=<optimized out>)
    at crypto/provider_core.c:339
#11 ossl_provider_deregister_child_cb (handle=0x55a2542fb0a0)
    at crypto/provider_core.c:1806
#12 0x00007f54283b9f47 in ossl_provider_deinit_child (ctx=0x55a2542f91b0)
    at crypto/provider_child.c:281
#13 OSSL_LIB_CTX_free (ctx=0x55a2542f91b0) at crypto/context.c:284
#14 OSSL_LIB_CTX_free (ctx=0x55a2542f91b0) at crypto/context.c:277
#15 0x00007f54180e7bee in legacy_teardown (provctx=0x55a254301fe0)
    at providers/legacyprov.c:168
#16 0x00007f54283c69fb in ossl_provider_teardown (prov=0x55a2542fb0a0)
    at crypto/provider_core.c:1558
#17 ossl_provider_free (prov=0x55a2542fb0a0) at crypto/provider_core.c:696
#18 0x00007f54283904bc in ossl_provider_free (prov=<optimized out>)
    at crypto/provider_core.c:681
#19 evp_cipher_free_int (cipher=0x55a25433dff0) at crypto/evp/evp_enc.c:1651
#20 EVP_CIPHER_free (cipher=0x55a25433dff0) at crypto/evp/evp_enc.c:1666
#21 0x00007f5428b96abd in ssl_evp_cipher_free (cipher=0x55a25433dff0)
    at ssl/ssl_lib.c:5967
#22 ssl_evp_cipher_free (cipher=0x55a25433dff0) at ssl/ssl_lib.c:5957
#23 SSL_CTX_free (a=0x55a254347620) at ssl/ssl_lib.c:3492
#24 SSL_CTX_free (a=0x55a254347620) at ssl/ssl_lib.c:3429
#25 0x00007f542af5c307 in ossl_close (cf=<optimized out>, data=<optimized out>)
    at ../../lib/vtls/openssl.c:1895
#26 0x00007f542af661cf in cf_close (data=<optimized out>, cf=0x55a254301050)
    at ../../lib/vtls/vtls.c:1423
#27 ssl_cf_destroy (cf=0x55a254301050, data=<optimized out>)
    at ../../lib/vtls/vtls.c:1483
#28 0x00007f542af02204 in Curl_conn_cf_discard_chain (pcf=<optimized out>, 
    data=0x55a254304ab0) at ../../lib/cfilters.c:165
#29 0x00007f542af5ea93 in Curl_conn_close (index=0, data=0x55a254304ab0)
    at ../../lib/cfilters.c:186
#30 conn_shutdown (data=0x55a254304ab0) at ../../lib/url.c:692
#31 Curl_disconnect (data=0x55a254304ab0, conn=0x55a2542fcd80, 
    dead_connection=<optimized out>) at ../../lib/url.c:803
#32 0x00007f542af0369c in Curl_conncache_close_all_connections (
    connc=0x55a254300a68) at ../../lib/conncache.c:532
#33 0x00007f542af3c9b3 in curl_multi_cleanup (multi=0x55a254300960)
    at ../../lib/multi.c:2801
#34 curl_multi_cleanup (multi=0x55a254300960) at ../../lib/multi.c:2763
#35 0x00007f542b73c54e in debuginfod_end (client=0x55a254300290)
    at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1892
#36 debuginfod_end (client=0x55a254300290)
    at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1887
#37 0x000055a251f7d9c5 in debuginfod_client_deleter::operator() (
    c=<optimized out>, this=<optimized out>)
    at ../../gdb/debuginfod-support.c:105
#38 std::unique_ptr<debuginfod_client, debuginfod_client_deleter>::~unique_ptr
    (this=<optimized out>, __in_chrg=<optimized out>)
    at /usr/include/c++/13/bits/unique_ptr.h:404
#39 0x00007f542b23e1a6 in __run_exit_handlers (status=0, 
    listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, 
    run_dtors=run_dtors@entry=true) at exit.c:111
#40 0x00007f542b23e2ee in __GI_exit (status=<optimized out>) at exit.c:141
#41 0x000055a25231309a in quit_force (exit_arg=<optimized out>, 
    from_tty=from_tty@entry=1) at ../../gdb/top.c:1894
#42 0x000055a251f0c14c in quit_command (args=0x0, from_tty=1)
    at ../../gdb/cli/cli-cmds.c:506
#43 0x000055a251f14065 in cmd_func (cmd=<optimized out>, args=<optimized out>, 
    from_tty=<optimized out>) at ../../gdb/cli/cli-decode.c:2543
#44 0x000055a2523142a5 in execute_command (p=<optimized out>, 
    p@entry=0x55a253f85c70 "quit", from_tty=<optimized out>)
    at ../../gdb/top.c:688
#45 0x000055a2520148df in command_handler (command=0x55a253f85c70 "quit")
    at ../../gdb/event-top.c:619
#46 0x000055a252016399 in command_line_handler (rl=...)
    at ../../gdb/event-top.c:855
#47 0x000055a252014f11 in gdb_rl_callback_handler (rl=0x55a254e3e1c0 "quit")
    at ../../gdb/event-top.c:247
#48 0x00007f542bf0a0cd in rl_callback_read_char () at ../callback.c:299
#49 0x000055a252015076 in gdb_rl_callback_read_char_wrapper_noexcept ()
    at ../../gdb/event-top.c:189
#50 0x000055a252015228 in gdb_rl_callback_read_char_wrapper (
    client_data=<optimized out>) at ../../gdb/event-top.c:221
#51 0x000055a252013f90 in stdin_event_handler (error=<optimized out>, 
    client_data=0x55a253f91290) at ../../gdb/event-top.c:544
#52 0x000055a25257397e in gdb_wait_for_event (block=block@entry=1)
    at ../../gdbsupport/event-loop.cc:716
#53 0x000055a252574532 in gdb_do_one_event (mstimeout=mstimeout@entry=-1)
    at ../../gdbsupport/event-loop.cc:264
#54 0x000055a2520fb482 in start_event_loop () at ../../gdb/main.c:411
#55 captured_command_loop () at ../../gdb/main.c:471
#56 0x000055a2520fd5f5 in captured_main (data=0x7ffe0d8fab60)
    at ../../gdb/main.c:1330
#57 gdb_main (args=args@entry=0x7ffe0d8fab90) at ../../gdb/main.c:1345
#58 0x000055a251dea9ef in main (argc=<optimized out>, argv=<optimized out>)
    at ../../gdb/gdb.c:40

Comment 1 amerey 2023-05-09 14:52:43 UTC

This is likely an openssl issue so I'm changing the component. I'm not seeing anything suspect on gdb's side. According to valgrind the cause of the crash is an invalid read:

==89655== Invalid read of size 4
==89655==    at 0x545D7FC: UnknownInlinedFun (pthread_rwlock_common.c:298)
==89655==    by 0x545D7FC: pthread_rwlock_rdlock@@GLIBC_2.34 (pthread_rwlock_rdlock.c:26)
==89655==    by 0x59D87AC: CRYPTO_THREAD_read_lock (threads_pthread.c:97)
==89655==    by 0x59D0D95: ossl_lib_ctx_get_data (context.c:377)
==89655==    by 0x59DD989: UnknownInlinedFun (provider_core.c:339)
==89655==    by 0x59DD989: ossl_provider_deregister_child_cb (provider_core.c:1806)
==89655==    by 0x59D1B5E: UnknownInlinedFun (provider_child.c:281)
==89655==    by 0x59D1B5E: UnknownInlinedFun (context.c:248)
==89655==    by 0x59D1B5E: OSSL_LIB_CTX_free (context.c:241)
==89655==    by 0x1BBAEBC5: legacy_teardown (legacyprov.c:168)
==89655==    by 0x59DDDBA: UnknownInlinedFun (provider_core.c:1558)
==89655==    by 0x59DDDBA: ossl_provider_free.part.0 (provider_core.c:696)
==89655==    by 0x59A925B: UnknownInlinedFun (provider_core.c:681)
==89655==    by 0x59A925B: UnknownInlinedFun (evp_enc.c:1651)
==89655==    by 0x59A925B: EVP_CIPHER_free (evp_enc.c:1666)
==89655==    by 0x57B32FC: UnknownInlinedFun (ssl_lib.c:5967)
==89655==    by 0x57B32FC: UnknownInlinedFun (ssl_lib.c:5957)
==89655==    by 0x57B32FC: UnknownInlinedFun (ssl_lib.c:3492)
==89655==    by 0x57B32FC: SSL_CTX_free (ssl_lib.c:3429)
==89655==    by 0x5626F8D: ossl_closeone.isra.0 (openssl.c:1462)
==89655==    by 0x5616135: ossl_close.lto_priv.0 (openssl.c:1473)
==89655==    by 0x560CAC4: UnknownInlinedFun (vtls.c:696)
==89655==    by 0x560CAC4: UnknownInlinedFun (url.c:758)
==89655==    by 0x560CAC4: Curl_disconnect (url.c:879)
==89655==  Address 0x18 is not stack'd, malloc'd or (recently) free'd

Comment 2 Dmitry Belyavskiy 2023-05-09 15:07:20 UTC

I'd say the problem is in Curl. There were similar bug reports - https://github.com/nmap/nmap/issues/2482 fixed by https://github.com/nmap/nmap/commit/5f88cbac30c438587de6d01c13327f9b7316c94f

Comment 3 Kamil Dudka 2023-05-09 16:12:57 UTC

Any idea how to reproduce it off the gdb process?

I tried the following command and it did not crash:
$ rm -rf ~/.cache/debuginfod_client && valgrind -q debuginfod-find -v executable /bin/ls
Progress 1 / 0
Progress 7 / 0
Progress 8 / 0
Progress 9 / 0
Progress 10 / 0
Progress 11 / 0
Progress 12 / 0
Progress 40490 / 0
Headers:
x-debuginfod-size: 142104
x-debuginfod-archive: /mnt/fedora_koji_prod/koji/packages/coreutils/9.1/11.fc38/x86_64/coreutils-9.1-11.fc38.x86_64.rpm
x-debuginfod-file: /usr/bin/ls
Downloaded from https://debuginfod.fedoraproject.org/buildid/813d7726e4ddb91b7f102d567852077c51f793c8/executable
/home/kdudka/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/executable

... while the following command does:
$ rm -rf ~/.cache/debuginfod_client && gdb -q /bin/ls <<< ''
Reading symbols from /bin/ls...
Downloading separate debug info for /usr/bin/ls...
Reading symbols from /home/kdudka/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo...
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo...
(gdb) (gdb) 

Fatal signal: Segmentation fault
[...]

Comment 4 amerey 2023-05-09 20:52:42 UTC

(In reply to Kamil Dudka from comment #3)
> Any idea how to reproduce it off the gdb process?

I have not been able to reproduce it outside of gdb.

At first I thought it may have to do with gdb performing
multiple queries with a single CURLM multi handle.  But
even when I ensure that gdb only downloads the ls debuginfo
and nothing else (not even ls' .dwz), I still get a crash.

The crash does not occur when gdb downloads from a local
debuginfod server using a localhost URL. Strangely the
local debuginfod server itself has no problem actually
downloading the debuginfo on behalf of gdb.  In this case
there is no crash and valgrind does not detect any memory
errors in either gdb or the debuginfod server.

In any case no issues have been detected when using the
default openssl.cnf.

Comment 5 Kamil Dudka 2023-05-10 08:04:34 UTC

I tried to run gdb through gdb with preloaded libcurl.so from the upstream git HEAD (f7170a8f2ed4dc5a4cfb3ef3c002d218c4bcecad) and the backtrace looks like this:

$ rm -rf $HOME/.cache/debuginfod_client && LD_LIBRARY_PATH=/home/kdudka/curl/lib/.libs gdb -q -ex run --args gdb -q /bin/ls
Reading symbols from gdb...
Reading symbols from /usr/lib/debug/usr/libexec/gdb-13.1-4.fc38.x86_64.debug...
Starting program: /usr/bin/gdb -q /bin/ls
Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000
Downloading separate debug info for /lib64/liblzma.so.5                                                                                                                                      
Downloading separate debug info for /lib64/libbabeltrace.so.1                                                                                                                                
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/c7cd104410d857eb743a49d645b8b703e66b0995/debuginfo                                                                 
Downloading separate debug info for /lib64/libbabeltrace-ctf.so.1                                                                                                                            
Downloading separate debug info for /lib64/libipt.so.2                                                                                                                                       
Downloading separate debug info for /lib64/libmpfr.so.6                                                                                                                                      
Downloading separate debug info for /lib64/libgmp.so.10                                                                                                                                      
Downloading separate debug info for /lib64/libxxhash.so.0                                                                                                                                    
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/3d77f993c64f6d4f487ca5bf7c8483149e6bd28e/debuginfo                                                                 
Downloading separate debug info for /lib64/libreadline.so.8                                                                                                                                  
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/da79378d50e73c9cbf3405fdc55705835e281313/debuginfo                                                                 
Downloading separate debug info for /lib64/libz.so.1                                                                                                                                         
Downloading separate debug info for /lib64/libzstd.so.1                                                                                                                                      
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/95fc1c046547d542d2913ebc72290732112d8b9b/debuginfo                                                                 
Downloading separate debug info for /lib64/libncursesw.so.6                                                                                                                                  
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/ce6b29b0c0bd3bbd68d3a87c7e51fd6a67d13f5e/debuginfo                                                                 
Downloading separate debug info for /lib64/libtinfo.so.6                                                                                                                                     
Downloading separate debug info for /lib64/libpython3.11.so.1.0                                                                                                                              
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/486392b9c965505e765de7bc81c249eaa9ccc414/debuginfo                                                                 
Downloading separate debug info for /lib64/libexpat.so.1                                                                                                                                     
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/1f06ca43854228d1ad3508eb0a8c4fb59f1f5c8e/debuginfo                                                                 
Downloading separate debug info for /lib64/libsource-highlight.so.4                                                                                                                          
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/22a663096876815f8215534dbc2f6f218f5f3388/debuginfo                                                                 
Downloading separate debug info for /lib64/libstdc++.so.6                                                                                                                                    
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/4fbdf6ca060f2fe96c2321d9d59d775fa4b2ba3a/debuginfo                                                                 
Downloading separate debug info for /lib64/libm.so.6                                                                                                                                         
Downloading separate debug info for /lib64/libgcc_s.so.1                                                                                                                                     
Downloading separate debug info for /lib64/libc.so.6                                                                                                                                         
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/245240a31888ad5c11bbc55b18e02d87388f59a9/debuginfo                                                                 
[Thread debugging using libthread_db enabled]                                                                                                                                                
Using host libthread_db library "/lib64/libthread_db.so.1".
Downloading separate debug info for /lib64/libpopt.so.0
Downloading separate debug info for /lib64/libuuid.so.1                                                                                                                                      
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/7f68b06dd9ec5a0f937cb1e19b9bc66f14997db1/debuginfo                                                                 
Downloading separate debug info for /lib64/libgmodule-2.0.so.0                                                                                                                               
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/12c007ff846734d4be86d9c1892f2ab3431ad824/debuginfo                                                                 
Downloading separate debug info for /lib64/libglib-2.0.so.0                                                                                                                                  
Downloading separate debug info for /lib64/libboost_regex.so.1.78.0                                                                                                                          
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/ecbb0bd3a76b91d4668e8cbdaef774b2b347121c/debuginfo                                                                 
Downloading separate debug info for /lib64/libbz2.so.1                                                                                                                                       
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f8237ff24b622047428b97b1486b393e3a9e38de/debuginfo                                                                 
Downloading separate debug info for /lib64/libpcre2-8.so.0                                                                                                                                   
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/1ef098ba40305d86ec0cdae1e6aac188b1656550/debuginfo                                                                 
Downloading separate debug info for /lib64/libicudata.so.72                                                                                                                                  
Downloading separate debug info for /lib64/libicui18n.so.72                                                                                                                                  
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/13e5dae11fa14e4ac67e847e96ff38e07b8470a7/debuginfo                                                                 
Downloading separate debug info for /lib64/libicuuc.so.72                                                                                                                                    
Downloading separate debug info for /lib64/libnghttp2.so.14                                                                                                                                  
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/e10861ccd7fab76bc152927562f85635e27008f9/debuginfo                                                                 
Downloading separate debug info for /lib64/libidn2.so.0                                                                                                                                      
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/c25f9a2ef9b6fb843df0fc37c6c8903af04cbbe1/debuginfo                                                                 
Downloading separate debug info for /lib64/libpsl.so.5                                                                                                                                       
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/973e286449bb9a8694cdb165b372442956f8ded0/debuginfo                                                                 
Downloading separate debug info for /lib64/libssl.so.3                                                                                                                                       
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f762bb3d54ccc26f5182b558e1dfde142d778f73/debuginfo                                                                 
Downloading separate debug info for /lib64/libcrypto.so.3                                                                                                                                    
Downloading separate debug info for /lib64/libldap.so.2                                                                                                                                      
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f28845a559a64c25c9febe01e95aadfef853e597/debuginfo                                                                 
Downloading separate debug info for /lib64/liblber.so.2                                                                                                                                      
Downloading separate debug info for /lib64/libbrotlidec.so.1                                                                                                                                 
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/b227ed45334b5f849f2585438a1b50e5437a6974/debuginfo                                                                 
Downloading separate debug info for /lib64/libunistring.so.2                                                                                                                                 
Downloading separate debug info for /lib64/libevent-2.1.so.7                                                                                                                                 
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/8e12d5357a281470bc5db4ef22e74d983bf2cd49/debuginfo                                                                 
Downloading separate debug info for /lib64/libsasl2.so.3                                                                                                                                     
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f88c0fc23af93f1bd86b93b6c8e2c9d72bbb98b7/debuginfo                                                                 
Downloading separate debug info for /lib64/libbrotlicommon.so.1                                                                                                                              
Downloading separate debug info for /lib64/libresolv.so.2                                                                                                                                    
Downloading separate debug info for /lib64/libcrypt.so.2                                                                                                                                     
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/ec8602178ec0a572468bc35fad45528176e55224/debuginfo                                                                 
[Detaching after vfork from child process 1201]                                                                                                                                              
[New Thread 0x7fffe69ff6c0 (LWP 1202)]
[New Thread 0x7fffe61fe6c0 (LWP 1203)]
[New Thread 0x7fffe59fd6c0 (LWP 1204)]
[New Thread 0x7fffe51fc6c0 (LWP 1205)]
[New Thread 0x7fffe49fb6c0 (LWP 1206)]
[New Thread 0x7fffd7fff6c0 (LWP 1207)]
[New Thread 0x7fffd77fe6c0 (LWP 1208)]
[New Thread 0x7fffd6ffd6c0 (LWP 1209)]
Reading symbols from /bin/ls...
Downloading separate debug info for /usr/lib64/ossl-modules/legacy.so
[New Thread 0x7fffd65fc6c0 (LWP 1210)]                                                                                                                                                       
Downloading separate debug info for /usr/bin/ls
[###                                                                                                                                                                                      ]Downloading separate debug info for /lib64/libnss_myhostname.so.2
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/e691afe9fc8f8d74f8ee0efcbee363eebe296b8e/debuginfo                                                                 
Downloading separate debug info for /lib64/libcap.so.2                                                                                                                                       
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/2fe8ebb5811811046e04d8f4c000696cb9b24e24/debuginfo                                                                 
Downloading separate debug info for /lib64/libnss_mdns4_minimal.so.2                                                                                                                         
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/fdb94adac135155164c94a95a342d03685b6804e/debuginfo                                                                 
Downloading separate debug info for /lib64/libnss_resolve.so.2                                                                                                                               
[Thread 0x7fffd65fc6c0 (LWP 1210) exited]                                                                                                                                                    
Reading symbols from /home/kdudka/.cache/debuginfod_client/a146002a40b622e4d8420c5b96513f0784c8eb7b/debuginfo...                                                                             
Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/a146002a40b622e4d8420c5b96513f0784c8eb7b/debuginfo
Downloading separate debug info for /usr/lib64/gconv/UTF-32.so                                                                                                                               
(gdb) quit                                                                                                                                                                                   

Thread 1 "gdb" received signal SIGSEGV, Segmentation fault.
___pthread_rwlock_rdlock (rwlock=0x0) at pthread_rwlock_rdlock.c:26
Downloading source file /usr/src/debug/glibc-2.37-4.fc38.x86_64/nptl/pthread_rwlock_rdlock.c
26        int result = __pthread_rwlock_rdlock_full64 (rwlock, CLOCK_REALTIME, NULL);                                                                                                        
(gdb) bt
#0  ___pthread_rwlock_rdlock (rwlock=0x0) at pthread_rwlock_rdlock.c:26
#1  0x00007ffff41c0f8d in CRYPTO_THREAD_read_lock (lock=<optimized out>) at crypto/threads_pthread.c:97
#2  0x00007ffff41b901f in ossl_lib_ctx_get_data (ctx=<optimized out>, index=1, meth=0x7ffff43ddaa0 <provider_store_method.lto_priv>) at crypto/context.c:413
#3  0x00007ffff41c659d in get_provider_store (libctx=<optimized out>) at crypto/provider_core.c:339
#4  ossl_provider_deregister_child_cb (handle=0x555556741220) at crypto/provider_core.c:1806
#5  0x00007ffff41b9f47 in ossl_provider_deinit_child (ctx=0x5555566fe840) at crypto/provider_child.c:281
#6  OSSL_LIB_CTX_free (ctx=0x5555566fe840) at crypto/context.c:284
#7  OSSL_LIB_CTX_free (ctx=0x5555566fe840) at crypto/context.c:277
#8  0x00007ffff4813bee in legacy_teardown (provctx=0x5555567414e0) at providers/legacyprov.c:168
#9  0x00007ffff41c69fb in ossl_provider_teardown (prov=0x555556741220) at crypto/provider_core.c:1558
#10 ossl_provider_free (prov=0x555556741220) at crypto/provider_core.c:696
#11 0x00007ffff41904bc in ossl_provider_free (prov=<optimized out>) at crypto/provider_core.c:681
#12 evp_cipher_free_int (cipher=0x555556783fe0) at crypto/evp/evp_enc.c:1651
#13 EVP_CIPHER_free (cipher=0x555556783fe0) at crypto/evp/evp_enc.c:1666
#14 0x00007ffff4996abd in ssl_evp_cipher_free (cipher=0x555556783fe0) at ssl/ssl_lib.c:5967
#15 ssl_evp_cipher_free (cipher=0x555556783fe0) at ssl/ssl_lib.c:5957
#16 SSL_CTX_free (a=0x55555678d610) at ssl/ssl_lib.c:3492
#17 SSL_CTX_free (a=0x55555678d610) at ssl/ssl_lib.c:3429
#18 0x00007ffff6da29b7 in ossl_close (cf=0x555556742ef8, data=0x55555674ab88) at vtls/openssl.c:1901
#19 0x00007ffff6daa737 in cf_close (cf=0x555556742ef8, data=0x55555674ab88) at vtls/vtls.c:1450
#20 0x00007ffff6daa991 in ssl_cf_destroy (cf=0x555556742ef8, data=0x55555674ab88) at vtls/vtls.c:1510
#21 0x00007ffff6d0f699 in Curl_conn_cf_discard_chain (pcf=0x555556744c70, data=0x55555674ab88) at cfilters.c:165
#22 0x00007ffff6d0b683 in cf_hc_close (cf=0x555556744c68, data=0x55555674ab88) at cf-https-connect.c:436
#23 0x00007ffff6d0f780 in Curl_conn_close (data=0x55555674ab88, index=0) at cfilters.c:186
#24 0x00007ffff6d8ebeb in conn_shutdown (data=0x55555674ab88) at url.c:685
#25 0x00007ffff6d8f1fb in Curl_disconnect (data=0x55555674ab88, conn=0x555556742ff8, dead_connection=false) at url.c:796
#26 0x00007ffff6d12039 in Curl_conncache_close_all_connections (connc=0x555556746b60) at conncache.c:532
#27 0x00007ffff6d62eb3 in curl_multi_cleanup (multi=0x555556746a38) at multi.c:2831
#28 0x00007ffff7bc154e in debuginfod_end (client=0x555556746360) at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1892
#29 debuginfod_end (client=0x555556746360) at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1887
#30 0x000055555589a9c5 in debuginfod_client_deleter::operator() (c=<optimized out>, this=<optimized out>) at ../../gdb/debuginfod-support.c:105
#31 std::unique_ptr<debuginfod_client, debuginfod_client_deleter>::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/unique_ptr.h:404
#32 0x00007ffff70621a6 in __run_exit_handlers (status=0, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:111
#33 0x00007ffff70622ee in __GI_exit (status=<optimized out>) at exit.c:141
#34 0x0000555555c3009a in quit_force (exit_arg=<optimized out>, from_tty=from_tty@entry=1) at ../../gdb/top.c:1894
#35 0x000055555582914c in quit_command (args=0x0, from_tty=1) at ../../gdb/cli/cli-cmds.c:506
#36 0x0000555555831065 in cmd_func (cmd=<optimized out>, args=<optimized out>, from_tty=<optimized out>) at ../../gdb/cli/cli-decode.c:2543
#37 0x0000555555c312a5 in execute_command (p=<optimized out>, from_tty=<optimized out>) at ../../gdb/top.c:688
#38 0x0000555555931f11 in gdb_rl_callback_handler (rl=0x0) at ../../gdb/event-top.c:247
#39 0x00007ffff7d7b0cd in rl_callback_read_char () at ../callback.c:299
#40 0x0000555555932076 in gdb_rl_callback_read_char_wrapper_noexcept () at ../../gdb/event-top.c:189
#41 0x0000555555932228 in gdb_rl_callback_read_char_wrapper (client_data=<optimized out>) at ../../gdb/event-top.c:221
#42 0x0000555555930f90 in stdin_event_handler (error=<optimized out>, client_data=0x5555563e1730) at ../../gdb/event-top.c:544
#43 0x0000555555e9097e in gdb_wait_for_event (block=block@entry=1) at ../../gdbsupport/event-loop.cc:716
#44 0x0000555555e91532 in gdb_do_one_event (mstimeout=mstimeout@entry=-1) at ../../gdbsupport/event-loop.cc:264
#45 0x0000555555a18482 in start_event_loop () at ../../gdb/main.c:411
#46 captured_command_loop () at ../../gdb/main.c:471
#47 0x0000555555a1a5f5 in captured_main (data=0x7fffffffe130) at ../../gdb/main.c:1330
#48 gdb_main (args=args@entry=0x7fffffffe160) at ../../gdb/main.c:1345
#49 0x00005555557079ef in main (argc=<optimized out>, argv=<optimized out>) at ../../gdb/gdb.c:40

Comment 6 Kamil Dudka 2023-05-10 13:42:09 UTC

The issue seems to be that both libcrypto.so and gdb register atexit() handlers, which are later executed in the unfavorable order:

1. OpenSSL's exit handler calls OPENSSL_cleanup(), which wipes global data of OpenSSL.
2. gdb's exit handler calls debuginfod_end() from libdebuginfod.so, which calls curl_multi_cleanup() from libcurl.so, which in turn calls SSL_CTX_free() when OpenSSL is already deinitialized.

I do not think there is anything that libcurl could do better.  Switching back to gdb...

Comment 7 amerey 2023-05-19 22:03:55 UTC

(In reply to Kamil Dudka from comment #6)
> The issue seems to be that both libcrypto.so and gdb register atexit()
> handlers, which are later executed in the unfavorable order:
> 
> 1. OpenSSL's exit handler calls OPENSSL_cleanup(), which wipes global data
> of OpenSSL.
> 2. gdb's exit handler calls debuginfod_end() from libdebuginfod.so, which
> calls curl_multi_cleanup() from libcurl.so, which in turn calls
> SSL_CTX_free() when OpenSSL is already deinitialized.
> 
> I do not think there is anything that libcurl could do better.  Switching
> back to gdb...

libcrypto's exit handler is registered during curl_global_init() and the error
happens only because curl_multi_cleanup is called during exit.

AFAICT it should be ok to call curl_multi_cleanup during exit.  I can't find
anything in the docs to suggest otherwise, and things work as expected when
we aren't using these openssl legacy settings.

This unfavorable order of handlers still looks like a libcurl issue. gdb is
using libcurl according to spec.

Comment 8 Kamil Dudka 2023-05-20 08:23:03 UTC

I think it is a bad idea to call curl_multi_cleanup() from an atexit() handler, where one can hardly assume anything about the global state of the underlying libraries.

Comment 9 Mark Wielaard 2023-05-21 15:23:29 UTC

The problem is that the atexit handlers are called in refers order. The unique_ptr deleter for the debuginfod_client is registered before the debuginfod_client is created for the first time.

This is how debuginfod_begin works (the first time it is called):

static pthread_once_t init_control = PTHREAD_ONCE_INIT;

static void
libcurl_init(void)
{
  curl_global_init(CURL_GLOBAL_DEFAULT);
}

debuginfod_client  *
debuginfod_begin (void)
{
  /* Initialize libcurl lazily, but only once.  */
  pthread_once (&init_control, libcurl_init);

...

So I think this code in gdb/debuginfod-support.c registers its at_exit handler for the deleter first:

/* Deleter for a debuginfod_client.  */

struct debuginfod_client_deleter
{
  void operator() (debuginfod_client *c)
  {
    debuginfod_end (c);
  }
};

using debuginfod_client_up
  = std::unique_ptr<debuginfod_client, debuginfod_client_deleter>;

So debuginfod_end will be called after libcrypto's exit handler has been called.

I don't know how to switch the registration order of the at_exit handlers.

Comment 10 Andrew Burgess 2023-05-23 09:18:43 UTC

I think the patch below will ensure that the debuginfod_client object is cleaned up (debuginfod_end is called) before the at_exit handlers are run.  I'm currently setting up an environment to test this, but thought I'd post this here in case anyone else was also looking at this.

diff --git a/gdb/debuginfod-support.c b/gdb/debuginfod-support.c
index 5853f420a18..c59f0f21956 100644
--- a/gdb/debuginfod-support.c
+++ b/gdb/debuginfod-support.c
@@ -180,20 +180,83 @@ progressfn (debuginfod_client *c, long cur, long total)
   return 0;
 }
 
-static debuginfod_client *
-get_debuginfod_client ()
+/* A class to manage setting up (and cleaning up) a debuginfod_client
+   object.  GDB should only have a single debuginfod_client object, so we
+   should only create a single instance of this class.  */
+
+struct debuginfod_client_manager
 {
-  static debuginfod_client_up global_client;
+  /* Return a pointer to the managed debuginfod_client object,
+     initialising it first if needed.  */
 
-  if (global_client == nullptr)
-    {
-      global_client.reset (debuginfod_begin ());
+  debuginfod_client *get ()
+  {
+    if (m_global_client == nullptr)
+      this->init ();
+    return m_global_client.get ();
+  }
 
-      if (global_client != nullptr)
-	debuginfod_set_progressfn (global_client.get (), progressfn);
-    }
+private:
+
+  /* Callback used by GDB's final cleanup mechanism.  Cleans up the
+     debuginfod_client object being managed by ARG, which should be a
+     debuginfod_client_manager pointer.  */
+
+  static void cleanup_debuginfod_client (void *arg)
+  {
+    debuginfod_client_manager *m = (debuginfod_client_manager *) arg;
+    m->cleanup ();
+  }
+
+  /* Initialise the managed debuginfod_client object and register with
+     GDB's global cleanup mechanism to ensure that the object is cleaned up
+     correctly.  */
+  void init ()
+  {
+    gdb_assert (m_global_client == nullptr);
+
+    debuginfod_client *client = debuginfod_begin ();
+
+    if (client != nullptr)
+      {
+	/* Debuginfod (or the libraries it uses) make use of at_exit for
+	   final cleanup.  Meanwhile, GDB makes use of C++ object
+	   destruction to cleanup global variables.  If we make
+	   m_global_client a global and rely on the C++ finalisation to
+	   call the unique_ptr destructor then this is done too late, by
+	   this point the at_exit handlers have already run and so the
+	   debuginfod_client object will already be partially cleaned up.
+
+	   Instead, register a cleanup with GDB's final cleanup mechanism.
+	   This is run by GDB before calling exit, which means we will
+	   cleanup the debuginfod_client before calling exit, and so before
+	   calling the at_exit handlers.  */
+	make_final_cleanup (cleanup_debuginfod_client, this);
+	debuginfod_set_progressfn (client, progressfn);
+	m_global_client.reset (client);
+      }
+  }
+
+  /* The managed debuginfod_client object.  */
+  debuginfod_client_up m_global_client;
+};
+
+/* Return a pointer to the single global debuginfod_client, initialising it
+   first if needed.  */
 
-  return global_client.get ();
+static debuginfod_client *
+get_debuginfod_client ()
+{
+  static debuginfod_client_manager global_client_manager;
+  return global_client_manager.get ();
 }
 
 /* Check if debuginfod is enabled.  If configured to do so, ask the user

Comment 11 Andrew Burgess 2023-05-23 11:24:26 UTC

I've sent a version of the above patch to the GDB mailing list:  https://sourceware.org/pipermail/gdb-patches/2023-May/199779.html for consideration.

Comment 12 Andrew Burgess 2023-06-20 09:11:19 UTC

Upstream commit f3eee5861743d635ea3a5be7eae3115f85ec9dae fixes this issue.  I've opened pull requests #87 and #88 to merge backport this fix to rawhide and f38 respectively.

Comment 13 Fedora Update System 2023-08-08 02:15:13 UTC

FEDORA-2023-42bcd5dc5d has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-42bcd5dc5d

Comment 14 Fedora Update System 2023-08-09 02:00:29 UTC

FEDORA-2023-42bcd5dc5d has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-42bcd5dc5d`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-42bcd5dc5d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 15 Fedora Update System 2023-08-10 00:41:33 UTC

FEDORA-2023-42bcd5dc5d has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.

aburgess
ahajkova
amerey
blarsen
crypto-team
dbelyavs
fweimer
jamacku
jan
jmigacz
kdudka
keiths
kevinb
lzaoral
mcermak
mjw
mkolar
msekleta
mspacek
mturk
paul
sahana
svashisht
tm
vmihalko