Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1937513 - [abrt] Crash under gnutls_x509_trust_list_verify_crt() called from multiple threads
Summary: [abrt] Crash under gnutls_x509_trust_list_verify_crt() called from multiple t...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: glib-networking
Version: rawhide
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Matthias Clasen
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:51347135d2f65c4f5c04a092d22...
: 1939668 1950195 1950636 (view as bug list)
Depends On:
Blocks: 1947188
TreeView+ depends on / blocked
 
Reported: 2021-03-10 19:45 UTC by Matti Linnanvuori
Modified: 2021-05-26 16:50 UTC (History)
26 users (show)

Fixed In Version: glib-networking-2.68.1-1.fc34
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1947188 (view as bug list)
Environment:
Last Closed: 2021-04-27 00:30:10 UTC
Type: ---
Embargoed:

Attachments (Terms of Use)
File: backtrace (104.45 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: core_backtrace (49.38 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: cpuinfo (1.84 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: dso_list (10.60 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: environ (1.31 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: exploitable (82 bytes, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: limits (1.29 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: maps (63.98 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: mountinfo (3.04 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: open_fds (2.47 KB, text/plain)
2021-03-10 19:45 UTC, Matti Linnanvuori 		no flags Details
File: proc_pid_status (1.38 KB, text/plain)
2021-03-10 19:47 UTC, Matti Linnanvuori 		no flags Details
Test program that shares a trust list object across threads (1.84 KB, text/plain)
2021-04-19 15:03 UTC, Daiki Ueno 		no flags Details
test-truct-list.c (modified) (3.33 KB, text/plain)
2021-04-19 16:48 UTC, Milan Crha 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
GNOME Gitlab GNOME/glib-networking - merge_requests 148 0 None None None 2021-04-01 19:24:22 UTC

Description Matti Linnanvuori 2021-03-10 19:45:27 UTC 
Version-Release number of selected component:
evolution-data-server-3.39.2-3.fc35

Additional info:
reporter:       libreport-2.14.0
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/evolution-calendar-factory.service
cmdline:        /usr/libexec/evolution-calendar-factory
crash_function: _asn1_delete_structure
executable:     /usr/libexec/evolution-calendar-factory
journald_cursor: s=1858fd87330646bcb13b6970d3f1aed6;i=157e;b=ca05816e129149739ab9259bd3f11156;m=dfeaece1;t=5bd32ea316e4b;x=9bbf4328ae9f5891
kernel:         5.11.0-155.fc35.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 _asn1_delete_structure at /usr/src/debug/libtasn1-4.16.0-4.fc34.x86_64/lib/structure.c:327
 #1 asn1_delete_structure at /usr/src/debug/libtasn1-4.16.0-4.fc34.x86_64/lib/structure.c:296
 #2 gnutls_x509_crt_deinit at x509.c:297
 #3 g_tls_certificate_gnutls_build_chain at ../tls/gnutls/gtlscertificate-gnutls.c:805
 #4 g_tls_connection_gnutls_retrieve_peer_certificate at ../tls/gnutls/gtlsconnection-gnutls.c:817
 #5 finish_handshake at ../tls/base/gtlsconnection-base.h:33
 #6 g_tls_connection_base_handshake at ../tls/base/gtlsconnection-base.c:1691
 #7 soup_socket_handshake_sync at ../libsoup/soup-socket.c:1501
 #8 soup_connection_connect_sync at ../libsoup/soup-connection.c:472
 #9 get_connection at ../libsoup/soup-session.c:1959
Comment 1 Matti Linnanvuori 2021-03-10 19:45:31 UTC 
Created attachment 1762425 [details]
File: backtrace
Comment 2 Matti Linnanvuori 2021-03-10 19:45:33 UTC 
Created attachment 1762426 [details]
File: core_backtrace
Comment 3 Matti Linnanvuori 2021-03-10 19:45:34 UTC 
Created attachment 1762427 [details]
File: cpuinfo
Comment 4 Matti Linnanvuori 2021-03-10 19:45:35 UTC 
Created attachment 1762428 [details]
File: dso_list
Comment 5 Matti Linnanvuori 2021-03-10 19:45:37 UTC 
Created attachment 1762429 [details]
File: environ
Comment 6 Matti Linnanvuori 2021-03-10 19:45:38 UTC 
Created attachment 1762430 [details]
File: exploitable
Comment 7 Matti Linnanvuori 2021-03-10 19:45:39 UTC 
Created attachment 1762431 [details]
File: limits
Comment 8 Matti Linnanvuori 2021-03-10 19:45:41 UTC 
Created attachment 1762432 [details]
File: maps
Comment 9 Matti Linnanvuori 2021-03-10 19:45:43 UTC 
Created attachment 1762433 [details]
File: mountinfo
Comment 10 Matti Linnanvuori 2021-03-10 19:45:44 UTC 
Created attachment 1762434 [details]
File: open_fds
Comment 11 Matti Linnanvuori 2021-03-10 19:47:53 UTC 
Created attachment 1762435 [details]
File: proc_pid_status
Comment 12 Milan Crha 2021-03-11 07:24:56 UTC 
Thanks for a bug report. I see in the backtrace that three threads are doing a similar thing, connecting to a server. All of them are deep in gnutls/glib-networking code. Maybe this will ring a bell to the glib-networking developers, thus I move the bug there for further investigation.

By the way, was this just a one time crash or you can reproduce it anyhow, please?
Comment 13 Michael Catanzaro 2021-03-11 15:59:05 UTC 
The most likely possibility is some unknown threadsafety bug in glib-networking, though I can't imagine what. libsoup and evolution-data-server do not appear to be doing anything wrong. evolution-data-server is using separate SoupSessions on separate threads, which is intended to be safe. libsoup is using separate GTlsConnections on separate threads, which is again intended to be safe.
Comment 14 Matti Linnanvuori 2021-03-11 16:20:31 UTC 
(In reply to Milan Crha from comment #12)
> By the way, was this just a one time crash or you can reproduce it anyhow,
> please?

I remember having seen it only once and I have no idea how to reproduce it.
Comment 15 Milan Crha 2021-03-11 16:39:27 UTC 
Thank you both for a quick update. Maybe that's just some kind of coincidence. it can sometimes happen that there's a use-after-free or similar problem, which touches memory occupied by something else, causing crashes (sometimes waaaay) later than when the "incident" happened. Let's close this for now and revisit in case it'll strike again and there'll be any clue why it was so.
Comment 16 Milan Crha 2021-03-16 08:23:14 UTC 
I gave this a try under valgrind, just in case I'd be lucky, but I was not lucky. I had installed:

glib-networking-2.68~alpha-1.fc35.x86_64
glib2-2.67.5-1.fc35.x86_64
gnutls-3.7.1-1.fc35.x86_64

and valgrind did not report any issue. I had configured 3 IMAP account and 1 EWS account. I tried Send/Receive and also going offline and back online, which causes reconnects of the accounts.

My test might not mean much, maybe just it did not trigger the conditions to reproduce this.
Comment 17 Milan Crha 2021-03-17 07:30:07 UTC 
*** Bug 1939668 has been marked as a duplicate of this bug. ***
Comment 18 Milan Crha 2021-03-17 07:31:21 UTC 
The bug #1939668 is a variant of this. I feel like being lucky yesterday, when I tried to reproduce.
Comment 19 Milan Crha 2021-03-29 06:47:14 UTC 
*** Bug 1943765 has been marked as a duplicate of this bug. ***
Comment 20 Milan Crha 2021-03-29 06:48:46 UTC 
I'm reopening this, there are several duplicate/similar reports, both from rawhide and from Fedora 34.
Comment 21 Milan Crha 2021-03-31 12:17:20 UTC 
*** Bug 1944854 has been marked as a duplicate of this bug. ***
Comment 22 Milan Crha 2021-03-31 12:17:52 UTC 
In bug #1944854, multiple threads are accessing the same `list` 0x7f8d6c02bd50.
Comment 23 Milan Crha 2021-03-31 12:19:46 UTC 
*** Bug 1944883 has been marked as a duplicate of this bug. ***
Comment 24 Michael Catanzaro 2021-04-01 18:30:02 UTC 
Good find, Milan. You've proved this is glib-networking's fault. gnutls_x509_trust_list_t is not supposed to be used from multiple threads at once. I think it is a locking problem in GTlsDatabaseGnutls.
Comment 25 Milan Crha 2021-04-12 06:46:56 UTC 
*** Bug 1948101 has been marked as a duplicate of this bug. ***
Comment 26 Milan Crha 2021-04-13 08:32:05 UTC 
*** Bug 1948858 has been marked as a duplicate of this bug. ***
Comment 27 Milan Crha 2021-04-16 05:43:13 UTC 
*** Bug 1950195 has been marked as a duplicate of this bug. ***
Comment 28 Milan Crha 2021-04-19 07:35:34 UTC 
*** Bug 1950636 has been marked as a duplicate of this bug. ***
Comment 29 Milan Crha 2021-04-19 07:35:54 UTC 
*** Bug 1950695 has been marked as a duplicate of this bug. ***
Comment 30 Milan Crha 2021-04-19 10:05:27 UTC 
*** Bug 1950941 has been marked as a duplicate of this bug. ***
Comment 31 Daiki Ueno 2021-04-19 10:20:01 UTC 
Sorry for chiming in late, but I'm wondering if this might be an issue in GnuTLS, as it has a thread-safety promise:
https://www.gnutls.org/manual/gnutls.html#Thread-safety
"The GnuTLS library is thread safe by design, meaning that objects of the library such as TLS sessions, can be safely divided across threads as long as a single thread accesses a single object"
and later:
"For several aspects of the library (e.g., the random generator, PKCS#11 operations), the library may utilize mutex locks".

I still haven't figured out the erroneous code yet, but the backtrace attached to bug 1944854 shows the direct cause of the crash is in:
  #8  sys_C_FindObjectsInit (handle=<optimized out>, template=0x7f8d84a1a920, count=3) at ../trust/module.c:1215
https://github.com/p11-glue/p11-kit/blob/master/trust/module.c#L1222

However, this part is actually protected with a mutex, and other 2 threads are blocked:
https://github.com/p11-glue/p11-kit/blob/master/trust/module.c#L1189

So I suspect that the race might be in between the C_OpenSession and C_FindObjectsInit calls, i.e., _pkcs11_traverse_tokens in GnuTLS.
Comment 32 Daiki Ueno 2021-04-19 15:03:25 UTC 
Created attachment 1773318 [details]
Test program that shares a trust list object across threads

FWIW I tested with the attached program, which calls gnutls_x509_trust_list_verify_crt2 on a shared trust list object from 100 threads but couldn't reproduce the issue:

  $ gcc -pthread -o test-trust-list test-trust-list.c `pkg-config gnutls --cflags --libs`
  $ gnutls-cli --save-cert=www.google.com.certs.pem www.google.com
  ...
  $ ./test-trust-list www.*.pem | wc -l
  100

with stock gnutls-devel package: gnutls-devel-3.6.15-1.fc33.x86_64
Comment 33 Milan Crha 2021-04-19 16:48:13 UTC 
Created attachment 1773326 [details]
test-truct-list.c (modified)

I slightly modified your test program, to make it show where the crash happens and to mimic what the code probably does. I confess, it's pretty hard to reproduce, but it does happen. I reproduced it with gnutls-3.7.1-2.fc35.x86_64, when I have downloaded the .pem files for google.com and facebook.com and I run the app as:

   for i in {1..1000}; do ./test-trust-list www*.pem; done

It usually works, but sometimes crashes. Here are few backtraces:

realloc(): invalid next size
[New LWP 299958]
[New LWP 299960]
[New LWP 299961]
[New LWP 299963]
[New LWP 299964]
[New LWP 299965]
[New LWP 299966]
[New LWP 299967]
Error while reading shared library symbols for /lib64/libpthread.so.0:
Cannot find user-level thread for LWP 299967: generic error
0x00007f307adfb51a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0

Thread 9 (LWP 299967 "test-trust-list"):
#0  0x00007f307adf8240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f307adf14f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f307a7b7f86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f307aea4641 in scan_slots (slots=slots@entry=0x7f3075f9b650, nslots=nslots@entry=0x7f3075f9b3a8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f307aea9707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f307aea3c70 <find_cert_cb>, input=input@entry=0x7f3075f9b830, info=0x7f30500859c0, pin_info=pin_info@entry=0x0, flags=8) at pkcs11.c:1574
#5  0x00007f307aeab995 in _gnutls_pkcs11_crt_is_known (url=0x7f3075f9b890 "\004", url@entry=0xd92e00 "pkcs11:", cert=0x7f3050037970, flags=flags@entry=1280, trusted_cert=trusted_cert@entry=0x0) at pkcs11.c:4614
#6  0x00007f307aeabdab in gnutls_pkcs11_crt_is_known (url=url@entry=0xd92e00 "pkcs11:", cert=<optimized out>, flags=flags@entry=1280) at pkcs11.c:4703
#7  0x00007f307aef24de in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdc6e70, url=0xd92e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f3075f9bad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1293
#8  0x00007f307af02880 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f3075f9bad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f3075f9bc44, func=0x0) at verify-high.c:1600
#9  0x000000000040152a in thread_func ()
#10 0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 8 (LWP 299966 "test-trust-list"):
#0  0x00007f307ace3e1f in wait4 () from /lib64/libc.so.6
#1  0x00007f307ac610cb in do_system () from /lib64/libc.so.6
#2  0x000000000040132e in segv_handler ()
#3  <signal handler called>
#4  0x00007f307ac54292 in raise () from /lib64/libc.so.6
#5  0x00007f307ac3d8a4 in abort () from /lib64/libc.so.6
#6  0x00007f307ac96cd7 in __libc_message () from /lib64/libc.so.6
#7  0x00007f307ac9e95c in malloc_printerr () from /lib64/libc.so.6
#8  0x00007f307aca2c3c in _int_realloc () from /lib64/libc.so.6
#9  0x00007f307aca3f66 in realloc () from /lib64/libc.so.6
#10 0x00007f307ae7000d in gnutls_realloc_fast (ptr=0x7f30700c7090, size=<optimized out>) at mem.c:56
#11 0x00007f307af0196b in trust_list_add_compat (cert=<optimized out>, list=0xdc6e70) at verify-high.c:301
#12 gnutls_x509_trust_list_get_issuer (list=list@entry=0xdc6e70, cert=0xdb6ec0, issuer=issuer@entry=0x7f307679caa0, flags=flags@entry=0) at verify-high.c:1142
#13 0x00007f307af021a0 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f307679cad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f307679cc44, func=0x0) at verify-high.c:1498
#14 0x000000000040152a in thread_func ()
#15 0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 7 (LWP 299965 "test-trust-list"):
#0  0x00007f307ac9d0bb in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f307aca44b3 in calloc () from /lib64/libc.so.6
#2  0x00007f307a7b9237 in sys_C_OpenSession () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f307ab11351 in managed_C_OpenSession () from /lib64/libp11-kit.so.0
#4  0x00007f307aea97d2 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f307aea3c70 <find_cert_cb>, input=input@entry=0x7f3076f9d830, info=0x7f3070068370, pin_info=pin_info@entry=0x0, flags=8) at pkcs11.c:1599
#5  0x00007f307aeab995 in _gnutls_pkcs11_crt_is_known (url=0x7f3076f9d890 "k]\361\331\376\276#,\003", url@entry=0xd92e00 "pkcs11:", cert=0xdb5da0, flags=flags@entry=1538, trusted_cert=trusted_cert@entry=0x7f3076f9d998) at pkcs11.c:4614
#6  0x00007f307aef22d5 in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdc6e70, url=0xd92e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f3076f9dad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1205
#7  0x00007f307af02880 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f3076f9dad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f3076f9dc44, func=0x0) at verify-high.c:1600
#8  0x000000000040152a in thread_func ()
#9  0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 6 (LWP 299964 "test-trust-list"):
#0  0x00007f307ac9d0bb in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f307aca404d in realloc () from /lib64/libc.so.6
#2  0x00007f307ae7000d in gnutls_realloc_fast (ptr=0x7f30700c7090, size=<optimized out>) at mem.c:56
#3  0x00007f307af0196b in trust_list_add_compat (cert=<optimized out>, list=0xdc6e70) at verify-high.c:301
#4  gnutls_x509_trust_list_get_issuer (list=list@entry=0xdc6e70, cert=0xdb6ec0, issuer=issuer@entry=0x7f307779eaa0, flags=flags@entry=0) at verify-high.c:1142
#5  0x00007f307af021a0 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f307779ead0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f307779ec44, func=0x0) at verify-high.c:1498
#6  0x000000000040152a in thread_func ()
#7  0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 5 (LWP 299963 "test-trust-list"):
#0  0x00007f307adf8240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f307adf14f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f307a7bc5dd in sys_C_GetAttributeValue () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f307aeac58d in pkcs11_override_cert_exts (sinfo=sinfo@entry=0x7f3079fa34e0, spki=spki@entry=0x7f3079fa3100, der=der@entry=0x7f3079fa30d0) at pkcs11x.c:134
#4  0x00007f307aea44a2 in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f3079fa34e0, tinfo=tinfo@entry=0x7f3079fa3410, lib_info=lib_info@entry=0x7f307b0127b8 <providers+24>, input=input@entry=0x7f3079fa3830) at pkcs11.c:4233
#5  0x00007f307aea974f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f307aea3c70 <find_cert_cb>, input=input@entry=0x7f3079fa3830, info=0x7f30480c0040, pin_info=pin_info@entry=0xdc5638, flags=8) at pkcs11.c:1631
#6  0x00007f307aeab1c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xd92e00 "pkcs11:", cert=0xdc5620, issuer=issuer@entry=0x7f3079fa39a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#7  0x00007f307aef23aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdc6e70, url=0xd92e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f3079fa3ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#8  0x00007f307af02880 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f3079fa3ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f3079fa3c44, func=0x0) at verify-high.c:1600
#9  0x000000000040152a in thread_func ()
#10 0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 4 (LWP 299961 "test-trust-list"):
#0  0x00007f307adf8240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f307adf14f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f307a7c2ffe in sys_C_FindObjectsFinal.lto_priv.0 () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f307aea409f in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f30797a24e0, tinfo=tinfo@entry=0x7f30797a2410, lib_info=lib_info@entry=0x7f307b0127b8 <providers+24>, input=input@entry=0x7f30797a2830) at pkcs11.c:4224
#4  0x00007f307aea974f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f307aea3c70 <find_cert_cb>, input=input@entry=0x7f30797a2830, info=0x7f306c095e50, pin_info=pin_info@entry=0xdc5638, flags=8) at pkcs11.c:1631
#5  0x00007f307aeab1c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xd92e00 "pkcs11:", cert=0xdc5620, issuer=issuer@entry=0x7f30797a29a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#6  0x00007f307aef23aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdc6e70, url=0xd92e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f30797a2ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#7  0x00007f307af02880 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f30797a2ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f30797a2c44, func=0x0) at verify-high.c:1600
#8  0x000000000040152a in thread_func ()
#9  0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 3 (LWP 299960 "test-trust-list"):
#0  0x00007f307adf8240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f307adf14f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f307a7bc5dd in sys_C_GetAttributeValue () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f307aeb2e1d in pkcs11_get_attribute_avalue (module=0xdccee0, sess=6839, object=103, type=type@entry=297, res=res@entry=0x7f3078fa1100) at pkcs11_int.c:154
#4  0x00007f307aea4489 in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f3078fa14e0, tinfo=tinfo@entry=0x7f3078fa1410, lib_info=lib_info@entry=0x7f307b0127b8 <providers+24>, input=input@entry=0x7f3078fa1830) at pkcs11.c:4231
#5  0x00007f307aea974f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f307aea3c70 <find_cert_cb>, input=input@entry=0x7f3078fa1830, info=0x7f305c055c00, pin_info=pin_info@entry=0xdb6ed8, flags=8) at pkcs11.c:1631
#6  0x00007f307aeab1c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xd92e00 "pkcs11:", cert=0xdb6ec0, issuer=issuer@entry=0x7f3078fa19a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#7  0x00007f307aef23aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdc6e70, url=0xd92e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f3078fa1ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#8  0x00007f307af02880 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f3078fa1ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f3078fa1c44, func=0x0) at verify-high.c:1600
#9  0x000000000040152a in thread_func ()
#10 0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 2 (LWP 299958 "test-trust-list"):
#0  0x00007f307adf8240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f307adf14f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f307a7c2ffe in sys_C_FindObjectsFinal.lto_priv.0 () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f307aeac476 in pkcs11_override_cert_exts (sinfo=sinfo@entry=0x7f3077f9f4e0, spki=spki@entry=0x7f3077f9f100, der=der@entry=0x7f3077f9f0d0) at pkcs11x.c:163
#4  0x00007f307aea44a2 in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f3077f9f4e0, tinfo=tinfo@entry=0x7f3077f9f410, lib_info=lib_info@entry=0x7f307b0127b8 <providers+24>, input=input@entry=0x7f3077f9f830) at pkcs11.c:4233
#5  0x00007f307aea974f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f307aea3c70 <find_cert_cb>, input=input@entry=0x7f3077f9f830, info=0x7f30680768e0, pin_info=pin_info@entry=0xdb6ed8, flags=8) at pkcs11.c:1631
#6  0x00007f307aeab1c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xd92e00 "pkcs11:", cert=0xdb6ec0, issuer=issuer@entry=0x7f3077f9f9a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#7  0x00007f307aef23aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdc6e70, url=0xd92e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f3077f9fad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#8  0x00007f307af02880 in gnutls_x509_trust_list_verify_crt2 (list=0xdc6e70, cert_list=0x7f3077f9fad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f3077f9fc44, func=0x0) at verify-high.c:1600
#9  0x000000000040152a in thread_func ()
#10 0x00007f307adef1c9 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f307ad17463 in clone () from /lib64/libc.so.6

Thread 1 (LWP 299717 "test-trust-list"):
#0  0x00007f307adfb51a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1  0x00007f307adf06c3 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
#2  0x00000000004017a0 in main ()
[Inferior 1 (process 299717) detached]
double free or corruption (out)
[New LWP 317514]
Error while reading shared library symbols for /lib64/libpthread.so.0:
Cannot find user-level thread for LWP 317514: generic error
0x00007f57b593f51a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0

Thread 2 (LWP 317514 "test-trust-list"):
#0  0x00007f57b5827e1f in wait4 () from /lib64/libc.so.6
#1  0x00007f57b57a50cb in do_system () from /lib64/libc.so.6
#2  0x000000000040132e in segv_handler ()
#3  <signal handler called>
#4  0x00007f57b5798292 in raise () from /lib64/libc.so.6
#5  0x00007f57b57818a4 in abort () from /lib64/libc.so.6
#6  0x00007f57b57dacd7 in __libc_message () from /lib64/libc.so.6
#7  0x00007f57b57e295c in malloc_printerr () from /lib64/libc.so.6
#8  0x00007f57b57e41e0 in _int_free () from /lib64/libc.so.6
#9  0x00007f57b57e7a58 in free () from /lib64/libc.so.6
#10 0x00007f57b57e7b62 in __malloc_arena_thread_freeres () from /lib64/libc.so.6
#11 0x00007f57b59331ef in start_thread () from /lib64/libpthread.so.0
#12 0x00007f57b585b463 in clone () from /lib64/libc.so.6

Thread 1 (LWP 317410 "test-trust-list"):
#0  0x00007f57b593f51a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1  0x00007f57b59346c3 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
#2  0x00000000004017a0 in main ()
[Inferior 1 (process 317410) detached]
double free or corruption (out)
[New LWP 317608]
[New LWP 317610]
[New LWP 317611]
[New LWP 317612]
[New LWP 317613]
[New LWP 317614]
[New LWP 317615]
[New LWP 317616]
Error while reading shared library symbols for /lib64/libpthread.so.0:
Cannot find user-level thread for LWP 317616: generic error
0x00007f2419c7251a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0

Thread 9 (LWP 317616 "test-trust-list"):
#0  0x00007f2419c6f240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2419c684f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f241962ef86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f2419d1b641 in scan_slots (slots=slots@entry=0x7f2417617650, nslots=nslots@entry=0x7f24176173a8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f2419d20707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f2417617830, info=0x7f23e8058290, pin_info=pin_info@entry=0x0, flags=8) at pkcs11.c:1574
#5  0x00007f2419d22995 in _gnutls_pkcs11_crt_is_known (url=0x7f2417617890 "\001\251\030\357\071\310M\210\215.\307\365", <incomplete sequence \305>, url@entry=0xdc8e00 "pkcs11:", cert=0xde38c0, flags=flags@entry=1538, trusted_cert=trusted_cert@entry=0x7f2417617998) at pkcs11.c:4614
#6  0x00007f2419d692d5 in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f2417617ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1205
#7  0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f2417617ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f2417617c44, func=0x0) at verify-high.c:1600
#8  0x000000000040152a in thread_func ()
#9  0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 8 (LWP 317615 "test-trust-list"):
#0  0x00007f2419c6f240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2419c684f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f241962ef86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f2419d1b641 in scan_slots (slots=slots@entry=0x7f2417e18650, nslots=nslots@entry=0x7f2417e183a8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f2419d20707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f2417e18830, info=0x7f240801e3b0, pin_info=pin_info@entry=0x0, flags=8) at pkcs11.c:1574
#5  0x00007f2419d22995 in _gnutls_pkcs11_crt_is_known (url=0x7f2417e18890 "k]\361\331\376\276#,\003", url@entry=0xdc8e00 "pkcs11:", cert=0xdebda0, flags=flags@entry=1538, trusted_cert=trusted_cert@entry=0x7f2417e18998) at pkcs11.c:4614
#6  0x00007f2419d692d5 in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f2417e18ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1205
#7  0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f2417e18ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f2417e18c44, func=0x0) at verify-high.c:1600
#8  0x000000000040152a in thread_func ()
#9  0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 7 (LWP 317614 "test-trust-list"):
#0  0x00007f2419c6f240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2419c684f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f2419639ffe in sys_C_FindObjectsFinal.lto_priv.0 () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f2419d1b09f in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f24186194e0, tinfo=tinfo@entry=0x7f2418619410, lib_info=lib_info@entry=0x7f2419e897b8 <providers+24>, input=input@entry=0x7f2418619830) at pkcs11.c:4224
#4  0x00007f2419d2074f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f2418619830, info=0x7f2404024e80, pin_info=pin_info@entry=0xdeced8, flags=8) at pkcs11.c:1631
#5  0x00007f2419d221c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xdc8e00 "pkcs11:", cert=0xdecec0, issuer=issuer@entry=0x7f24186199a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#6  0x00007f2419d693aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f2418619ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#7  0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f2418619ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f2418619c44, func=0x0) at verify-high.c:1600
#8  0x000000000040152a in thread_func ()
#9  0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 6 (LWP 317613 "test-trust-list"):
#0  0x00007f2419c6f240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2419c684f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f241962f280 in sys_C_GetTokenInfo () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f2419d2078e in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f2418e1a830, info=0x7f23f4027c60, pin_info=pin_info@entry=0x0, flags=8) at pkcs11.c:1585
#4  0x00007f2419d22be2 in _gnutls_pkcs11_crt_is_known (url=<optimized out>, url@entry=0xdc8e00 "pkcs11:", cert=0xdebda0, flags=flags@entry=1538, trusted_cert=trusted_cert@entry=0x7f2418e1a998) at pkcs11.c:4638
#5  0x00007f2419d692d5 in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f2418e1aad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1205
#6  0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f2418e1aad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f2418e1ac44, func=0x0) at verify-high.c:1600
#7  0x000000000040152a in thread_func ()
#8  0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 5 (LWP 317612 "test-trust-list"):
#0  0x00007f2419c6f240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2419c684f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f24196335dd in sys_C_GetAttributeValue () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f2419d2358d in pkcs11_override_cert_exts (sinfo=sinfo@entry=0x7f241961b4e0, spki=spki@entry=0x7f241961b100, der=der@entry=0x7f241961b0d0) at pkcs11x.c:134
#4  0x00007f2419d1b4a2 in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f241961b4e0, tinfo=tinfo@entry=0x7f241961b410, lib_info=lib_info@entry=0x7f2419e897b8 <providers+24>, input=input@entry=0x7f241961b830) at pkcs11.c:4233
#5  0x00007f2419d2074f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f241961b830, info=0x7f23f8014990, pin_info=pin_info@entry=0xdeced8, flags=8) at pkcs11.c:1631
#6  0x00007f2419d221c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xdc8e00 "pkcs11:", cert=0xdecec0, issuer=issuer@entry=0x7f241961b9a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#7  0x00007f2419d693aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f241961bad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#8  0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f241961bad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f241961bc44, func=0x0) at verify-high.c:1600
#9  0x000000000040152a in thread_func ()
#10 0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 4 (LWP 317611 "test-trust-list"):
#0  0x00007f2419c6f240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2419c684f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f2419639ffe in sys_C_FindObjectsFinal.lto_priv.0 () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f2419d1b09f in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f24166154e0, tinfo=tinfo@entry=0x7f2416615410, lib_info=lib_info@entry=0x7f2419e897b8 <providers+24>, input=input@entry=0x7f2416615830) at pkcs11.c:4224
#4  0x00007f2419d2074f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f2416615830, info=0x7f240c05f4b0, pin_info=pin_info@entry=0xdfb638, flags=8) at pkcs11.c:1631
#5  0x00007f2419d221c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xdc8e00 "pkcs11:", cert=0xdfb620, issuer=issuer@entry=0x7f24166159a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#6  0x00007f2419d693aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f2416615ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#7  0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f2416615ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f2416615c44, func=0x0) at verify-high.c:1600
#8  0x000000000040152a in thread_func ()
#9  0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 3 (LWP 317610 "test-trust-list"):
#0  0x00007f2419b5ae1f in wait4 () from /lib64/libc.so.6
#1  0x00007f2419ad80cb in do_system () from /lib64/libc.so.6
#2  0x000000000040132e in segv_handler ()
#3  <signal handler called>
#4  0x00007f2419acb292 in raise () from /lib64/libc.so.6
#5  0x00007f2419ab48a4 in abort () from /lib64/libc.so.6
#6  0x00007f2419b0dcd7 in __libc_message () from /lib64/libc.so.6
#7  0x00007f2419b1595c in malloc_printerr () from /lib64/libc.so.6
#8  0x00007f2419b171e0 in _int_free () from /lib64/libc.so.6
#9  0x00007f2419b1aa58 in free () from /lib64/libc.so.6
#10 0x00007f241963aa7b in p11_dict_remove () from /usr/lib64/pkcs11/p11-kit-trust.so
#11 0x00007f241963ab15 in loader_gone_file.lto_priv () from /usr/lib64/pkcs11/p11-kit-trust.so
#12 0x00007f241963d4c2 in loader_load_if_file () from /usr/lib64/pkcs11/p11-kit-trust.so
#13 0x00007f241963dc2f in loader_load_path.lto_priv () from /usr/lib64/pkcs11/p11-kit-trust.so
#14 0x00007f241963565a in sys_C_FindObjectsInit () from /usr/lib64/pkcs11/p11-kit-trust.so
#15 0x00007f2419d1af45 in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f2415e144f0, tinfo=tinfo@entry=0x7f2415e14420, lib_info=lib_info@entry=0x7f2419e897b8 <providers+24>, input=input@entry=0x7f2415e14830) at pkcs11.c:4170
#16 0x00007f2419d2074f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f2415e14830, info=0x7f2410027140, pin_info=pin_info@entry=0x0, flags=8) at pkcs11.c:1631
#17 0x00007f2419d22be2 in _gnutls_pkcs11_crt_is_known (url=<optimized out>, url@entry=0xdc8e00 "pkcs11:", cert=0x7f2410037d20, flags=flags@entry=1280, trusted_cert=trusted_cert@entry=0x0) at pkcs11.c:4638
#18 0x00007f2419d22dab in gnutls_pkcs11_crt_is_known (url=url@entry=0xdc8e00 "pkcs11:", cert=<optimized out>, flags=flags@entry=1280) at pkcs11.c:4703
#19 0x00007f2419d694de in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f2415e14ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1293
#20 0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f2415e14ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f2415e14c44, func=0x0) at verify-high.c:1600
#21 0x000000000040152a in thread_func ()
#22 0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 2 (LWP 317608 "test-trust-list"):
#0  0x00007f2419c6f240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2419c684f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f241963375f in sys_C_FindObjects () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f2419d23538 in pkcs11_override_cert_exts (sinfo=sinfo@entry=0x7f2414e124e0, spki=spki@entry=0x7f2414e12100, der=der@entry=0x7f2414e120d0) at pkcs11x.c:144
#4  0x00007f2419d1b4a2 in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f2414e124e0, tinfo=tinfo@entry=0x7f2414e12410, lib_info=lib_info@entry=0x7f2419e897b8 <providers+24>, input=input@entry=0x7f2414e12830) at pkcs11.c:4233
#5  0x00007f2419d2074f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f2419d1ac70 <find_cert_cb>, input=input@entry=0x7f2414e12830, info=0x7f24000548c0, pin_info=pin_info@entry=0xdeced8, flags=8) at pkcs11.c:1631
#6  0x00007f2419d221c3 in gnutls_pkcs11_get_raw_issuer (url=url@entry=0xdc8e00 "pkcs11:", cert=0xdecec0, issuer=issuer@entry=0x7f2414e129a0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=17410, flags@entry=17408) at pkcs11.c:4341
#7  0x00007f2419d693aa in _gnutls_pkcs11_verify_crt_status (tlist=tlist@entry=0xdfce70, url=0xdc8e00 "pkcs11:", certificate_list=certificate_list@entry=0x7f2414e12ad0, clist_size=<optimized out>, clist_size@entry=2, purpose=purpose@entry=0x0, flags=flags@entry=0, func=0x0) at verify.c:1245
#8  0x00007f2419d79880 in gnutls_x509_trust_list_verify_crt2 (list=0xdfce70, cert_list=0x7f2414e12ad0, cert_list_size=2, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f2414e12c44, func=0x0) at verify-high.c:1600
#9  0x000000000040152a in thread_func ()
#10 0x00007f2419c661c9 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f2419b8e463 in clone () from /lib64/libc.so.6

Thread 1 (LWP 317537 "test-trust-list"):
#0  0x00007f2419c7251a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1  0x00007f2419c676c3 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
#2  0x00000000004017a0 in main ()
[Inferior 1 (process 317537) detached]
malloc(): unsorted double linked list corrupted
[New LWP 341104]
[New LWP 341106]
[New LWP 341109]
[New LWP 341111]
Error while reading shared library symbols for /lib64/libpthread.so.0:
Cannot find user-level thread for LWP 341111: generic error
0x00007fc48fa7851a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0

Thread 5 (LWP 341111 "test-trust-list"):
#0  0x00007fc48f91a0bb in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007fc48f92104d in realloc () from /lib64/libc.so.6
#2  0x00007fc48faed00d in gnutls_realloc_fast (ptr=0x7fc484029f10, size=<optimized out>) at mem.c:56
#3  0x00007fc48fb7e96b in trust_list_add_compat (cert=<optimized out>, list=0xdfee70) at verify-high.c:301
#4  gnutls_x509_trust_list_get_issuer (list=list@entry=0xdfee70, cert=0xdeeec0, issuer=issuer@entry=0x7fc48b419aa0, flags=flags@entry=0) at verify-high.c:1142
#5  0x00007fc48fb7f1a0 in gnutls_x509_trust_list_verify_crt2 (list=0xdfee70, cert_list=0x7fc48b419ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7fc48b419c44, func=0x0) at verify-high.c:1498
#6  0x000000000040152a in thread_func ()
#7  0x00007fc48fa6c1c9 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fc48f994463 in clone () from /lib64/libc.so.6

Thread 4 (LWP 341109 "test-trust-list"):
#0  0x00007fc48f91a0bb in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007fc48f92104d in realloc () from /lib64/libc.so.6
#2  0x00007fc48faed00d in gnutls_realloc_fast (ptr=0x7fc484029f10, size=<optimized out>) at mem.c:56
#3  0x00007fc48fb7e96b in trust_list_add_compat (cert=<optimized out>, list=0xdfee70) at verify-high.c:301
#4  gnutls_x509_trust_list_get_issuer (list=list@entry=0xdfee70, cert=0xdeeec0, issuer=issuer@entry=0x7fc48c41baa0, flags=flags@entry=0) at verify-high.c:1142
#5  0x00007fc48fb7f1a0 in gnutls_x509_trust_list_verify_crt2 (list=0xdfee70, cert_list=0x7fc48c41bad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7fc48c41bc44, func=0x0) at verify-high.c:1498
#6  0x000000000040152a in thread_func ()
#7  0x00007fc48fa6c1c9 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fc48f994463 in clone () from /lib64/libc.so.6

Thread 3 (LWP 341106 "test-trust-list"):
#0  0x00007fc48f91a0bb in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007fc48f92104d in realloc () from /lib64/libc.so.6
#2  0x00007fc48faed00d in gnutls_realloc_fast (ptr=0x7fc484029f10, size=<optimized out>) at mem.c:56
#3  0x00007fc48fb7e96b in trust_list_add_compat (cert=<optimized out>, list=0xdfee70) at verify-high.c:301
#4  gnutls_x509_trust_list_get_issuer (list=list@entry=0xdfee70, cert=0xdfd620, issuer=issuer@entry=0x7fc48e41faa0, flags=flags@entry=0) at verify-high.c:1142
#5  0x00007fc48fb7f1a0 in gnutls_x509_trust_list_verify_crt2 (list=0xdfee70, cert_list=0x7fc48e41fad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7fc48e41fc44, func=0x0) at verify-high.c:1498
#6  0x000000000040152a in thread_func ()
#7  0x00007fc48fa6c1c9 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fc48f994463 in clone () from /lib64/libc.so.6

Thread 2 (LWP 341104 "test-trust-list"):
#0  0x00007fc48f960e1f in wait4 () from /lib64/libc.so.6
#1  0x00007fc48f8de0cb in do_system () from /lib64/libc.so.6
#2  0x000000000040132e in segv_handler ()
#3  <signal handler called>
#4  0x00007fc48f8d1292 in raise () from /lib64/libc.so.6
#5  0x00007fc48f8ba8a4 in abort () from /lib64/libc.so.6
#6  0x00007fc48f913cd7 in __libc_message () from /lib64/libc.so.6
#7  0x00007fc48f91b95c in malloc_printerr () from /lib64/libc.so.6
#8  0x00007fc48f91ef9c in _int_malloc () from /lib64/libc.so.6
#9  0x00007fc48f9211e5 in calloc () from /lib64/libc.so.6
#10 0x00007fc48f5a096c in _asn1_copy_structure3 () from /lib64/libtasn1.so.6
#11 0x00007fc48f5aa3ea in asn1_create_element () from /lib64/libtasn1.so.6
#12 0x00007fc48fb7798d in gnutls_x509_dn_init (dn=dn@entry=0x7fc48d41d950) at x509_dn.c:403
#13 0x00007fc48fb47f50 in gnutls_x509_rdn_get2 (idn=idn@entry=0xdfd648, str=str@entry=0x7fc48d41d9a0, flags=flags@entry=0) at dn.c:844
#14 0x00007fc48fb485d4 in _gnutls_x509_compare_raw_dn (dn1=dn1@entry=0xdfd658, dn2=dn2@entry=0xdfd648) at dn.c:1013
#15 0x00007fc48fb6da0e in is_issuer (issuer=0xdfd620, cert=0xdfd620) at verify.c:268
#16 gnutls_x509_crt_check_issuer (cert=0xdfd620, issuer=0xdfd620) at verify.c:872
#17 0x00007fc48fb7f61b in shorten_clist (clist_size=<optimized out>, certificate_list=<optimized out>, list=<optimized out>) at verify-high.c:895
#18 gnutls_x509_trust_list_verify_crt2 (list=0xdfee70, cert_list=0x7fc48d41dad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7fc48d41dc44, func=0x0) at verify-high.c:1540
#19 0x000000000040152a in thread_func ()
#20 0x00007fc48fa6c1c9 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fc48f994463 in clone () from /lib64/libc.so.6

Thread 1 (LWP 340962 "test-trust-list"):
#0  0x00007fc48fa7851a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1  0x00007fc48fa6d6c3 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
#2  0x00000000004017a0 in main ()
[Inferior 1 (process 340962) detached]
^[[1;2Brealloc(): invalid next size
[New LWP 344049]
[New LWP 344050]
[New LWP 344051]
[New LWP 344052]
[New LWP 344053]
[New LWP 344054]
[New LWP 344055]
[New LWP 344056]
[New LWP 344057]
[New LWP 344058]
Error while reading shared library symbols for /lib64/libpthread.so.0:
Cannot find user-level thread for LWP 344058: generic error
0x00007f8e81a9e51a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0

Thread 11 (LWP 344058 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e7cc3e680, nslots=nslots@entry=0x7f8e7cc3e3d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e7cc3e870, info=0x7f8e6c026620, pin_info=pin_info@entry=0xc71638, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc71620, issuer=issuer@entry=0x7f8e7cc3e9b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc71620, issuer=issuer@entry=0x7f8e7cc3eaa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e7cc3ead0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e7cc3ec44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 10 (LWP 344057 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e7d43f680, nslots=nslots@entry=0x7f8e7d43f3d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e7d43f870, info=0x7f8e64006080, pin_info=pin_info@entry=0xc62ed8, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc62ec0, issuer=issuer@entry=0x7f8e7d43f9b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc62ec0, issuer=issuer@entry=0x7f8e7d43faa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e7d43fad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e7d43fc44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 9 (LWP 344056 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e7dc40680, nslots=nslots@entry=0x7f8e7dc403d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e7dc40870, info=0x7f8e500121f0, pin_info=pin_info@entry=0xc71638, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc71620, issuer=issuer@entry=0x7f8e7dc409b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc71620, issuer=issuer@entry=0x7f8e7dc40aa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e7dc40ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e7dc40c44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 8 (LWP 344055 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e7e441680, nslots=nslots@entry=0x7f8e7e4413d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e7e441870, info=0x7f8e78040ff0, pin_info=pin_info@entry=0xc62ed8, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc62ec0, issuer=issuer@entry=0x7f8e7e4419b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc62ec0, issuer=issuer@entry=0x7f8e7e441aa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e7e441ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e7e441c44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 7 (LWP 344054 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e80c46680, nslots=nslots@entry=0x7f8e80c463d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e80c46870, info=0x7f8e68024420, pin_info=pin_info@entry=0xc71638, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc71620, issuer=issuer@entry=0x7f8e80c469b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc71620, issuer=issuer@entry=0x7f8e80c46aa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e80c46ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e80c46c44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 6 (LWP 344053 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e81447680, nslots=nslots@entry=0x7f8e814473d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e81447870, info=0x7f8e5404a570, pin_info=pin_info@entry=0xc62ed8, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc62ec0, issuer=issuer@entry=0x7f8e814479b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc62ec0, issuer=issuer@entry=0x7f8e81447aa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e81447ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e81447c44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 5 (LWP 344052 "test-trust-list"):
#0  0x00007f8e819400bb in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f8e819432ca in _int_free () from /lib64/libc.so.6
#2  0x00007f8e81946a58 in free () from /lib64/libc.so.6
#3  0x00007f8e81466a7b in p11_dict_remove () from /usr/lib64/pkcs11/p11-kit-trust.so
#4  0x00007f8e81466b15 in loader_gone_file.lto_priv () from /usr/lib64/pkcs11/p11-kit-trust.so
#5  0x00007f8e814694c2 in loader_load_if_file () from /usr/lib64/pkcs11/p11-kit-trust.so
#6  0x00007f8e81469c2f in loader_load_path.lto_priv () from /usr/lib64/pkcs11/p11-kit-trust.so
#7  0x00007f8e8146165a in sys_C_FindObjectsInit () from /usr/lib64/pkcs11/p11-kit-trust.so
#8  0x00007f8e81b46f45 in find_cert_cb (module=<optimized out>, sinfo=sinfo@entry=0x7f8e80445520, tinfo=tinfo@entry=0x7f8e80445450, lib_info=lib_info@entry=0x7f8e81cb57b8 <providers+24>, input=input@entry=0x7f8e80445870) at pkcs11.c:4170
#9  0x00007f8e81b4c74f in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e80445870, info=0x7f8e5c002f20, pin_info=pin_info@entry=0xc71638, flags=8) at pkcs11.c:1631
#10 0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc71620, issuer=issuer@entry=0x7f8e804459b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#11 0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc71620, issuer=issuer@entry=0x7f8e80445aa0, flags=flags@entry=0) at verify-high.c:1115
#12 0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e80445ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e80445c44, func=0x0) at verify-high.c:1498
#13 0x000000000040152a in thread_func ()
#14 0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 4 (LWP 344051 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e7fc44680, nslots=nslots@entry=0x7f8e7fc443d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e7fc44870, info=0x7f8e60026a20, pin_info=pin_info@entry=0xc62ed8, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc62ec0, issuer=issuer@entry=0x7f8e7fc449b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc62ec0, issuer=issuer@entry=0x7f8e7fc44aa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e7fc44ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e7fc44c44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 3 (LWP 344050 "test-trust-list"):
#0  0x00007f8e81a9b240 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f8e81a944f0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f8e8145af86 in sys_C_GetSlotList () from /usr/lib64/pkcs11/p11-kit-trust.so
#3  0x00007f8e81b47641 in scan_slots (slots=slots@entry=0x7f8e7f443680, nslots=nslots@entry=0x7f8e7f4433d8, p=<optimized out>, p=<optimized out>) at pkcs11.c:222
#4  0x00007f8e81b4c707 in _pkcs11_traverse_tokens (find_func=find_func@entry=0x7f8e81b46c70 <find_cert_cb>, input=input@entry=0x7f8e7f443870, info=0x7f8e5800d8a0, pin_info=pin_info@entry=0xc71638, flags=8) at pkcs11.c:1574
#5  0x00007f8e81b4e1c3 in gnutls_pkcs11_get_raw_issuer (url=0xc3ee00 "pkcs11:", cert=cert@entry=0xc71620, issuer=issuer@entry=0x7f8e7f4439b0, fmt=fmt@entry=GNUTLS_X509_FMT_DER, flags=1026, flags@entry=1024) at pkcs11.c:4341
#6  0x00007f8e81ba487a in gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc71620, issuer=issuer@entry=0x7f8e7f443aa0, flags=flags@entry=0) at verify-high.c:1115
#7  0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e7f443ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e7f443c44, func=0x0) at verify-high.c:1498
#8  0x000000000040152a in thread_func ()
#9  0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 2 (LWP 344049 "test-trust-list"):
#0  0x00007f8e81986e1f in wait4 () from /lib64/libc.so.6
#1  0x00007f8e819040cb in do_system () from /lib64/libc.so.6
#2  0x000000000040132e in segv_handler ()
#3  <signal handler called>
#4  0x00007f8e818f7292 in raise () from /lib64/libc.so.6
#5  0x00007f8e818e08a4 in abort () from /lib64/libc.so.6
#6  0x00007f8e81939cd7 in __libc_message () from /lib64/libc.so.6
#7  0x00007f8e8194195c in malloc_printerr () from /lib64/libc.so.6
#8  0x00007f8e81945c3c in _int_realloc () from /lib64/libc.so.6
#9  0x00007f8e81946f66 in realloc () from /lib64/libc.so.6
#10 0x00007f8e81b1300d in gnutls_realloc_fast (ptr=0x7f8e70059a10, size=<optimized out>) at mem.c:56
#11 0x00007f8e81ba496b in trust_list_add_compat (cert=<optimized out>, list=0xc72e70) at verify-high.c:301
#12 gnutls_x509_trust_list_get_issuer (list=list@entry=0xc72e70, cert=0xc62ec0, issuer=issuer@entry=0x7f8e7ec42aa0, flags=flags@entry=0) at verify-high.c:1142
#13 0x00007f8e81ba51a0 in gnutls_x509_trust_list_verify_crt2 (list=0xc72e70, cert_list=0x7f8e7ec42ad0, cert_list_size=<optimized out>, data=<optimized out>, elements=<optimized out>, flags=0, voutput=0x7f8e7ec42c44, func=0x0) at verify-high.c:1498
#14 0x000000000040152a in thread_func ()
#15 0x00007f8e81a921c9 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f8e819ba463 in clone () from /lib64/libc.so.6

Thread 1 (LWP 343928 "test-trust-list"):
#0  0x00007f8e81a9e51a in __futex_abstimed_wait_common64 () from /lib64/libpthread.so.0
#1  0x00007f8e81a936c3 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
#2  0x00000000004017a0 in main ()
Comment 34 Milan Crha 2021-04-19 16:50:39 UTC 
It seems to me it's easier to reproduce when the machine is busy, like when compiling something large in the background, or when running this test application in multiple terminals at the same time. It's only a feeling, I can be wrong.
Comment 35 Michael Catanzaro 2021-04-19 17:33:55 UTC 
(In reply to Daiki Ueno from comment #31)
> Sorry for chiming in late, but I'm wondering if this might be an issue in
> GnuTLS, as it has a thread-safety promise:
> https://www.gnutls.org/manual/gnutls.html#Thread-safety
> "The GnuTLS library is thread safe by design, meaning that objects of the
> library such as TLS sessions, can be safely divided across threads as long
> as a single thread accesses a single object"

Well glib-networking is using the same gnutls_x509_trust_list_t across multiple threads at the same time, which violates GnuTLS's rule that the object should not be shared across threads. Yes?

It might still be a GnuTLS bug if the same crash can be triggered in some other way that doesn't violate GnuTLS's threadsafety rules. E.g. I guess the PKCS#11 code is somehow intended to be used across multiple threads, since it has locking.
Comment 36 Daiki Ueno 2021-04-20 13:44:28 UTC 
(In reply to Michael Catanzaro from comment #35)
> (In reply to Daiki Ueno from comment #31)
> > Sorry for chiming in late, but I'm wondering if this might be an issue in
> > GnuTLS, as it has a thread-safety promise:
> > https://www.gnutls.org/manual/gnutls.html#Thread-safety
> > "The GnuTLS library is thread safe by design, meaning that objects of the
> > library such as TLS sessions, can be safely divided across threads as long
> > as a single thread accesses a single object"
> 
> Well glib-networking is using the same gnutls_x509_trust_list_t across
> multiple threads at the same time, which violates GnuTLS's rule that the
> object should not be shared across threads. Yes?

Yeah, sorry, I obviously misread the last sentence, though the above quote follows: "read-only access to objects, for example the credentials holding structures, is also thread-safe" and this specific usage should be considered read-only in my opinion (i.e., it doesn't involve any side-effect).
Comment 37 Michael Catanzaro 2021-04-20 15:34:48 UTC 
Hm, OK then. This certainly seems like read-only usage. Shall I reassign this to GnuTLS component?

I guess I should reconsider how much of https://gitlab.gnome.org/GNOME/glib-networking/-/commit/8c034ff04d865a1b2c4dbb93e6e1c47278997b09 is still needed if read-only use of the gnutls_x509_trust_list_t across threads is considered safe.
Comment 38 Milan Crha 2021-04-21 07:06:13 UTC 
*** Bug 1951690 has been marked as a duplicate of this bug. ***
Comment 39 Michael Catanzaro 2021-04-22 14:28:07 UTC 
(In reply to Michael Catanzaro from comment #37)
> Hm, OK then. This certainly seems like read-only usage. Shall I reassign
> this to GnuTLS component?

Hi Daiki, feel free to create your own separate GnuTLS bug for this issue.

> I guess I should reconsider how much of
> https://gitlab.gnome.org/GNOME/glib-networking/-/commit/
> 8c034ff04d865a1b2c4dbb93e6e1c47278997b09 is still needed if read-only use of
> the gnutls_x509_trust_list_t across threads is considered safe.

Turns out we need the mutex there anyway to guard the operation cancellable, so it cannot be removed even if GnuTLS is fixed. Otherwise, each subsequent concurrent certificate verification will cause the cancellable of the previous verification to be clobbered. Oops. So my original fix should be shipped without change.
Comment 40 Daiki Ueno 2021-04-22 15:52:06 UTC 
Sure, I agree that that would be the safest approach. I've filed a GnuTLS issue at:
https://gitlab.com/gnutls/gnutls/-/issues/1212
Comment 41 Fedora Update System 2021-04-22 18:56:24 UTC 
FEDORA-2021-f186e0bfa4 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-f186e0bfa4
Comment 42 Fedora Update System 2021-04-23 19:49:26 UTC 
FEDORA-2021-f186e0bfa4 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-f186e0bfa4`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-f186e0bfa4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 43 Thomas Citharel 2021-04-26 07:28:27 UTC 
Similar problem has been detected:

Closing my session

reporter:       libreport-2.14.0
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/evolution-calendar-factory.service
cmdline:        /usr/libexec/evolution-calendar-factory
crash_function: gnutls_realloc_fast
dso_list:       /usr/libexec/evolution-calendar-factory evolution-data-server-3.40.0-3.fc34.x86_64 (Fedora Project) 1617087213
executable:     /usr/libexec/evolution-calendar-factory
journald_cursor: s=27bbc631282946ec9b441c95279315b7;i=1daf0;b=370028ddfd5140078e9af5b8e6d147df;m=26f84f52;t=5c0dae8aaf9a1;x=5390b6284f316ba9
kernel:         5.11.15-300.fc34.x86_64
package:        evolution-data-server-3.40.0-3.fc34
reason:         evolution-calendar-factory killed by SIGABRT
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000
Comment 44 Fedora Update System 2021-04-27 00:30:10 UTC 
FEDORA-2021-f186e0bfa4 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 45 Michael Catanzaro 2021-05-26 16:40:39 UTC 
OK, so this bug got somewhat repurposed by accident. The backtrace here did reveal the safety issue when calling gnutls_x509_trust_list_verify_crt() on multiple threads at once, so it did match bug #1944854 and marking that bug as a duplicate of this one was a reasonable thing to do.

But apparently there were actually two different bugs here, and the bug that caused the original crash was not fixed. There is a new report for it now, bug #1964703. We can continue there.
Comment 46 Michael Catanzaro 2021-05-26 16:48:39 UTC 
Well maybe not, I got confused because ABRT seems to have stopped reporting backtraces for multiple threads, which of course makes solving a threadsafety bug like this impossible and tricked me into thinking that GTlsDatabase was not involved in bug #1964703.
Comment 47 Michael Catanzaro 2021-05-26 16:50:31 UTC 
*** Bug 1964703 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.