Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1973026 - Thread debugging broken in glibc-2.33.9000-18.fc35
Summary: Thread debugging broken in glibc-2.33.9000-18.fc35
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: glibc
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Florian Weimer
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-17 06:43 UTC by Kevin Buettner
Modified: 2021-06-17 20:58 UTC (History)
11 users (show)

Fixed In Version: glibc-2.33.9000-20.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-17 15:59:02 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Kevin Buettner 2021-06-17 06:43:29 UTC 
Description of problem:

I updated one of my x86-64 rawhide VMs and then installed the following packages:

glibc-all-langpacks-2.33.9000-18.fc35.x86_64
glibc-common-2.33.9000-18.fc35.x86_64
glibc-langpack-en-2.33.9000-18.fc35.x86_64
glibc-headers-x86-2.33.9000-18.fc35.noarch
glibc-devel-2.33.9000-18.fc35.x86_64
glibc-doc-2.33.9000-18.fc35.noarch
glibc-debuginfo-2.33.9000-18.fc35.x86_64
glibc-2.33.9000-18.fc35.i686
glibc-devel-2.33.9000-18.fc35.i686
glibc-static-2.33.9000-18.fc35.x86_64
glibc-common-debuginfo-2.33.9000-18.fc35.x86_64
glibc-2.33.9000-18.fc35.x86_64

I fetched the RPMs for those packages from this page:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1772416

When I attempt to debug one of the gdb.threads test cases, I see:

[kev@rawhide-1 gdb]$ ./gdb -q testsuite/outputs/gdb.threads/tls/tls 
Reading symbols from testsuite/outputs/gdb.threads/tls/tls...
(gdb) start
Temporary breakpoint 1 at 0x4015fd: file /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c, line 231.
Starting program: /mesquite2/sourceware-git/rawhide-glibc234/bld/gdb/testsuite/outputs/gdb.threads/tls/tls 

Temporary breakpoint 1, main () at /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c:231
231	   do_pass ();
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007ffff7fcc090  0x00007ffff7ff0fd6  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff7e2e2f0  0x00007ffff7f27eb2  Yes         /lib64/libstdc++.so.6
0x00007ffff7cc1390  0x00007ffff7d30fd8  Yes         /lib64/libm.so.6
0x00007ffff7c9a5f0  0x00007ffff7cab955  Yes         /lib64/libgcc_s.so.1
0x00007ffff7acd700  0x00007ffff7c31d5d  Yes (*)     /lib64/libc.so.6
(*): Shared library is missing debugging information.

Note that the output from "info shared" indicates that /lib64/libc.so.6 is missing debugging information.  Also, there should be a libthread_db initialization message; it is missing from the above session.

Poking around on the filesystem, I find:

[kev@rawhide-1 gdb]$ ls -l {,/usr/lib/debug}/usr/lib64/libc[.-]*
-rw-r--r-- 1 root root 15319958 Jun 16 01:20 /usr/lib64/libc.a
-rw-r--r-- 1 root root      253 Jun 16 01:16 /usr/lib64/libc.so
-rwxr-xr-x 1 root root  2023736 Jun 16 01:20 /usr/lib64/libc.so.6
-rw-r--r-- 1 root root  6936944 Jun 16 01:20 /usr/lib/debug/usr/lib64/libc.so.6-2.33.9000-18.fc35.x86_64.debug

Now, let's look at the build id using eu-unstrip:

[kev@rawhide-1 gdb]$ eu-unstrip --list-only -e /usr/lib64/libc.so.6 
0+0x1f30d0 47b3653bd8d4a01f90cfcfdb63f57b2451bb6572@0x3b0 /usr/lib64/libc.so.6 - 
[kev@rawhide-1 gdb]$ eu-unstrip --list-only -e /usr/lib/debug/usr/lib64/libc.so.6-2.33.9000-18.fc35.x86_64.debug
0+0x1f30d0 dd8443cd4b5b0eb7191cbf884725c3706bebc519@0x3b0 /usr/lib/debug/usr/lib64/libc.so.6-2.33.9000-18.fc35.x86_64.debug .

It's 47b3653bd8d4a01f90cfcfdb63f57b2451bb6572 for the library
and dd8443cd4b5b0eb7191cbf884725c3706bebc519 for the .debug file.

Note that the build ids don't match.

Version-Release number of selected component (if applicable):

See above.

How reproducible:

Always.

Steps to Reproduce:

See above.

Actual results:

See above.

Expected results:

On another rawhide machine with glibc-2.33.9000-12.fc35.x86_64 (and related packages) installed, I see:

[kev@rawhide-glibc-test ~]$ ls -l {,/usr/lib/debug}/usr/lib64/libc[.-]*
-rwxr-xr-x 1 root root  2054848 Jun  3 05:13 /usr/lib64/libc-2.33.9000.so
-rw-r--r-- 1 root root 15287622 Jun  3 05:13 /usr/lib64/libc.a
-rw-r--r-- 1 root root      253 Jun  3 05:07 /usr/lib64/libc.so
lrwxrwxrwx 1 root root       17 Jun  3 05:08 /usr/lib64/libc.so.6 -> libc-2.33.9000.so
-rw-r--r-- 1 root root  6935592 Jun  3 05:13 /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug
-rw-r--r-- 1 root root 36054446 Apr 15  2020 /usr/lib/debug/usr/lib64/libc.a

eu-unstrip shows matching build ids for libc and the debuginfo file:

[kev@rawhide-glibc-test ~]$ eu-unstrip --list-only -e /usr/lib64/libc-2.33.9000.so
0+0x1f30b0 8502c0eadde138b04c06d3f8b311c69d6be65669@0x3b0 /usr/lib64/libc-2.33.9000.so /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug 
[kev@rawhide-glibc-test ~]$ eu-unstrip --list-only -e /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug
0+0x1f30b0 8502c0eadde138b04c06d3f8b311c69d6be65669@0x3b0 /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug . 

Also gdb shows that it's loaded / initialized libthread_db:

[kev@rawhide-glibc-test gdb]$ ./gdb -q testsuite/outputs/gdb.threads/tls/tls
Reading symbols from testsuite/outputs/gdb.threads/tls/tls...
(gdb) start
Temporary breakpoint 1 at 0x4015fd: file /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c, line 231.
Starting program: /mesquite2/sourceware-git/rawhide-glibc234/bld/gdb/testsuite/outputs/gdb.threads/tls/tls 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Temporary breakpoint 1, main () at /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c:231
231	   do_pass ();
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007ffff7fcc090  0x00007ffff7ff0fc6  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff7e2e2f0  0x00007ffff7f27eb2  Yes         /lib64/libstdc++.so.6
0x00007ffff7cc1390  0x00007ffff7d30fd8  Yes         /lib64/libm.so.6
0x00007ffff7c9a5f0  0x00007ffff7cab955  Yes         /lib64/libgcc_s.so.1
0x00007ffff7acd700  0x00007ffff7c31e9d  Yes         /lib64/libc.so.6

Note this gdb session printed messages regarding libthread_db; these messages are missing when debugging on the machine w/ glibc-2.33.9000-18.fc35 installed.

Also note that "info shared" shows that the symbols are loaded and has no asterisk indicating missing debug info.
Comment 1 Kevin Buettner 2021-06-17 06:52:00 UTC 
In case the original report isn't convincing enough...

[kev@rawhide-glibc-test gdb]$ rpm -q -a | grep -e ^glibc-2.33 | grep x86
glibc-2.33.9000-12.fc35.x86_64
[kev@rawhide-glibc-test gdb]$ make check TESTS=gdb.threads/tls.exp
...
Running /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.exp ...

		=== gdb Summary ===

# of expected passes		77
# of known failures		1
...

VERSUS:

[kev@rawhide-1 gdb]$ rpm -q -a | grep -e ^glibc-2.33 | grep x86
glibc-2.33.9000-18.fc35.x86_64
[kev@rawhide-1 gdb]$ make check TESTS=gdb.threads/tls.exp
...
Running /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.exp ...
FAIL: gdb.threads/tls.exp: at least one th in spin while stopped at first th
FAIL: gdb.threads/tls.exp: first thread local storage
FAIL: gdb.threads/tls.exp: first get symbol value without frame
FAIL: gdb.threads/tls.exp: first another thread local storage
FAIL: gdb.threads/tls.exp: at least one th in spin while stopped at second th
FAIL: gdb.threads/tls.exp: second thread local storage
FAIL: gdb.threads/tls.exp: second get symbol value without frame
FAIL: gdb.threads/tls.exp: second another thread local storage
FAIL: gdb.threads/tls.exp: at least one th in spin while stopped at third th
FAIL: gdb.threads/tls.exp: third thread local storage
FAIL: gdb.threads/tls.exp: third get symbol value without frame
FAIL: gdb.threads/tls.exp: third another thread local storage
FAIL: gdb.threads/tls.exp: get number of threads
FAIL: gdb.threads/tls.exp: no thread backtrace reported spin (vsyscall kernel problem?)
FAIL: gdb.threads/tls.exp: mess at end
FAIL: gdb.threads/tls.exp: p a_thread_local
FAIL: gdb.threads/tls.exp: p file2_thread_local
FAIL: gdb.threads/tls.exp: p a_thread_local second time

		=== gdb Summary ===

# of expected passes		25
# of unexpected failures	18
# of known failures		1
Comment 2 Florian Weimer 2021-06-17 08:20:48 UTC 
Okay, my attempt with a minimal symtab clearly does not work. I'm going to revert this. Sorry about that.

However, libthread_db should be loaded even if glibc-debuginfo is not installed. Would it possible to change GDB to do that?
Comment 3 Fedora Update System 2021-06-17 10:49:30 UTC 
FEDORA-2021-783138c496 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 4 Florian Weimer 2021-06-17 15:15:55 UTC 
So it seems that pthread_create is actually visible to GDB without debuginfo (although I added it to .symtab as well), but we have hidden a bunch of stuff that libthread_db needs and can't see without at least a symbol table. I will investigate moving those symbols to .dynsym and if that enables libthread_db without debuginfo.
Comment 5 Kevin Buettner 2021-06-17 15:35:28 UTC 
(In reply to Florian Weimer from comment #2)

> However, libthread_db should be loaded even if glibc-debuginfo is not
> installed. Would it possible to change GDB to do that?

GDB used to work without glibc's debuginfo present.  I'm not sure when or why this changed, but I'll investigate.
Comment 6 Kevin Buettner 2021-06-17 15:59:02 UTC 
I've tested glibc-2.33.9000-21.fc35. Thread debugging is working again!
Comment 7 Fedora Update System 2021-06-17 20:58:30 UTC 
FEDORA-2021-72b593648e has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.