Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1721483 - Python 3: test_gdb fails on aarch64 and ppc64le: Missing separate debuginfo for /lib/ld-linux-aarch64.so.1 /lib64/ld64.so.2
Summary: Python 3: test_gdb fails on aarch64 and ppc64le: Missing separate debuginfo f...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Python Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1722178
Blocks: F31FTBFS PYTHON38 1732841
TreeView+ depends on / blocked
 
Reported: 2019-06-18 11:44 UTC by Miro Hrončok
Modified: 2019-07-31 22:40 UTC (History)
13 users (show)

Fixed In Version: gdb-8.3.50.20190610-17.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-22 22:39:17 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
root.log from Koji (aarch64) (97.70 KB, text/plain)
2019-06-18 11:45 UTC, Miro Hrončok
no flags Details
build.log from Koji (aarch64) (1.93 MB, text/plain)
2019-06-18 11:45 UTC, Miro Hrončok
no flags Details
root.log from Koji (ppc64le) (97.60 KB, text/plain)
2019-06-18 11:46 UTC, Miro Hrončok
no flags Details
build.log from Koji (ppc64le) (1.97 MB, text/plain)
2019-06-18 11:46 UTC, Miro Hrončok
no flags Details

Description Miro Hrončok 2019-06-18 11:44:47 UTC
This is Python 3.8.0b1 on aarch64:

======================================================================
FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with NULL ob_type is handled gracefully
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 537, in test_NULL_ob_type
    self.assertSane('id(42)',
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 508, in assertSane
    self.get_gdb_repr(source,
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 273, in get_gdb_repr
    gdb_output = self.get_stack_trace(source, breakpoint=BREAKPOINT_FN,
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 253, in get_stack_trace
    self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ['Missing separate debuginfo for /lib/ld-l[975 chars]bug"] != []
First list contains 12 additional elements.
First extra element 0:
'Missing separate debuginfo for /lib/ld-linux-aarch64.so.1'
Diff is 1099 characters long. Set self.maxDiff to None to see it.

Other test failures with the same problem:

FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests)
FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests)
FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests)
FAIL: test_bytes (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests)
FAIL: test_dicts (test.test_gdb.PrettyPrintTests)
FAIL: test_exceptions (test.test_gdb.PrettyPrintTests)
FAIL: test_frames (test.test_gdb.PrettyPrintTests)
FAIL: test_frozensets (test.test_gdb.PrettyPrintTests)
FAIL: test_getting_backtrace (test.test_gdb.PrettyPrintTests)
FAIL: test_int (test.test_gdb.PrettyPrintTests)
FAIL: test_lists (test.test_gdb.PrettyPrintTests)
FAIL: test_modern_class (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_sets (test.test_gdb.PrettyPrintTests)
FAIL: test_singletons (test.test_gdb.PrettyPrintTests)
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests)
FAIL: test_truncation (test.test_gdb.PrettyPrintTests)
FAIL: test_tuples (test.test_gdb.PrettyPrintTests)
FAIL: test_down_at_bottom (test.test_gdb.StackNavigationTests)
FAIL: test_up_at_top (test.test_gdb.StackNavigationTests)
FAIL: test_threads (test.test_gdb.PyBtTests)
FAILED (failures=30, skipped=16)

1 test failed again:
    test_gdb
== Tests result: FAILURE then FAILURE ==

This is ppc64le:

======================================================================
FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with NULL ob_type is handled gracefully
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 537, in test_NULL_ob_type
    self.assertSane('id(42)',
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 508, in assertSane
    self.get_gdb_repr(source,
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 273, in get_gdb_repr
    gdb_output = self.get_stack_trace(source, breakpoint=BREAKPOINT_FN,
  File "/builddir/build/BUILD/Python-3.8.0b1/Lib/test/test_gdb.py", line 253, in get_stack_trace
    self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ['Missing separate debuginfo for /lib64/ld[965 chars]bug"] != []
First list contains 12 additional elements.
First extra element 0:
'Missing separate debuginfo for /lib64/ld64.so.2'
Diff is 1089 characters long. Set self.maxDiff to None to see it.

Other test failures with the same problem:

FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests)
FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests)
FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests)
FAIL: test_bytes (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests)
FAIL: test_dicts (test.test_gdb.PrettyPrintTests)
FAIL: test_exceptions (test.test_gdb.PrettyPrintTests)
FAIL: test_frames (test.test_gdb.PrettyPrintTests)
FAIL: test_frozensets (test.test_gdb.PrettyPrintTests)
FAIL: test_getting_backtrace (test.test_gdb.PrettyPrintTests)
FAIL: test_int (test.test_gdb.PrettyPrintTests)
FAIL: test_lists (test.test_gdb.PrettyPrintTests)
FAIL: test_modern_class (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_sets (test.test_gdb.PrettyPrintTests)
FAIL: test_singletons (test.test_gdb.PrettyPrintTests)
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests)
FAIL: test_truncation (test.test_gdb.PrettyPrintTests)
FAIL: test_tuples (test.test_gdb.PrettyPrintTests)
FAIL: test_down_at_bottom (test.test_gdb.StackNavigationTests)
FAIL: test_up_at_top (test.test_gdb.StackNavigationTests)
FAIL: test_threads (test.test_gdb.PyBtTests)
FAILED (failures=30, skipped=16)
1 test failed again:
    test_gdb
== Tests result: FAILURE then FAILURE ==

Comment 1 Miro Hrončok 2019-06-18 11:45:23 UTC
Created attachment 1581664 [details]
root.log from Koji (aarch64)

Comment 2 Miro Hrončok 2019-06-18 11:45:50 UTC
Created attachment 1581665 [details]
build.log from Koji (aarch64)

Comment 3 Miro Hrončok 2019-06-18 11:46:19 UTC
Created attachment 1581666 [details]
root.log from Koji (ppc64le)

Comment 4 Miro Hrončok 2019-06-18 11:46:57 UTC
Created attachment 1581667 [details]
build.log from Koji (ppc64le)

Logs attached. Koji scratch build at https://koji.fedoraproject.org/koji/taskinfo?taskID=35612158

Comment 5 Miro Hrončok 2019-06-18 11:48:37 UTC
The spec file has:

# Expensive optimizations (mainly, profile-guided optimizations)
%ifarch %{ix86} x86_64
%bcond_without optimizations
%else
# On some architectures, the optimized build takes tens of hours, possibly
# longer than Koji's 24-hour timeout. Disable optimizations here.
%bcond_with optimizations
%endif

So neither aarch64 or ppc64le has profile-guided optimizations, but there are other arches (armv7hl, s390x) where they are also not enabled.

Comment 6 Miro Hrončok 2019-06-18 11:50:46 UTC
Dependency changes since last build:

 glibc-devel
    2.29.9000-25.fc31
    2.29.9000-28.fc31

 mesa-libGL-devel
    19.1.0~rc4-1.fc31
    19.1.0-1.fc31

 libuuid-devel
    2.34-0.2.fc31
    2.34-1.fc31

 systemtap-sdt-devel
    4.2-0.20190513git...
    4.2-0.20190613git...

 libappstream-glib
    0.7.15-1.fc31
    0.7.15-3.fc31

 gdb
    8.3.50.20190601-15.fc31
    8.3.50.20190610-16.fc31

 redhat-rpm-config
    130-1.fc31
    132-1.fc31

 gcc-c++
    9.1.1-1.fc31
    9.1.1-2.fc31

 glibc-all-langpacks
    2.29.9000-25.fc31
    2.29.9000-28.fc31

 bluez-libs-devel
    5.50-7.fc31
    5.50-8.fc31

Comment 7 Miro Hrončok 2019-06-18 11:58:18 UTC
Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is even possible.

Also, running a python3 scratchbuild to see if there is a similar issue.

Comment 8 Miro Hrončok 2019-06-18 11:59:02 UTC
Same problem with the python3 package: https://koji.fedoraproject.org/koji/taskinfo?taskID=35613210

Comment 9 Miro Hrončok 2019-06-18 12:04:23 UTC
(In reply to Miro Hrončok from comment #7)
> Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is
> even possible.

DEBUG util.py:585:  BUILDSTDERR: No matching package to install: 'glibc-debuginfo'

Comment 10 Miro Hrončok 2019-06-18 12:05:33 UTC
https://apps.fedoraproject.org/koschei/package/python3 says:

glibc-devel was updated from 2.29.9000-27.fc31 to 2.29.9000-28.fc31

Florian, any idea what is going on here?

Comment 11 Miro Hrončok 2019-06-18 12:27:06 UTC
(Oour other Python versions appear to skip test_gdb either entirely or at least on power and aarch64.)

Comment 12 Florian Weimer 2019-06-18 12:27:52 UTC
(In reply to Miro Hrončok from comment #9)
> (In reply to Miro Hrončok from comment #7)
> > Will try to BR glibc-debuginfo to workaround this, but I'm not sure that is
> > even possible.
> 
> DEBUG util.py:585:  BUILDSTDERR: No matching package to install:
> 'glibc-debuginfo'

This is a missing Koji feature: <https://pagure.io/koji/issue/540>

(In reply to Miro Hrončok from comment #10)
> https://apps.fedoraproject.org/koschei/package/python3 says:
> 
> glibc-devel was updated from 2.29.9000-27.fc31 to 2.29.9000-28.fc31
> 
> Florian, any idea what is going on here?

There's no explicit change in the glibc spec file related to debugging information.

I will try to get an aarch64 or ppc64le system to investigate.  In general, you will need to run glibc-debuginfo to run such tests.

Comment 13 Miro Hrončok 2019-06-18 12:36:14 UTC
I now also got this on x86_64:

======================================================================
FAIL: test_locals_after_up (test.test_gdb.PyLocalsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
BUILDSTDERR:   File "/builddir/build/BUILD/Python-3.7.3/Lib/test/test_gdb.py", line 964, in test test_gdb failed
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Da5jiz (%check)
test_locals_after_up
    cmds_after_breakpoint=['py-up', 'py-up', 'py-locals'])
  File "/builddir/build/BUILD/Python-3.7.3/Lib/test/test_gdb.py", line 250, in get_stack_trace
    self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ['Missing separate debuginfo for /lib64/ld[976 chars]bug"] != []
First list contains 12 additional elements.
First extra element 0:
'Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2'
Diff is 1100 characters long. Set self.maxDiff to None to see it.
----------------------------------------------------------------------

Comment 14 Miro Hrončok 2019-06-19 12:34:09 UTC
(In reply to Miro Hrončok from comment #5)
> So neither aarch64 or ppc64le has profile-guided optimizations, but there
> are other arches (armv7hl, s390x) where they are also not enabled.


But there is:


 %ifarch %{arm} s390x
    -x test_gdb

So it can be that the test fails on non-profile-guided optimizations builds. On profile-guided optimizations builds, it is skipped already by logic in upstream tests.

Comment 15 Petr Viktorin 2019-06-19 13:44:18 UTC
We can fix this by adding to the ignored warnings. Unfortunately, the "dnf" line is Fedora-specific, so it can't go upstream like this:


diff -U3 -r Python-3.7.3-orig/Lib/test/test_gdb.py Python-3.7.3/Lib/test/test_gdb.py
--- Python-3.7.3-orig/Lib/test/test_gdb.py	2019-03-25 21:21:05.000000000 +0100
+++ Python-3.7.3/Lib/test/test_gdb.py	2019-06-19 15:39:49.312801739 +0200
@@ -230,6 +230,9 @@
             'BFD: ',
             # ignore all warnings
             'warning: ',
+            # ignore "Missing separate debuginfo" warnings
+            'Missing separate debuginfo for ',
+            "Try: dnf --enablerepo='*debug*' install ",
             )
         for line in errlines:
             if not line:

Comment 16 Petr Viktorin 2019-06-19 14:44:35 UTC
Ah, got it: There's an undocumented setting in GDB, `build-id-verbose` that can disable the "Missing separate debuginfo" messages.
Given that these tests tend to match exact wording, I don't think an undocumented setting would be a problem. Victor, what do you think?


diff -U3 -r Python-3.7.3-orig/Lib/test/test_gdb.py Python-3.7.3/Lib/test/test_gdb.py
--- Python-3.7.3-orig/Lib/test/test_gdb.py	2019-03-25 21:21:05.000000000 +0100
+++ Python-3.7.3/Lib/test/test_gdb.py	2019-06-19 16:11:58.443775063 +0200
@@ -172,6 +172,12 @@
                     # #0 0x00003fffb7dd1798 in PyObject_Print ...
                     'set print address off',
 
+                    # Ignore "Missing separate debuginfo" messages,
+                    # which typically carry platform-specific instructions
+                    # (like `debuginfo-install` or `dnf debuginfo install`)
+                    # so they can't be ignored in "ignore_patterns" below.
+                    'set build-id-verbose 0',
+
                     'run']
 
         # GDB as of 7.4 onwards can distinguish between the

Comment 17 Petr Viktorin 2019-06-19 15:31:09 UTC
I filed Bug #1722178 to ask if this is a deliberate change in GDB.

Comment 18 Florian Weimer 2019-06-19 17:39:06 UTC
(In reply to Florian Weimer from comment #12)
> I will try to get an aarch64 or ppc64le system to investigate.  In general,
> you will need to run glibc-debuginfo to run such tests.

Sorry, I can seem to get a matching Fedora machine right now, but I see this issue has already been diagnosed as something not caused by glibc.

Comment 19 Victor Stinner 2019-06-21 13:52:49 UTC
Petr:
> We can fix this by adding to the ignored warnings. Unfortunately, the "dnf" line is Fedora-specific, so it can't go upstream like this: (...)

Honestly, I don't understand why test_gdb is so strict about stderr: it sounds more like a bug that I wanted to fix for years in test_gdb. So I proposed a change upstream:

* https://bugs.python.org/issue37362
* https://github.com/python/cpython/pull/14287

That should fix this issue, but also any new "unexpected" message on stderr.

Comment 20 Miro Hrončok 2019-06-22 22:39:17 UTC
We are good for now. For leftovers, let's follow on them in the upstream ticket.


Note You need to log in before you can comment on or make changes to this bug.