Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2272267 - glib v0.19.3 test suite segfaults when built with optimizations
Summary: glib v0.19.3 test suite segfaults when built with optimizations
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: rust-glib
Version: rawhide
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Rust SIG
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-03-29 23:24 UTC by Fabio Valentini
Modified: 2024-04-19 21:28 UTC (History)
6 users (show)

Fixed In Version: rust-glib-0.19.3-2.fc39 rust-glib-0.19.3-2.fc40
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
If this bug requires documentation, please select an appropriate Doc Type value.
Last Closed: 2024-04-12 01:14:11 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Fabio Valentini 2024-03-29 23:24:07 UTC
The test suite of the "glib" crate started failing in Fedora with segmentation faults. It is possible to track this back to the update from Rust 1.75 to Rust 1.76, across all current branches of Fedora:

https://koschei.fedoraproject.org/package/rust-glib

The following tests now cause segmentation faults (list updated for glib v0.19.3):

- variant_iter::tests::test_variant_iter_array (a --lib test)
- variant_iter::tests::test_variant_str_iter_last (a --lib test)
- variant_iter::tests::test_variant_str_iter_nth (a --lib test)
- variant::Variant::array_iter_str (a --doc test)

The error message looks something like this:

"""
error: test failed, to rerun pass `--lib`
Caused by:
  process didn't exit successfully: `/builddir/build/BUILD/glib-0.18.5/target/rpm/deps/glib-8a813dbb6056e1aa --skip structured_log` (signal: 11, SIGSEGV: invalid memory reference)
"""

Since the test harness is killed by the SIGSEGV, the output of "cargo test" does not actually show the failing test. I isolated the failure cases by running the tests with "--test-threads 1" and added known bad cases with "--skip" until the test suite passed.

The "structured_log" test always fails - it expects some paths to match the workspace layout in the upstream git repository. It can be skipped and is not relevant to this issue.

============================================================

Test code for the first three failing tests is in this file upstream (which has not been changed in the last two years):
https://github.com/gtk-rs/gtk-rs-core/blob/master/glib/src/variant_iter.rs#L204

Test code for the failing doctest is in this file upstream:
https://github.com/gtk-rs/gtk-rs-core/blob/master/glib/src/variant.rs#L843

============================================================

I tried the following scenarios to reproduce the issue (always running "cargo clean" inbetween):

1. system rustc (1.77.0 on Fedora 39):

a. "/usr/bin/cargo test --release -- --skip structured_log": crashes
b. "cargo +system test --release -- --skip structured_log": passes (system = linked toolchain at /usr)

(I have *no idea* how case 1.a and 1.b can behave differently.)

2. rustup toolchains:

a. "cargo +stable test --release -- --skip structured_log": passes (stable = 1.77.1 (7cf61ebde 2024-03-27))
b. "cargo +beta test --release -- --skip structured_log": crashes (beta = 1.78.0-beta.3 (4147533e0 2024-03-27))
c. "cargo +nightly test --release -- --skip structured_log": crashes (nightly = 1.79.0-nightly (c9f8f3438 2024-03-27))

3. different compiler flags for system rustc (1.77.0 on Fedora 39):

a. "RUSTFLAGS="-Copt-level=1" /usr/bin/cargo test -- --skip structured_log": passes
b. "RUSTFLAGS="-Copt-level=2" /usr/bin/cargo test -- --skip structured_log": crashes
c. "RUSTFLAGS="-Copt-level=3" /usr/bin/cargo test -- --skip structured_log": crashes

4. different compiler flags for rustup toolchains:

a. "RUSTFLAGS="-Copt-level=1" cargo +stable test -- --skip structured_log": passes
b. "RUSTFLAGS="-Copt-level=2" cargo +stable test -- --skip structured_log": passes
c. "RUSTFLAGS="-Copt-level=3" cargo +stable test -- --skip structured_log": passes
d. "RUSTFLAGS="-Copt-level=1" cargo +beta test -- --skip structured_log": passes
e. "RUSTFLAGS="-Copt-level=2" cargo +beta test -- --skip structured_log": crashes
f. "RUSTFLAGS="-Copt-level=3" cargo +beta test -- --skip structured_log": crashes
g. "RUSTFLAGS="-Copt-level=1" cargo +nightly test -- --skip structured_log": passes
h. "RUSTFLAGS="-Copt-level=2" cargo +nightly test -- --skip structured_log": crashes
i. "RUSTFLAGS="-Copt-level=3" cargo +nightly test -- --skip structured_log": crashes

============================================================

I cannot determine why the Fedora Rust toolchain seems to have introduced this issue with Rust 1.76, whereas upstream rust toolchains only start exhibiting it with Rust 1.78.0 beta.

I don't think different LLVM versions should be the cause, since the crashes started happening with Rust 1.76 across all branches of Fedora, where

- on Fedora Rawhide, rustc links LLVM 18,
- on Fedora 39 and 40, rustc links LLVM 17, and
- on Fedora 38, rustc links LLVM 16.

Maybe the glib crate has some UB which is only triggered by optimizations in recent Rust releases? But that wouldn't necessarily explain why the Fedora and the upstream toolchains behave differently.


Reproducible: Always

Steps to Reproduce:
1. unpack sources for the "glib" crate
2. run "/usr/bin/cargo test --release"
Actual Results:  
Tests pass.

Expected Results:  
Test runner segfaults.

Comment 1 Josh Stone 2024-03-30 00:22:57 UTC
(In reply to Fabio Valentini from comment #0)
> 1. system rustc (1.77.0 on Fedora 39):
> 
> a. "/usr/bin/cargo test --release -- --skip structured_log": crashes
> b. "cargo +system test --release -- --skip structured_log": passes (system =
> linked toolchain at /usr)
> 
> (I have *no idea* how case 1.a and 1.b can behave differently.)

Is your `rustup default` also "system"? Because 1.a will still use "rustc" from your PATH, which will probably go through the rustup shim again.

> I cannot determine why the Fedora Rust toolchain seems to have introduced
> this issue with Rust 1.76, whereas upstream rust toolchains only start
> exhibiting it with Rust 1.78.0 beta.

I think there is UB, and I would not try to reason about the compiler beyond that. Any subtle difference in the toolchains could change optimization in a way that may or may not manifest in noticeably bad behavior.

There's an unsafe block in variant_iter.rs:

        unsafe {
            let p: *mut libc::c_char = std::ptr::null_mut();
            let s = b"&s\0";
            ffi::g_variant_get_child(
                self.variant.to_glib_none().0,
                i,
                s as *const u8 as *const _,
                &p,
                std::ptr::null::<i8>(),
            );
            let p = std::ffi::CStr::from_ptr(p);
            p.to_str().unwrap()
        }

The first p is initialized to null_mut(), but it is not itself mutable. It is passed immutably as &p to the FFI call, and then passed to CStr::from_ptr. By Rust semantics, it must still be null! But clearly this code expects g_variant_get_child to have filled in a valid pointer.

You can get a fun demonstration of UB here too. I added an assertion before the CStr::from_ptr call:

            assert_ne!(p, std::ptr::null_mut());

and it fails in a confusing way:

thread 'variant_iter::tests::test_variant_iter_array' panicked at glib/src/variant_iter.rs:129:13:
assertion `left != right` failed
  left: 0x7f1fa80011b0
 right: 0x0


The fix is to start with "let mut p" and pass "&mut p" to g_variant_get_child. With that, all tests pass in release mode for me on F39 +system, +stable, +beta, and +nightly.

Comment 2 Fabio Valentini 2024-03-30 00:52:02 UTC
Thank you for the pointers!

I submitted the suggested fix as a PR to upstream:
https://github.com/gtk-rs/gtk-rs-core/pull/1343

Comment 3 Fedora Update System 2024-04-03 19:53:44 UTC
FEDORA-2024-f8adb1bac8 (rust-glib0.15-0.15.12-5.fc40, rust-glib0.17-0.17.10-3.fc40, and 2 more) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-f8adb1bac8

Comment 4 Fedora Update System 2024-04-03 19:53:46 UTC
FEDORA-2024-3ee122ae90 (rust-glib0.15-0.15.12-5.fc38 and rust-glib0.17-0.17.10-3.fc38) has been submitted as an update to Fedora 38.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-3ee122ae90

Comment 5 Fedora Update System 2024-04-03 19:53:46 UTC
FEDORA-2024-ec54a6adea (rust-glib0.15-0.15.12-5.fc39, rust-glib0.17-0.17.10-3.fc39, and 2 more) has been submitted as an update to Fedora 39.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-ec54a6adea

Comment 6 Fedora Update System 2024-04-04 01:38:03 UTC
FEDORA-2024-3ee122ae90 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-3ee122ae90`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-3ee122ae90

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2024-04-04 01:39:39 UTC
FEDORA-2024-f8adb1bac8 has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-f8adb1bac8`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-f8adb1bac8

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2024-04-04 02:00:25 UTC
FEDORA-2024-ec54a6adea has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-ec54a6adea`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ec54a6adea

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2024-04-12 01:14:11 UTC
FEDORA-2024-3ee122ae90 (rust-glib0.15-0.15.12-5.fc38 and rust-glib0.17-0.17.10-3.fc38) has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 10 Fedora Update System 2024-04-12 01:21:20 UTC
FEDORA-2024-ec54a6adea (rust-glib0.15-0.15.12-5.fc39, rust-glib0.17-0.17.10-3.fc39, and 2 more) has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 11 Fedora Update System 2024-04-19 21:28:53 UTC
FEDORA-2024-f8adb1bac8 (rust-glib0.15-0.15.12-5.fc40, rust-glib0.17-0.17.10-3.fc40, and 2 more) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.