2064025 – Flaky build failures on ppc64le due to CordzInfoStatisticsTest.ThreadSafety

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 2064025 - Flaky build failures on ppc64le due to CordzInfoStatisticsTest.ThreadSafety

Summary: Flaky build failures on ppc64le due to CordzInfoStatisticsTest.ThreadSafety

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	abseil-cpp
Sub Component:
Version:	39
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Ben Beasley
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:	2229015
Blocks:	PPCTracker
TreeView+	depends on / blocked

Reported:	2022-03-14 21:40 UTC by Ben Beasley
Modified:	2023-09-18 00:16 UTC (History)
CC List:	2 users (show)
Fixed In Version:	abseil-cpp-20230802.0-1.fc39
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-09-18 00:16:11 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
Full log from a failing build (deleted) 2022-03-14 21:40 UTC, Ben Beasley	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	abseil abseil-cpp issues 1132	0	None	open	CordzInfoStatisticsTest.ThreadSafety flaky failure on ppc64le	2022-03-14 21:40:34 UTC

Description Ben Beasley 2022-03-14 21:40:35 UTC

Created attachment 1865926 [details]
Full log from a failing build

Description of problem:

Since updating to 20211102.0 in the development version of the distribution, we see frequent flaky failures of CordzInfoStatisticsTest.ThreadSafety in absl_cordz_info_statistics_test on the ppc64le architecture. When the test fails, a message like
> [mutex.cc : 1682] RAW: Mutex unlocked when destroyed or not locked: v=0x0
is printed.

Version-Release number of selected component (if applicable): 20211102.0-1.fc37


How reproducible:


Steps to Reproduce:
1. fedpkg scratch-build --srpm --arches ppc64le

Actual results:

Always succeeds.

Expected results:

Frequently (roughly 10–80% of the time) fails with the described error.

Additional info:

I reported this upstream as https://github.com/abseil/abseil-cpp/issues/1132.

It seems likely that this is a real problem, but I am not sure we can solve it downstream.

It would be easy enough to skip the test on ppc64le, if that’s the way we want to go:

> %ifarch ppc64le
> # CordzInfoStatisticsTest.ThreadSafety flaky failure on ppc64le
> # https://github.com/abseil/abseil-cpp/issues/1132
> # 
> # 144/182 Test #150: absl_cordz_info_statistics_test > ....................Subprocess aborted***Exception:   0.31 sec
> # Running main() from gmock_main.cc
> # [==========] Running 14 tests from 1 test suite.
> # [----------] Global test environment set-up.
> # [----------] 14 tests from CordzInfoStatisticsTest
> # [ RUN      ] CordzInfoStatisticsTest.Flat
> # [       OK ] CordzInfoStatisticsTest.Flat (0 ms)
> #   …
> # [ RUN      ] CordzInfoStatisticsTest.ThreadSafety
> # [mutex.cc : 1682] RAW: Mutex unlocked when destroyed or not locked: v=0x0
> sed -r -i 's/\bThreadSafety\b/DISABLED_&/' \
>     absl/strings/internal/cordz_info_statistics_test.cc
> %endif

Comment 1 Ben Beasley 2022-03-15 18:01:11 UTC

It looks like this could actually be fixed by disabling LTO on ppc64le. I think that’s an appropriate workaround. (Of course, problems that show up with LTO enabled are usually actually some kind of undefined behavior upstream, but it’s not always easy to find the root cause.)

I am also planning to disable LTO on s390x, since it fixes the (reliable) failure in Symbolize.SymbolizeWithMultipleMaps in absl_symbolize_test on that architecture (https://github.com/abseil/abseil-cpp/issues/1133).

I’m still inclined to leave LTO enabled on other architectures as long as we don’t observe any specific issues on them.

Comment 2 Ben Beasley 2022-03-15 18:12:53 UTC

I take it back. The problem is still present with LTO. It *might* be rarer, or I might have just gotten (un)lucky.

Comment 3 Ben Beasley 2022-03-15 18:33:55 UTC

In doing ten or so scratch builds to look at this issue, I observed a couple of other flaky failures. Since these are relatively infrequent, I’m not trying to look into them or report them upstream right now.

-----

On aarch64:

 10/182 Test  #13: absl_sysinfo_test ..................................***Failed    0.06 sec
Running main() from /builddir/build/BUILD/googletest-release-1.11.0/googletest/src/gtest_main.cc
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from SysinfoTest
[ RUN      ] SysinfoTest.NumCPUs
[       OK ] SysinfoTest.NumCPUs (0 ms)
[ RUN      ] SysinfoTest.NominalCPUFrequency
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/base/internal/sysinfo_test.cc:54: Failure
Expected equality of these values:
  NominalCPUFrequency()
    Which is: 3e+09
  1.0
    Which is: 1
CPU frequency detection was fixed! Please update unittest.

-----

On s390x (without LTO):

        Start  81: absl_random_mocking_bit_gen_test
 80/182 Test  #81: absl_random_mocking_bit_gen_test ...................***Failed    0.05 sec
Running main() from /builddir/build/BUILD/googletest-release-1.11.0/googletest/src/gtest_main.cc
[==========] Running 22 tests from 7 test suites.
[----------] Global test environment set-up.
[----------] 9 tests from BasicMocking
[ RUN      ] BasicMocking.AllDistributionsAreOverridable
[       OK ] BasicMocking.AllDistributionsAreOverridable (0 ms)
[ RUN      ] BasicMocking.OnDistribution
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:82:
    Function call: Call(1, 1000000)
          Returns: 20
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:87:
    Function call: Call(0, 100)
          Returns: 5
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:92:
    Function call: Call(1)
          Returns: 42
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:97:
    Function call: Call(1)
          Returns: 500
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:101:
    Function call: Call(1e-06)
          Returns: true
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:105: Failure
Expected: (absl::Zipf<int>(gen, 1000000, 2.0, 1.0)) != (1221), actual: 1221 vs 1221
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:106:
    Function call: Call(1000000, 2, 1)
          Returns: 1221
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:111:
    Function call: Call(0, 1)
          Returns: 0.001
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:116:
    Function call: Call(0, 1000000, 2)
          Returns: 2040
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[  FAILED  ] BasicMocking.OnDistribution (0 ms)
[ RUN      ] BasicMocking.GMockMatchers
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:125:
    Function call: Call(1000000, 2, 1)
          Returns: 1221
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[       OK ] BasicMocking.GMockMatchers (0 ms)
[ RUN      ] BasicMocking.OverridesWithMultipleGMockExpectations
[       OK ] BasicMocking.OverridesWithMultipleGMockExpectations (0 ms)
[ RUN      ] BasicMocking.DefaultArgument
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:145:
    Function call: Call(1)
          Returns: 200
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:145:
    Function call: Call(1)
          Returns: 200
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[       OK ] BasicMocking.DefaultArgument (0 ms)
[ RUN      ] BasicMocking.MultipleGenerators
[       OK ] BasicMocking.MultipleGenerators (0 ms)
[ RUN      ] BasicMocking.MocksNotTrigeredForIncorrectTypes
[       OK ] BasicMocking.MocksNotTrigeredForIncorrectTypes (0 ms)
[ RUN      ] BasicMocking.FailsOnUnsatisfiedMocks
[       OK ] BasicMocking.FailsOnUnsatisfiedMocks (0 ms)
[ RUN      ] BasicMocking.WillByDefaultWithArgs
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:331:
    Function call: Call(1.7)
          Returns: 17
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
GMOCK WARNING:
Uninteresting mock function call - taking default action specified at:
/builddir/build/BUILD/abseil-cpp-20211102.0/absl/random/mocking_bit_gen_test.cc:331:
    Function call: Call(0.03)
          Returns: 0
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md#knowing-when-to-expect for details.
[       OK ] BasicMocking.WillByDefaultWithArgs (0 ms)
[----------] 9 tests from BasicMocking (0 ms total)

Comment 4 Ben Beasley 2022-03-16 23:58:37 UTC

For an update on the SysinfoTest.NominalCPUFrequency issue, see discussion in https://src.fedoraproject.org/rpms/abseil-cpp/pull-request/3. That test is now disabled in our package, and will be removed in future upstream releases.

Comment 5 Ben Cotton 2022-08-09 13:13:51 UTC

This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle.
Changing version to 37.

Comment 6 Ben Beasley 2023-08-12 14:17:39 UTC

I can’t reproduce this, even on Fedora 37, with abseil-cpp-20230802.0; see https://github.com/abseil/abseil-cpp/issues/1132#issuecomment-1675180517. I’m going to consider this probably-fixed-upstream in that release.

Comment 7 Fedora Release Engineering 2023-08-16 07:05:20 UTC

This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle.
Changing version to 39.

Comment 8 Fedora Update System 2023-09-16 21:02:05 UTC

FEDORA-2023-b5fb7bba8c has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-b5fb7bba8c

Comment 9 Fedora Update System 2023-09-17 02:03:32 UTC

FEDORA-2023-b5fb7bba8c has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-b5fb7bba8c`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-b5fb7bba8c

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2023-09-18 00:16:11 UTC

FEDORA-2023-b5fb7bba8c has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.