Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1973663 - glibc: glibc-all-langpacks should require glibc-gconv-extra in buildroots
Summary: glibc: glibc-all-langpacks should require glibc-gconv-extra in buildroots
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: glibc
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Florian Weimer
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.10
TreeView+ depends on / blocked
 
Reported: 2021-06-18 12:20 UTC by Miro Hrončok
Modified: 2021-06-21 11:59 UTC (History)
13 users (show)

Fixed In Version: glibc-2.33.9000-23.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-18 16:50:04 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Sourceware 27996 0 P2 NEW setlocale should fail if the required gconv module is missing 2021-06-18 14:33:13 UTC

Description Miro Hrončok 2021-06-18 12:20:00 UTC
Description of problem:
Since glibc-2.33.9000-13.fc35, Python (3.10, 3.9, 3.8, 3.7...) fails to build on Fedora Rawhide due to test failures:


======================================================================
ERROR: test_float_parsing (test.test__locale._LocaleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 188, in test_float_parsing
    if localeconv()['decimal_point'] != '.':
UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error
======================================================================
ERROR: test_lc_numeric_basic (test.test__locale._LocaleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 155, in test_lc_numeric_basic
    li_radixchar = localeconv()[lc]
UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error
======================================================================
ERROR: test_lc_numeric_localeconv (test.test__locale._LocaleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 134, in test_lc_numeric_localeconv
    formatting = localeconv()
UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error
======================================================================
ERROR: test_lc_numeric_nl_langinfo (test.test__locale._LocaleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 120, in test_lc_numeric_nl_langinfo
    if self.numeric_tester('nl_langinfo', nl_langinfo(li), lc, loc):
UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error
----------------------------------------------------------------------


Version-Release number of selected component (if applicable): glibc-2.33.9000-13.fc35 or newer, at least up to -18.fc35


Steps to Reproduce: Build python3.X package in rawhide buildroot.

(I'll try to isolate a smaller reproducer later.)

Actual results: %check fails with the errors above.


Expected results: %check passes


Additional info:

This was discovered by Koschei, but I've bisected the build that started this further down:

https://koschei.fedoraproject.org/package/python3.10?collection=f35
https://koschei.fedoraproject.org/package/python3.9?collection=f35
https://koschei.fedoraproject.org/package/python3.8?collection=f35
https://koschei.fedoraproject.org/package/python3.7?collection=f35

This blocks upgrade from Python 3.10.0b2 to Python 3.10.0b3.

Comment 1 Miro Hrončok 2021-06-18 12:24:17 UTC
Also reproduced with glibc-2.33.9000-13.fc35 (will edit the initial comment here to have that version).

Comment 2 Florian Weimer 2021-06-18 12:31:53 UTC
Would you please check if adding

BuildRequires: glibc-gconv-extra

fixes this? Thanks.

Comment 3 Miro Hrončok 2021-06-18 12:45:03 UTC
(In reply to Florian Weimer from comment #2)
> Would you please check if adding
> 
> BuildRequires: glibc-gconv-extra
> 
> fixes this? Thanks.


$ git diff
diff --git a/python3.9.spec b/python3.9.spec
index 9abe196..91143c7 100644
--- a/python3.9.spec
+++ b/python3.9.spec
@@ -222,6 +222,7 @@ BuildRequires: gdbm-devel
 BuildRequires: git-core
 BuildRequires: glibc-all-langpacks
 BuildRequires: glibc-devel
+BuildRequires: glibc-gconv-extra
 BuildRequires: gmp-devel
 BuildRequires: gnupg2
 BuildRequires: libappstream-glib


https://koji.fedoraproject.org/koji/taskinfo?taskID=70359053

Comment 4 Miro Hrončok 2021-06-18 13:22:20 UTC
Yes indeed. Should that be required by Python on runtime?

Comment 5 Florian Weimer 2021-06-18 13:26:23 UTC
(In reply to Miro Hrončok from comment #4)
> Yes indeed. Should that be required by Python on runtime?

Great. Outside mock, we default to installing weak dependencies, so the package will be installed by default. A strong dependency from Python would make it basically impossible to deinstall it, so it entirely defeats the gconv split.

We are considering adding a stronger dependency to glibc-all-langpacks or redhat-rpm-config to help along downstream. There is a strong desire to backport this to reduce minimal container size.

Comment 6 Miro Hrončok 2021-06-18 13:38:19 UTC
So, what should we do?

If we add:

  BuildRequires: (glibc-gconv-extra if glibc > 2.33.9000-12)

It will most likely fix the build but break Python for some users who install without recommends.

Comment 7 Florian Weimer 2021-06-18 13:43:14 UTC
I think we should add

Requires: (glibc-gconv-extra if redhat-rpm-config)

to glibc-all-langpacks. This is based on the observation that

BuildRequires: glibc-all-langpacks

is used by many packages to indicate that they can't work with the minimal set of locales in the buildroot.

Comment 8 Miro Hrončok 2021-06-18 13:45:06 UTC
I agree. I still don't understand the impact of this change for the actual users, but at least that would unblock our test failures.

Comment 9 Miro Hrončok 2021-06-18 13:48:00 UTC
Also, maybe the split something to be communicated more explicitly in https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/7R4RBS2MD75S2MTY42L44IW3KOUT5K36/ ?

Comment 10 Florian Weimer 2021-06-18 13:51:12 UTC
(In reply to Miro Hrončok from comment #9)
> Also, maybe the split something to be communicated more explicitly in
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/
> thread/7R4RBS2MD75S2MTY42L44IW3KOUT5K36/ ?

The split isn't really related to the snapshots, it was a case of parallel development.

We simply forgot that mock defaults to not installing weak dependencies, we expected the change to be invisible to users by default.

Comment 11 Victor Stinner 2021-06-18 14:25:42 UTC
On Rawhide (glibc-2.33.9000-18.fc35.x86_64), if I uninstall glibc-gconv-extra, mbstowcs() fails to decode non-ASCII bytes (0x80..0xFF) for the lv_LV locale, whereas nl_langinfo(CODESET) announces the ISO-8859-13 encoding. If I install the package, mbstowcs() decodes b"\xA0" byte as the U+00A0 character as expected (ISO-8859-13 encoding).

Maybe setlocale(LC_CTYPE, "lv_LV") must fail if glibc-gconv-extra is not installed. Or at least, localeconv() and nl_langinfo() must only return byte strings which can be decoded with mbstowcs() (only return ASCII strings).

Modifying the Python package to require glibc-gconv-extra works around the issue, but it would make the Python installation larger, even if it's not needed. I would suggest to only use a *strong* dependency to *test* Python (as we do for glibc-all-langpacks).

Python doesn't stricly *require* all available and working locales to run. Only tests require them.

Comment 12 Victor Stinner 2021-06-18 14:28:34 UTC
Python locale.nl_langinfo() function also has an old know encoding issue with complex locale configuration: https://bugs.python.org/issue25812

But it doesn't seem to matter here. Here it seems like it's a glibc change which causes Python test_locale and test__locale to fail.

Comment 13 Florian Weimer 2021-06-18 14:33:14 UTC
(In reply to Victor Stinner from comment #11)
> On Rawhide (glibc-2.33.9000-18.fc35.x86_64), if I uninstall
> glibc-gconv-extra, mbstowcs() fails to decode non-ASCII bytes (0x80..0xFF)
> for the lv_LV locale, whereas nl_langinfo(CODESET) announces the ISO-8859-13
> encoding. If I install the package, mbstowcs() decodes b"\xA0" byte as the
> U+00A0 character as expected (ISO-8859-13 encoding).

Thank you for pointing that out, I had not considered this aspect. I have filed this as an upstream bug: https://sourceware.org/bugzilla/show_bug.cgi?id=27996

We will fix this separate in Fedora (unless upstream does not want to).

glibc-2.33.9000-23.fc35 is building in the f35-build-side-42875 side tag (with just the dependency change).  I will try to rebuild python3.10 there and merge the side tag if that succeeds. Sorry for the inconvenience.

Comment 14 Miro Hrončok 2021-06-18 14:44:58 UTC
Just a note, python3.9 will build faster than python3.10 as it does not build/test the debug build.

Comment 15 Miro Hrončok 2021-06-18 15:36:06 UTC
Successfully waited 3:19 for glibc-2.33.9000-23.fc35 to appear in the f35-build-side-42875 repo

Building python3.9-3.9.5-3.fc35 for f35-build-side-42875
Created task: 70367829
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=70367829

Comment 16 Miro Hrončok 2021-06-18 16:16:59 UTC
The arm builds are still running but I think we can call it a success.

Comment 17 Florian Weimer 2021-06-18 16:50:04 UTC
My builds passed as well, merging the side tag.

Comment 18 Miro Hrončok 2021-06-18 16:58:14 UTC
Thanks for the fast response.

Comment 19 Fedora Update System 2021-06-18 16:58:57 UTC
FEDORA-2021-88b61c4e53 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 Victor Stinner 2021-06-21 11:59:28 UTC
> Status: MODIFIED → CLOSED

Oh, that was quick! Thanks for the fix Florian.


Note You need to log in before you can comment on or make changes to this bug.