Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2107826 - PYC magic number has changed, all pyc files must be re-created
Summary: PYC magic number has changed, all pyc files must be re-created
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: distribution
Version: 37
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Tomáš Hrnčiar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 2021510 2046851 2093197 2101963 2105063 2105253 2113140 2113156 2113159 2113161 2113214 2113233 2113243 2113434 2113546 2113607 2113611 2113613 2113618 2113620 2113622 2113629 2113630 2113631 2113633 2113634 2113635 2113636 2113637 2113639 2113640 2113641 2113642 2113643 2113644 2113646 2113647 2113648 2113649 2113650 2113655 2113656 2113659 2113660 2113661 2113662 2113675 2113680 2113753 2113762 2113776 2113777 2113983 2113986 2114528 2114550 2114555 2114557 2114558 2114559 2114560 2122597
Blocks: PYTHON3.11
TreeView+ depends on / blocked
 
Reported: 2022-07-16 12:33 UTC by Jan Pazdziora
Modified: 2022-12-21 13:40 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-12-21 13:40:37 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Jan Pazdziora 2022-07-16 12:33:07 UTC 
Description of problem:

When pycparser is imported, its __pycache__ gets regenerated. This suggests that the way the rpm content got built does not align nicely with the rest of the python stack.

Version-Release number of selected component (if applicable):

python3-pycparser-2.20-7.fc37.noarch

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have Dockerfile

FROM registry.fedoraproject.org/fedora:rawhide
RUN dnf install -y python3-pycparser

2. Build a container image:
   $ podman build -t pycparser-fedora .

3. Run a container
   $ podman run --rm -ti --name pycparser pycparser-fedora
   [root@74c5879d48e3 /]# 

4. Check that nothing got changed in the container just yet -- from other terminal run
   $ podman diff pycparser

5. Use pycparser in the container
   [root@6e34aeedb6ec /]# python3 -c 'import pycparser'

6. In that other terminal, do podman diff again:
   $ podman diff pycparser

   
Actual results:

C /usr/lib/python3.11/site-packages/ply/__pycache__
C /usr/lib/python3.11/site-packages/ply/__pycache__/__init__.cpython-311.pyc
C /usr/lib/python3.11/site-packages/ply/__pycache__/lex.cpython-311.pyc
C /usr/lib/python3.11/site-packages/ply/__pycache__/yacc.cpython-311.pyc
C /usr/lib/python3.11/site-packages/pycparser/__pycache__
C /usr/lib/python3.11/site-packages/pycparser/__pycache__/__init__.cpython-311.pyc
C /usr/lib/python3.11/site-packages/pycparser/__pycache__/ast_transforms.cpython-311.pyc
C /usr/lib/python3.11/site-packages/pycparser/__pycache__/c_ast.cpython-311.pyc
C /usr/lib/python3.11/site-packages/pycparser/__pycache__/c_lexer.cpython-311.pyc
C /usr/lib/python3.11/site-packages/pycparser/__pycache__/c_parser.cpython-311.pyc
C /usr/lib/python3.11/site-packages/pycparser/__pycache__/plyparser.cpython-311.pyc

Expected results:

Nothing should have been regenerated in the container in run-time.

Additional info:

I see in https://github.com/freeipa/freeipa-container/runs/7352518475?check_suite_focus=true that samba, cryptography, wrapt, dbus, cffi, gssapi, lxml, systemd, ldap, SSSDConfig, jwcrypto, dns, ipapython, ipaplatform, lib389, qrcode, dateutil, deprecated, pki, usb, yubico, ipaserver, ipaclient, ipalib, or augeas have the same problem.

I started with report against pycparser because, well, the issue seems with the .pyc files, so it felt the closest.
Comment 1 Miro Hrončok 2022-07-16 19:57:22 UTC 
This is because Python 3.11.0b4 has updated the pyc magic number: https://github.com/python/cpython/blob/v3.11.0b4/Lib/importlib/_bootstrap_external.py#L406

Thanks for checking. If this is critical for you, rebuild pycparsing. Otherwise, I'd like the Fedora 37 mass rebuild [1] to handle things before we take any mass action.

See also bz1748018.


[1] Wed 2022-07-20 https://fedorapeople.org/groups/schedule/f-37/f-37-key-tasks.html
Comment 2 Miro Hrončok 2022-07-25 20:24:51 UTC 
The mass rebuild is over, let's check what remains to be rebuilt.
Comment 3 Miro Hrončok 2022-07-26 10:45:32 UTC 
3.11.0b5 is released and according to https://github.com/python/cpython/blob/v3.11.0b5/Lib/importlib/_bootstrap_external.py#L406 the magic number remains unchanged.
Comment 4 Jan Pazdziora 2022-07-29 10:50:03 UTC 
I still see

/usr/lib64/python3.11/site-packages/systemd/__pycache__/__init__.cpython-311.pyc
/usr/lib64/python3.11/site-packages/systemd/__pycache__/daemon.cpython-311.pyc

getting rebuilt in runtime, for the FreeIPA server use case: https://github.com/adelton/freeipa-container/runs/7573207697?check_suite_focus=true.
Comment 5 Jan Pazdziora 2022-07-30 08:24:17 UTC 
It seems that python3-psutil might also need the rebuild, seeing also

/usr/lib64/python3.11/site-packages/psutil/__pycache__/__init__.cpython-311.pyc
/usr/lib64/python3.11/site-packages/psutil/__pycache__/_pslinux.cpython-311.pyc
/usr/lib64/python3.11/site-packages/psutil/__pycache__/_common.cpython-311.pyc
/usr/lib64/python3.11/site-packages/psutil/__pycache__/_compat.cpython-311.pyc
/usr/lib64/python3.11/site-packages/psutil/__pycache__/_psposix.cpython-311.pyc

show up in a slightly different test scenario.
Comment 6 Miro Hrončok 2022-07-31 12:28:26 UTC 
Yes, python-systemd and python-psutil both failed to rebuild. So did many others. We will track the remaining failures -- once releng opens FTBFS bugzillas, we will mark them as blocking this one. Please, don't comment here for every single individual package that has failed the mass rebuild, it will likely be at least a hundred.

If you care about python-systemd and python-psutil, you can file the FTBFS bugzillas for them before releng does. Just make sure they block this bugzilla and the F37FTBFS tracker.
Comment 7 Tomáš Hrnčiar 2022-08-01 12:56:22 UTC 
Here is the list of failed packages that need to be rebuilt. 

    "torebuild": [
        "bind",
        "capstone",
        "copr-keygen",
        "crypto-policies",
        "csound",
        "espresso",
        "fedmsg",
        "future",
        "gnome-builder",
        "grass",
        "libsbml",
        "link-grammar",
        "lldb",
        "net-snmp",
        "osbs-client",
        "pesign",
        "psi4",
        "pungi",
        "pybind11",
        "python-APScheduler",
        "python-TestSlide",
        "python-astropy",
        "python-databases",
        "python-dirq",
        "python-django-pyscss",
        "python-enrich",
        "python-fasjson-client",
        "python-gear",
        "python-httpretty",
        "python-igor",
        "python-ipyparallel",
        "python-jedi",
        "python-jsons",
        "python-m2r",
        "python-mplcursors",
        "python-networkx",
        "python-neutronclient",
        "python-nose2",
        "python-numpydoc",
        "python-odml",
        "python-owl_rl",
        "python-pandas",
        "python-patsy",
        "python-pecan-notario",
        "python-pikepdf",
        "python-pingouin",
        "python-psutil",
        "python-pybids",
        "python-pycdlib",
        "python-pymatreader",
        "python-pynwb",
        "python-pysam",
        "python-pyunicorn",
        "python-reportlab",
        "python-socks",
        "python-sphinx-panels",
        "python-systemd",
        "python-typish",
        "python-vdf",
        "python-websockets",
        "python-wsaccel",
        "python-xarray",
        "resalloc",
        "restview",
        "rpy",
        "veusz",
        "xapian-bindings"
    ]
Comment 8 Tomáš Hrnčiar 2022-08-02 08:00:09 UTC 
Bugzillas for packages below weren't opened yet.

        "fedmsg",
        "link-grammar",
        "lldb",
        "osbs-client",
        "pesign",
        "pungi",
        "pybind11",
        "python-fasjson-client",
        "python-gear",
        "python-neutronclient",
        "python-nose2",
        "python-pikepdf",
        "python-psutil",
        "python-pycdlib",
        "python-pynwb",
        "python-pysam",
        "restview",
Comment 9 Miro Hrončok 2022-08-02 09:06:09 UTC 
The following packages appear to be rebuilt in the meantime:

link-grammar https://koji.fedoraproject.org/koji/buildinfo?buildID=2040588 --should be OK
pesign https://koji.fedoraproject.org/koji/buildinfo?buildID=2020730 -- pesign is special, it requires signing permission to tag to rawhide, we need to ask the maintainers or releng


The following packages are listed at https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html but we see no F37FTBFS bugzilla:

fedmsg
lldb
pungi
pybind11
python-fasjson-client
python-nose2
python-pikepdf
python-psutil
python-pycdlib
python-pynwb
restview


The rest probably failed to even build SRPM and is only listed at https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html

osbs-client
python-gear
python-neutronclient
Comment 10 Miro Hrončok 2022-08-02 10:05:06 UTC 
osbs-client -- releng script has not incremented the release correctly: https://src.fedoraproject.org/rpms/osbs-client/c/9fcd8043e4bf26f9f1ef92176bf1d2c6a5c0c667?branch=rawhide -- attempted fix in https://src.fedoraproject.org/rpms/osbs-client/pull-request/10


The following two builds hanged and were eventually cancelled:

python-gear --https://koji.fedoraproject.org/koji/buildinfo?buildID=2022987
python-neutronclient -- https://koji.fedoraproject.org/koji/buildinfo?buildID=2023678
Comment 11 Miro Hrončok 2022-08-02 13:59:45 UTC 
(In reply to Miro Hrončok from comment #10)
> osbs-client -- releng script has not incremented the release correctly:
> https://src.fedoraproject.org/rpms/osbs-client/c/
> 9fcd8043e4bf26f9f1ef92176bf1d2c6a5c0c667?branch=rawhide -- attempted fix in
> https://src.fedoraproject.org/rpms/osbs-client/pull-request/10

Merged and built.
Comment 12 Miro Hrončok 2022-08-02 22:54:51 UTC 
All bugzillas open. pybind11 was built in the meantime.
Comment 13 Tomáš Hrnčiar 2022-08-03 07:35:50 UTC 
(In reply to Miro Hrončok from comment #9)
> pesign https://koji.fedoraproject.org/koji/buildinfo?buildID=2020730 --
> pesign is special, it requires signing permission to tag to rawhide, we need
> to ask the maintainers or releng

pesign was rebuilt and the maintainer added "noautobuild" file into the dist-git.

https://koji.fedoraproject.org/koji/buildinfo?buildID=2040965
Comment 14 Ben Cotton 2022-08-09 13:40:05 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle.
Changing version to 37.
Comment 15 Miro Hrončok 2022-08-30 10:34:04 UTC 
Here is the current list of failed packages that need to be rebuilt:

    "torebuild": [
        "capstone",
        "copr-keygen",
        "csound",
        "fedmsg",
        "psi4",
        "python-APScheduler",
        "python-astropy",
        "python-databases",
        "python-enrich",
        "python-fasjson-client",
        "python-jsons",
        "python-nose2",
        "python-numpydoc",
        "python-odml",
        "python-pandas",
        "python-pecan-notario",
        "python-pybids",
        "python-pycdlib",
        "python-pysam",
        "python-pyunicorn",
        "python-typish",
        "python-xarray",
        "resalloc",
        "restview",
        "rpy"
    ]
Comment 16 Petr Viktorin (pviktori) 2022-12-21 13:40:37 UTC 
All packages have open bugzillas. We're no longer tracking this so I'm closing the tracker.
Note You need to log in before you can comment on or make changes to this bug.