Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1748018
| Summary: | PYC magic number has changed, all pyc files must be re-created | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Christian Heimes <cheimes> |
| Component: | python3 | Assignee: | Miro Hrončok <mhroncok> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 32 | CC: | cstratak, dmalcolm, lslebodn, mail, m.cyprian, mhroncok, pviktori, rkuska, shcherbina.iryna, slavek.kabrda, tomspur, torsava, vstinner |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-05-05 12:56:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1770859, 1779753, 1781891, 1792942, 1795451, 1799511, 1799513, 1799928, 1799929, 1799934, 1799936, 1799938, 1799951 | ||
| Bug Blocks: | 1686977 | ||
|
Description
Christian Heimes
2019-09-02 13:38:09 UTC
We could do one of the following: A) targeted rebuild of everything with the old bytecode version B) wait for the mass rebuild Mass rebuild happens in January 2020, hence quite late from now. What are the chances that this will happen again between b4 and rc1? I suggest we wait for rc1 and rebuild then. The magic number for byte code is stable for a minor release. Since Python 3.8 is still in beta, byte code changes can rarely occur. The byte code has changed between 3.8b3 and 3.8b4. Virtually all python3-* packages have been created with 3.8b3. This causes three different kinds of problems: 1) If a non-root user executes Python code, then startup will be slower because Python cannot use the pyc files with the old magic number. 2) If a root user with restricted SELinux context executes Python code, then write operation will fail and the audit log will be pumped with AVC violations. The result is (1) + lots of noise. 3) If a root user with unrestricted SELinux context runs Python code, Python is able to regenerate and store the updated PYC files. Afterwards rpm --verify will fail for all touched files. I highly recommend to rebuild all Python packages before Fedora 32 is released. I don't expect any PYC format changes now. To be safe you could wait until 3.8.0 final is out. > I highly recommend to rebuild all Python packages before Fedora 32 is released.
This will happen either way, the mass rebuild is scheduled before the Fedora 32 release date. Packages that fail to build will however still be affected.
I suggest we wait **at least** for rc1 and then try to rebuild everything.
> I suggest we wait **at least** for rc1 and then try to rebuild everything.
+1
*** Bug 1748005 has been marked as a duplicate of this bug. *** The rc1 is almost upon us: https://mail.python.org/archives/list/python-dev@python.org/message/ACMKEQNGK4FVUIZ6TYD5H26OSPIO5GSN/ The magic number is still 3413: https://github.com/python/cpython/blob/3.8/Lib/importlib/_bootstrap_external.py I'll draft a script that should tell us what packages need rebuilding. ### Notes: This lists all source RPMs that ship Python 3.8 bytecode: $ repoquery --repo=rawhide -f '*.cpython-38.pyc' --source This lists all successful builds in Koji of a source package <pkg>, after python3-3.8.0~b4-1.fc32 finished: $ koji list-builds --package <pkg> --after '2019-08-31 16:11:41' --state=COMPLETE --quiet It took an arbitrary amount of extra time before python3-3.8.0~b4-1.fc32 was present in the buildroot. I suggest that we check builds from 2019-08-31 manually. The following packages were built on 2019-08-31, after 16:11:41. With Python 3.8.0b3: - python-efel - python-jep - python-fastavro - python-lfpy - python-hdmf With Python 3.8.0b4: - duplicity - python-molecule - python-django-cors-headers In theory, some package may have started the build with 3.8.0b3 but finished the next day, however given the nature of this bug, such packages will be taken care of during the mass rebuild. This was announced on devel yesterday: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ESAVCLLOGXOLUX2V6BQWDCH5TBUEO4C5/ Rebuilds have started. The following packages either haven't finished building or failed: PyGreSQL bodhi diffoscope enki file future gcc gmsh gtts hatch hovercraft httpwatcher kicad lammps mir otf2 pcs protobuf pyOpenSSL pybluez pylast pyparted pytest python-PyPDF2 python-ansi2html python-arpeggio python-ase python-astunparse python-autobahn python-basemap python-colorclass python-compreffor python-conda-package-handling python-cpuinfo python-dictdiffer python-diff-cover python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-google-apputils python-hug python-inotify python-jaraco-packaging python-keystoneauth1 python-mapnik python-mechanize python-mimeparse python-openshift python-pdfminer python-pip-shims python-pluggy python-pocketlint python-portend python-progressbar2 python-prometheus_client python-rosinstall python-ryu python-setuptools-lint python-shortuuid python-stem python-subliminal python-tw2-forms python-txaio python-urllib3 python-verboselogs python-watchdog python-webassets python-xmp-toolkit python3-pytest-asyncio pyusb qutebrowser scipy sshuttle targetcli torbrowser-launcher veusz vigra virt-bootstrap The remaining 2585 is fine. I'll recheck after couple days. Current list: PyGreSQL bodhi diffoscope enki future gtts hatch hovercraft httpwatcher lammps mir otf2 pcs pyOpenSSL pybluez pylast pyparted pytest python-PyPDF2 python-arpeggio python-ase python-astunparse python-autobahn python-basemap python-colorclass python-conda-package-handling python-cpuinfo python-dictdiffer python-diff-cover python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-google-apputils python-hug python-inotify python-jaraco-packaging python-keystoneauth1 python-mapnik python-mechanize python-pdfminer python-pip-shims python-pluggy python-pocketlint python-portend python-progressbar2 python-prometheus_client python-rosinstall python-ryu python-setuptools-lint python-shortuuid python-stem python-subliminal python-tw2-forms python-txaio python-urllib3 python-verboselogs python-watchdog python-webassets python-xmp-toolkit python3-pytest-asyncio pyusb sshuttle targetcli torbrowser-launcher veusz vigra virt-bootstrap Current list: bodhi diffoscope enki gtts hovercraft httpwatcher lammps otf2 pcs pylast python-PyPDF2 python-arpeggio python-ase python-basemap python-conda-package-handling python-cpuinfo python-diff-cover python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-hug python-inotify python-jaraco-packaging python-mapnik python-mechanize python-pdfminer python-pip-shims python-pluggy python-pocketlint python-progressbar2 python-rosinstall python-ryu python-setuptools-lint python-shortuuid python-stem python-subliminal python-tw2-forms python-verboselogs python-watchdog python-webassets python-xmp-toolkit python3-pytest-asyncio sshuttle targetcli vigra virt-bootstrap Current list: bodhi diffoscope hovercraft httpwatcher otf2 pylast python-arpeggio python-basemap python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-hug python-jaraco-packaging python-mapnik python-pdfminer python-pip-shims python-pluggy python-progressbar2 python-ryu python-shortuuid python-stem python-subliminal python-tw2-forms python-watchdog python-xmp-toolkit vigra I've resubmitted the builds. Current list: bodhi hovercraft httpwatcher otf2 pylast python-arpeggio python-basemap python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-hug python-jaraco-packaging python-mapnik python-pip-shims python-pluggy python-progressbar2 python-ryu python-shortuuid python-stem python-subliminal python-watchdog python-xmp-toolkit Fabian, could you please stop associating EPEL updates with his bugzilla? Thanks. That's your own changelog entry that links to this report, isn't it? * Thu Oct 03 2019 Miro Hrončok <mhroncok> - 2.17.2-4 - Rebuilt for Python 3.8.0rc1 (#1748018) This looks like a processing issue to me and should be handled like that. If a package is built for EPEL then there is no match for the product and the version (this report is for Fedora Rawhide). The bug no. simply ends in the template and the updates shows up here. I can keep an eye on the updates but no promises as there are a lot of packages on the way. When you file the package update, you are supposed to fill in the bugs and notes according to reality. fedpkg tries to help you by taking the info from the changelog, but in this case, it requires a human to verify the information. You can use fedpkg update without the template by providing options on the command line: $ fedpkg update --type newpackage --bugs "" --notes "Introduce this package to EPEL 8 🥳" --stable-karma 1 -unstable-karma -1 (untested) Aha, sorry. This bug was now manually removed from all updates. To my understanding they shouldn't close this bug. (In reply to Fabian Affolter from comment #42) > This bug was now manually removed from all updates. To my understanding they > shouldn't close this bug. Thanks. The mass rebuild of Fedora 32 is in progress. I will run the script that collects the information once the mass rebuild is over. All the remaining packages will later have open FTBFS bugzillas and we can use the bugzilla depends-on/blocks thing to rack them. After the mass rebuild: hovercraft httpwatcher pylast python-django-post_office python-flake8-polyfill python-hug python-jaraco-packaging python-mapnik python-pip-shims python-progressbar2 python-watchdog python-xmp-toolkit (I've marked all the unrelated EPEL Bodhi comments as private to make this bugzilla more readable for those who cannot see them.) This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle. Changing version to 32. I don't think this needs to be tracked any more. The remaining packages are either orphaned or being taken care of. |