I wonder what needs to be done here. I understand that this is not expected.

Technically, /local/ is not part of prefix here, it is part of the installation scheme inside that prefix.

Could you help me understand the CI failure? What are you trying to achieve with `pip install --prefix=/usr --root=/tmp/root .` exactly and how does it manifest as a failure? The build that failed only on Rawhide seems to be  https://copr.fedorainfracloud.org/coprs/praiskup/argparse-manpage-pull-requests/build/2990789/ -- is that assumption correct? The failure seems to be in tests. Do tests run `pip install --prefix=/usr --root=/tmp/root .` inside them?

Might it be that you call `python setup.py install` from within the tests here https://github.com/praiskup/argparse-manpage/blob/main/tests/test_examples.py#L30 that's failing? Have you considered passing a patched surrounding environment to subprocess instead of creating a new one?

I see that in the PR, you run pip in subprocess in the same way.

https://github.com/praiskup/argparse-manpage/pull/39 should solve your problem on the CI.

Thanks, but this still feels wrong.  Setuptools had it correct (rootdir + prefix is
something like DESTDIR+prefix in autotools, etc.).   But I may be wrong?
In Autotools is default prefix is /usr/local, too, and we in Fedora replace
it with /usr (rpm --eval %configure).

I agree that this feels wrong. Will discuss this with the team.

I've cross-posted this bugzilla to https://bugs.python.org/issue43976#msg409677

We keep talking about this in the team, but we have no idea how to actually fix this UX problem.

Next time, when we redesign the whole thing, we'll make sure we'll take this issue into consideration. However, with the current state of things, this might need to remain a documented caveat.

We're leaving this BZ open in the hopes of someone finding a solution one day, but we have no immediate solution right now. We don't consider this a major issue, so it is not a high priority.

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Another project is confused by this behavior in https://github.com/cockpit-project/bots/pull/3016 / https://github.com/candlepin/subscription-manager/pull/2989

At the very least, we should ensure this behavior is explained in the release notes of Fedora 36: https://pagure.io/fedora-docs/release-notes/issue/808

This should probably be documented in CommonBugs.

The section of the release notes that talks about this (drafted at https://pagure.io/fedora-docs/release-notes/issue/808) should be linked from that common bugs entry.

Miro, this bug seems too specific and complex for my little brain. I wonder if it is a good fit for CommonBugs, where only the most common and high-impact issues should be listed. If you believe it should be, can you please create the description yourself? We moved it to Ask Fedora [1] recently, see how the other entries look, and write something similar in Proposed Common Issues [2], see instructions [3], thanks.

[1] https://ask.fedoraproject.org/c/common-issues/141/none
[2] https://ask.fedoraproject.org/c/common-issues/common-issues-proposed/142
[3] https://ask.fedoraproject.org/t/about-the-proposed-common-issues-category/18527

This isn't high-impact, but I don't know of a better place than common bugs. If you feel like this does not belong there, I won't add it.

(In reply to Miro Hrončok from comment #3)
> https://github.com/praiskup/argparse-manpage/pull/39 should solve your
> problem on the CI.

JFTR, neither this seems to help now (I'm on an F36 machine).  Do we have
any work-around?

A hack I installed for now to make the upstream CI succeed:
https://github.com/praiskup/argparse-manpage/pull/54
Per the current variant of the relevant patch:
https://src.fedoraproject.org/rpms/python3.10/blob/fdfd6c1d945a381cbc85aa6d60b2369ac3ce7f50/f/00251-change-user-install-location.patch

(In reply to Pavel Raiskup from comment #14)
> (In reply to Miro Hrončok from comment #3)
> > https://github.com/praiskup/argparse-manpage/pull/39 should solve your
> > problem on the CI.
> 
> JFTR, neither this seems to help now (I'm on an F36 machine).  Do we have
> any work-around?

It depends on what does actually happen without the workaround and what are you expecting to happen. I can have a look at your tests and see what is going on. You should not need to force the RPM_BUILD_ROOT installation scheme.

https://github.com/praiskup/argparse-manpage/pull/55 -- please continue any followups there

> It depends on what does actually happen without the workaround and what are you expecting to happen.

I expected to revert the Fedora Python path behavior to the "upstream" (correct) behavior.
But your PR works, thanks.

(In reply to Pavel Raiskup from comment #18)
> > It depends on what does actually happen without the workaround and what are you expecting to happen.
> 
> I expected to revert the Fedora Python path behavior to the "upstream"
> (correct) behavior.

I mean, if I interpret it correctly -- what changes the behavior of --prefix is:
	
+    if ENABLE_USER_SITE and 'RPM_BUILD_ROOT' not in os.environ:
+        PREFIXES.insert(0, "/usr/local")

See the patch in comment #15 above.  Using "export RPM_BUILD_ROOT=fake"
I just wanted to reverse that effect of the patch.

No, that does not change the behavior of --prefix at all. That only affects what's going to be on sys.path. This used to be in the patch even in Fedora 35: https://src.fedoraproject.org/rpms/python3.10/blob/f35/f/00251-change-user-install-location.patch.

Using "export RPM_BUILD_ROOT=fake" indeed reverses the effect of that patch. But testing solely with that environment set makes your test fragile.



See the release notes https://pagure.io/fedora-docs/release-notes/blob/f36/f/modules/release-notes/pages/developers/Development_Python.adoc (not sure why this is not yet visible at https://docs.fedoraproject.org/en-US/fedora/f36/release-notes/developers/Development_Python/

(In reply to Miro Hrončok from comment #20)
> No, that does not change the behavior of --prefix at all.

Huh?  At least transitively it affects the prefix (at least in a sense
how I understand what --prefix is, aka RPM/Autotools meaning), with
empty PYTHONPATH try this (broken):

$ python3 setup.py install --root `pwd`/rootdir --prefix=/usr
...
Installing copr-cli script to /tmp/copr/cli/rootdir/usr/local/bin

And then this (OK):
$ RPM_BUILD_ROOT=fake python3 setup.py install --root `pwd`/rootdir --prefix=/usr
...
Installing copr-cli script to /tmp/copr/cli/rootdir/usr/bin

$ rpm -q python3
python3-3.10.4-1.fc36.x86_64

(In reply to Miro Hrončok from comment #7)
> Next time, when we redesign the whole thing, we'll make sure we'll take this issue into consideration.

I can see this is a documented caveat, though I fail to understand how
to track this issue in the future (when we are WONTFIX now).

(In reply to Pavel Raiskup from comment #21)
> (In reply to Miro Hrončok from comment #20)
> > No, that does not change the behavior of --prefix at all.
> 
> Huh?  At least transitively it affects the prefix (at least in a sense
> how I understand what --prefix is, aka RPM/Autotools meaning), with
> empty PYTHONPATH try this (broken):
> 
> $ python3 setup.py install --root `pwd`/rootdir --prefix=/usr
> ...
> Installing copr-cli script to /tmp/copr/cli/rootdir/usr/local/bin
> 
> And then this (OK):
> $ RPM_BUILD_ROOT=fake python3 setup.py install --root `pwd`/rootdir
> --prefix=/usr
> ...
> Installing copr-cli script to /tmp/copr/cli/rootdir/usr/bin

Yes, the environment variable affects this, but that is **not** caused by the bit of the patch you quoted. The bit you quoted only affects sys.path.


The patch changes 2 files (+ tests). site.py and sysconfig.py. The behavior described by this bug is caused by the change of sysconfig.py. The bit you quoted is site.py.


Simplified explanation of the changes:

 sysconfig.py -> where to install stuff **to**
 site.py -> where to import stuff **from**

Both changes can be effectively disabled by setting $RPM_BUILD_ROOT.



> (In reply to Miro Hrončok from comment #7)
> > Next time, when we redesign the whole thing, we'll make sure we'll take this issue into consideration.
> 
> I can see this is a documented caveat, though I fail to understand how
> to track this issue in the future (when we are WONTFIX now).

We triage all open Fedora bugzillas every 14 days. Tracking years-ahead stuff in Bugzilla does not work for us. This is tracked upstream in https://github.com/python/cpython/issues/88142#issuecomment-1093913431

I've also made sure this bugzilla is referenced in https://fedoraproject.org/wiki/SIGs/Python/UpstreamPythonPatches#00251-change-user-install-location.patch

Makes sense, thanks!

*** Bug 2083337 has been marked as a duplicate of this bug. ***

Sorry, I'm a little confused at what a valid work-around is to get expected Python behaviour.  For example, from the distutils documentation, --prefix has a default value of /usr/local. However on Fedora 36, something like "python setup.py install --prefix=/usr/local" attempts to install "/usr/local/local".  Setting RPM_BUILD_ROOT=fake appears to revert this behaviour, but I don't see why users installing Python modules this way need to concern themselves with RPM build details?

> I don't see why users installing Python modules this way need to concern themselves with RPM build details.

They don't. They need to concern themselves with installation scheme details from Python. Ideally, they should not care what the installation scheme is, but if they make assumptions about them, they might need to be adapted.


I understand that this is a change that may be quite confusing but it would really help to know exactly what is it that you want to achieve.


BTW the release notes are out https://docs.stg.fedoraproject.org/en-US/fedora/latest/release-notes/developers/Development_Python/#sect-install_scheme

At the risk of piling onto a big bug with no actionable information, I want to add colcon to the list of projects feeling adverse effects from the changes discussed here. Like other projects, we're changing --prefix, and were not expecting `local` to show up in package installation paths. At this point we seem to be playing whack-a-mole with each distro's differing patching approach and system packaging override mechanism ('RPM_BUILD_ROOT', 'DEB_PYTHON_INSTALL_LAYOUT', 'IN_NIX_BUILD', etc).

It would be nice if there was a platform-independent way to avoid this.

Rest assured, I hear you all. This is a problem. It is just that we cannot fix this in Fedora 36. I plan to look into this in the next month or two and will try to figure out a way out. In fact, let me reassign this to rawhide's 3.11 and reopen.


As for Fedora 36, all I can think of is to add yet another env var that will be more understandable than RPM_BUILD_ROOT. But considering there is no such general env var, I don't think it'll help much.

*** Bug 2094352 has been marked as a duplicate of this bug. ***

On a related grumble, the same downstream patch causes sysconfig and distutils.sysconfig to disagree on something as basic as "where do I find the Python system libraries?" I know distutils is deprecated but it's not always your choice: autoconf <= 2.69 (and latest cmake) still use distutils, but autoconf >= 2.70 prefers sysconfig, which now produces different paths. So predicting where your stuff lands in became an unfortunate guesswork on Fedora.

This also broke the path mangling in fedora flatpak runtime, https://bugzilla.redhat.com/show_bug.cgi?id=2112499

In Fedora 35 and older, the following would return /app/lib/python3.10/site-packages, but in F36 this has unexpectedly changed to /app/local/lib:

python -c "import sysconfig; purelib = sysconfig.get_path('purelib', vars=dict(base='/app')); print(purelib)"

I've created a patch that I believe should mitigate this issue. Next steps: Verify and tested for regressions.

https://github.com/fedora-python/cpython/pull/48/
https://src.fedoraproject.org/rpms/python3.11/pull-request/74

This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle.
Changing version to 37.

FEDORA-2022-63c831ce92 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-63c831ce92

FEDORA-2022-f4df41ffb4 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f4df41ffb4

FEDORA-2022-63c831ce92 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-f4df41ffb4 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

This has been fixed in Fedora 37. I plan to let it sink for a while before backporting. testing and feedback would be appreciated.

Nice, thank you! I played with the new Python package a bit on F37 and as much as I can tell, this should nicely fix the flatpak runtime issue from above.

FEDORA-2022-af60f7160a has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-af60f7160a

FEDORA-2022-850a0a68d4 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-850a0a68d4

FEDORA-2022-af60f7160a has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-af60f7160a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-af60f7160a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-850a0a68d4 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-850a0a68d4`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-850a0a68d4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

All the folks here who are impacted. Please go to https://src.fedoraproject.org/rpms/python3.10/pull-request/124, fetch the CI scratch build form https://koji.fedoraproject.org/koji/taskinfo?taskID=91129331 and let us know if this fixes the issue for you. Thank you.

I just tested the F36 build and as far as I can tell it fixes the issue for me. Thanks!

FEDORA-2022-af60f7160a has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-850a0a68d4 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-f0c3f99a57 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f0c3f99a57

FEDORA-2022-f0c3f99a57 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-a9cbedd528 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-a9cbedd528

FEDORA-2022-9d69377abe has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-9d69377abe

FEDORA-2022-9d69377abe has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-a9cbedd528 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-a9cbedd528`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-a9cbedd528

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-c072cdc3c8 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-c072cdc3c8

(In reply to Fedora Update System from comment #55)
> FEDORA-2022-c072cdc3c8 has been submitted as an update to Fedora 36.
> https://bodhi.fedoraproject.org/updates/FEDORA-2022-c072cdc3c8

This is finally the update you've been waiting for. Please test and provide karma.

FEDORA-2022-c072cdc3c8 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-c072cdc3c8`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-c072cdc3c8

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-f330bbfda2 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-f330bbfda2`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f330bbfda2

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-c072cdc3c8 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-c072cdc3c8`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-c072cdc3c8

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-f330bbfda2 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-c072cdc3c8 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2023-446b0b621c has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-446b0b621c