I already did some investigation. Currently, mercurial is not prepared for Python3. To put here more information, mercurial 5.1 without extensions is possible to use with python3 when building is modified appropriately. Many extensions (not sure how many) are broken - in case that users are using any additional extensions that are not provided in fedora, its utmost sure that it will be broken.

Additionally, mercurial is using their `source_to_code` function they created for transition to Python3, but unfortunately, that breaks some stuff - like hgk extension, which would work otherwise. I can fix at least partially hgk extension (with patch that I would rather not present). I will attach my curernt patch for spec and hgk (no comment) here. I am not sure whether we should already apply it or wait a little - even when we are talking about rawhide.

For now, I am sharing my copr repo with testing rpms:
  $ dnf copr enable pstodulk/mercurial
- provided packages for F30+

Any feedback and help with testing is welcomed (things that are broken and presented by failed tests it is not needed to report [0])

[0] https://copr-be.cloud.fedoraproject.org/results/pstodulk/mercurial/fedora-rawhide-x86_64/00995430-mercurial/builder-live.log.gz
[1] https://copr.fedorainfracloud.org/coprs/pstodulk/mercurial/

More info about active port of Mercurial to Python3 can be seen here:
  https://www.mercurial-scm.org/wiki/Python3

Created attachment 1601192 [details]
first attempt for hg with Py3

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Just sharing a general plan we have:

The current plan is to remove packages with dependency on Python 2 from Fedora 32 in the middle of November 2019. If you want to keep your package in Fedora after that date and you cannot port it to Python 3 yet, you need to request a FESCo exception for the package and all its Python 2 dependencies (even transitive) [1]. If you don't want to maintain it anymore, please orphan the package.

If you're considering filing the exception request, let us know. We can help (for example, we can help find all the dependencies).

[1] https://fedoraproject.org/wiki/Changes/RetirePython2#FESCo_exceptions

Note that a python2 exception not really is an option as the only solution. As many other packages (like python2-urwid) has been removed already, other packages depending on Mercurial *must* have a Python3 Mercurial ASAP so they have the possibility of moving forward.

An instantaneous Mercurial package switch from python2 to python3 will also cause a lot of breakage for all dependencies and is not a good option. Side by side seems like the only good option to me. Perhaps ask for exception to keeep the python2 package around in Fedora 31.

Yes, a Python 2 exception is the only solution. Please file for one. Here is an example: https://pagure.io/fesco/issue/2208
but please mention plans for urwid as well, if needed.


We didn't plan to remove python2-urwid while other packages depend on it. What happened is that the maintainer of python-urwid orphaned the package. When it got retired die to lack of maintenance, someone stepped to maintain it -- but took with the python3-urwid variant only.

Do you want to take python2-urwid? We can help to bring it back (if it gets a FESCo exception with e.g. hgview-curses). It would need a miantainer, though.

Honestly, I do not plan to take any python2 exception here. Mercurial will be maybe kept for Python2 on F31, but F32 is already planned to be on Python3 - switch to Python3 mercurial in rawhide is planned for tonight. See the thread [0] about that. If we success to prepare everything for Python3 (including other rpms that depends on mercurial) before beta freeze of F31, maybe we could do the update even for F31 - but I guess this will not happen. 

Currently, mercurial is not the only problem we have, there are other rpms that already started to have broken dependencies because of orphaned python2 libraries.

[0] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/7MMYNWCVDDUENRTDQQYMZCLSDFNF6D4P/

@pstodulk: The problem is that an instantaneous switch to Python3 will introduce even more broken dependencies. Please give us a chance to move to a Python 3 Mercurial without having broken dependencies. Broken dependencies would not only make it important, but also urgent.

@petr: Thanks for the explanation of python2-urwid and the idea of getting it back. But no big deal. I will fix my broken dependency by migrating hgview to Python 3 when I can. That's why we need Mercurial packaged for Python 3. (But again: that does *not* mean that we need Python 2 Mercurial to go away while other packages depend on it.)

@mads - we are going already to have broken dependencies with python2 - what is making here different level of urgency? As mercurial is not used by so many people (and how many of them are using rawhide..) what is so big problem to switch to F31 branches that are pretty close to rawhide now and switch back later when porting will be done?... They can even stop upgrades of mercurial related rpms until it will be resolved. Or create separate mercurial COPR repo for all related rpms and people can use that.

Anyway, I spent tonight with mercurial to look at way to provide Py2 & Py3 mercurial builds. Honestly, I would like to kickoff Py2-mercurial from the train..

Created attachment 1608765 [details]
POC: create subpackages for py2 & py3

The most probably it's broken. I haven't had time to test it yet but maybe there is already something done about that. Extensions will be almost sure broken for such builds but that's all what I could do for now.

One of the main problems here is the executable script. To be able to have script for py2 and py3, I created /usr/bin/hg3 script for mercurial-py3 packages. Not sure whether it is good idea, but it is the only one I had in my mind how to resolve it under one component.

What's the current plan? Is it requesting an exception, switching to python3 only, or retiring mercurial?

Current plan is to have both, mercurial for Python2 & Python3. But the correct solution is not prepared yet. To have just Python3 version is easy otherwise. Just bunch of other rpms will be broken immediately in such case. So we are trying to have both.

In order to keep the Python 2 version a FESCo exception is needed. We can help you draft the request. In order to do that, we need some info:

- Currently, mercurial BuildRequires python2-docutils. Is that dependency mandatory? Can it build docs with python3-docutils instead?

- From the following packages, are you planning to keep some on Python 2 as well? Should they be included in the exception ?

  git-cinnabar
  git-remote-hg
  hg-git
  hgview
  tortoisehg
  trac-mercurial-plugin

Note that they bring a handful of other Python 2 packages.

As we still don't have Mercurial on Python 3 available, all dependent packages are *forced* to stay on Python 2 even if we have plans for moving to Python 3.

I ported hgview to python 3, and it is in latest upstream release.

TortoiseHg is also getting there. The development branch with some patches works with Python 3. It would be valuable to have it packaged next to the stable python2 package and get feedback. But if necessary, it could be the only one.

But yes, because of the timing, I guess everything that depends on Mercurial packaging will need an exception too, and should ideally be included in the exception. But with python2 support falling a part, I doubt it will be feasible for other packages to use it anyway.

Keeping mercurial itself and git-cinnabar on Python 2 would be easy; they
don't pull in a lot of other packages:

- mercurial
  - python-docutils (probably only for docs, which can be done on py3 or pregenerated)
- git-cinnabar
  - mercurial
  - python-nose (test-only)
    - python-coverage

trac-mercurial-plugin depends just on
  - mercurial
  - Trac (which is probably also getting an exception)
  - python2-setuptools (which shouldn't be hard to get an exception for; I'll omit it below)



hg-git would be more challenging to coordinate, since it depends on dulwich,
which drags in a lot of network/SSL-related packages.
We'd need to coordinate with the maintainers, and possibly find people to
(co-)maintain the Python 2 bits:

- hg-git
  - mercurial
  - python-dulwich
    - python-certifi
    - python-docutils (doc-only)
    - python-nose (test only)
    - python-urllib3
      - python-idna
      - python-pysocks
        - python-psutil (has an exception already)
        - python2-pytest (test-only)
      - python-six
      - python-backports-ssl_match_hostname (orphaned)
      - python-ipadress (orphaned)
- git-remote-hg
  - mercurial
  - hg-git (above)



And then there are GUI tools, which depend on Qt4, Qt5 *and* GTK:

- hgview
  - mercurial
  - python-docutils (for run time)
  - python-inotify
  - python-pygments
    - python-nose (test-only)
  - PyQt4
    - dbus-python
      - pygobject3 (see below)
      - python-docutils (doc-only?)
    - sip
  - qscintilla
    - PyQt4 (see above)
    - python-qt5
      - dbus-python, sip (see above)
- tortoisehg:
  - mercurial
  - pygobject3
    - pycairo (has an exception already)
  - python-iniparse
    - python-six
  - python-pygments
  - qscintilla, python-qt5 (see above)
  - python-enum34

I went ahead and requested an exception request (only) for mercurial: https://pagure.io/fesco/issue/2243

Hi guys, sorry for delay. I've got sick again. Yes, we will need exception for otherpakcages as well. I believe I will be able to look at it tomorrow. At least, rebase of mercurial should be done finally for F31. The new version of mercurial requires some small changes in spec as well.

Thanks Miro for opening request for mercurial.

I rebased mercurial to version 5.1.2 in rawhide. As well, you can find opened PR to add the mercurial-python3 subpackage (among other things). Hopefully things will be moving faster now.

I looked at the python2-docutils dependency but I it doesn't look that it can be easily dropped. We will need it as well now, unless someone modify build&installation scripts.

Here is the PR: https://src.fedoraproject.org/rpms/mercurial/pull-request/5

FYI, about git-remote-hg, I dropped the build dependency on hg-git. Checking the code, there is nothing related to hg-git and was there by mistake.

- or at least, nowadays it was invalid.

Thanks for the Mercurial py3 subpackage in rawhide. https://koji.fedoraproject.org/koji/taskinfo?taskID=39064842 is now waiting for it.

But also, this py3 packaging of Mercurial 5.1 has a big problem:

Mercurial 5.1 use a custom importer with byte code conversion for Python 3. The .pyc files are thus in a special format. When brp-python-bytecompile overwrite them with standard byte codes, Mercurial will have to re-import them (and try to write out new .pyc files). That can give several seconds of startup time, as seen when running for example a simple `hg debuginstall` (as a regular user or with SELinux enabled). mercurial-py3 is thus not really useful ... oir a significant unnecessary regression.

That custom importer is however gone in Mercurial 5.2, released earlier this month. Python 3 support is also out of preview (except on Windows), and HGPYTHON3 is no longer needed. I have updated the upstream Fedora packaging to Python 3 on Fedora 31.

I thus suggest upgrading rawhide to Mercurial 5.2 ASAP and clean up the packaging.

Considering the time pressure and bitrot of Python2 in Fedora 31, I also suggest backporting the new Python 3 sub package there. Everything considered, I think the least risky option is to upgrade Mercurial to 5.2 there too, so users have a (long term) less risky option of using Python 3.

As a short them workaround, brp-python-bytecompile can be turned of for mercurial.

Thanks Mads for feedback. I plan to upgrade mercurial to 5.2 in the following week. I wanted to do it the week ago but...

Anyway, I am not sure that rebase in stable fedora is good idea, because I believe it will break additional packages. I would really continue the work just in rawhide. Regarding the time, if anyone else is able to work on that, feel free.

I created new PR with Mercurial 5.2: https://src.fedoraproject.org/rpms/mercurial/pull-request/6#

I kept there still the HGPYTHON3 envar which I use to detect that /usr/bin/hg3 should be created (I know it would be used just the PYTHON envar, but this seems to me more obvious solution for now). As well, I found that the previous build was not created (5.1.2-2; sorry guys. that was pebkac combined with time pressure and I have installed local build). So the *-py3 subpackage has not been delivered in the rawhide. It will be finally with the v5.2. 

If you have time, please check the PR. In the worst case, I will merge it this afternoon (Tue: CET) to deliver builds finally.

I will send email during the day to the other maintainers of components depedent on mercurial to see how does it look like with Py3 porting.

How is the work to get python2 out of mercurial coming along?
It looks like getting python2 out of mercurial was progressing, and then it stopped halfway through it's progress.

https://src.fedoraproject.org/rpms/mercurial/pull-request/11 is the work in progress PR. But AFAIK Petr is not working on this in his Red Hat paid time, so indeed, it stopped halfway through.

I started to work on it, but right now I'm in the phase of getting to know Mercurial. I wanted to sync with Petr, but according to his IRC nick he is sick right now.

I had meeting with pstodulk and I'm going to continue on working on this.
Here is my PR[0] where I'm going to post changes and updates.
Even though the work is halfway through, Mercurial builds successfully. 

[0]https://src.fedoraproject.org/rpms/mercurial/pull-request/13

This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Ondrej Pohorelsky is working on this. Mercurial is going to be Python3 only for F34 (current rawhide).

This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.