Bug 1244234

Hi Mark,

Since you've already done the research, let me assign this bug to you.

Thanks!
--noriko

Mark, the Alpha Freeze is on Tuesday. We need to have this done before then. Is that doable on your side? If not, I could do it.

(In reply to Miro Hrončok from comment #2)
> Mark, the Alpha Freeze is on Tuesday. We need to have this done before then.
> Is that doable on your side? If not, I could do it.

We have already pushed the core DS python updates to upstream.  Sorry this bug was not linked to the upstream ticket at that time of the work, so it was not updated.

(In reply to mreynolds from comment #3)
> We have already pushed the core DS python updates to upstream.

I know about that, the commit is linked in the bug description.

You haven't actually answered the main question:

(In reply to Miro Hrončok from comment #2)
> Mark, the Alpha Freeze is on Tuesday. We need to have this done before then.
> Is that doable on your side?

Thanks

(In reply to Miro Hrončok from comment #4)
> (In reply to mreynolds from comment #3)
> > We have already pushed the core DS python updates to upstream.
> 
> I know about that, the commit is linked in the bug description.
> 
> You haven't actually answered the main question:

I don't understand the question then.  The code is pushed to master branch, not sure what else you need from me.

Mark

> 
> (In reply to Miro Hrončok from comment #2)
> > Mark, the Alpha Freeze is on Tuesday. We need to have this done before then.
> > Is that doable on your side?
> 
> Thanks

(In reply to mreynolds from comment #5)
> (In reply to Miro Hrončok from comment #4)
> > (In reply to mreynolds from comment #3)
> > > We have already pushed the core DS python updates to upstream.
> > 
> > I know about that, the commit is linked in the bug description.
> > 
> > You haven't actually answered the main question:
> 
> I don't understand the question then.  The code is pushed to master branch,
> not sure what else you need from me.

The original bug description says: "Could you please adjust the package, take the patches or a new release if it exists, to make sure all those scripts use Python 3?"

We need this to be present in Fedora 23 before Tuesday.

You say it is pushed to (upstream) master branch, which is great, but does not fix the thing we need. Latest Fedora build is from 2015-06-24 and therefor cannot contain the commit from three weeks ago.

I'll attach a patch for dist git in a few minutes.

Created attachment 1056405 [details]
Patch for dist-git

Koji scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=10489427

Since there has been no reaction and the alpha freeze is today, I've built it.

There has been some error with empty line in sources, I believe Noriko fixed faster than me (Thank you).

During the process, I've realized f23 is one version ahead of rawhide/master, which should probably not happen.

Please do not override the necessary changes...

diff --git a/389-ds-base.spec b/389-ds-base.spec
index 68fe3cf..47f38a3 100644
--- a/389-ds-base.spec
+++ b/389-ds-base.spec
@@ -74,8 +74,11 @@ BuildRequires:    tcp_wrappers
 BuildRequires:    pam-devel
 BuildRequires:    systemd-units
 
+# for python3 macros
+BuildRequires:    python3-devel
+
 # this is needed for using semanage from our setup scripts
-Requires:         policycoreutils-python-utils
+Requires:         /usr/sbin/semanage
 
 # the following are needed for some of our scripts
 %if %{use_openldap}
===============================================================
Also, please do not apply the patch (e.g., 389-ds-base-py3.patch) against the 389-ds-base source code.

I have to undo this change you made against 389-ds-base dist-git.
commit 44d1dcbaecd86ad8683f6be8c9eb4c6497dc3288
Author: Miro Hrončok <miro>
Date:   Mon Jul 27 01:07:59 2015 +0200

    Resolves: Bug 1244234 - Use python3 in scripts in 389-ds-base
    
    Conflicts:
        389-ds-base.spec

We strictly manage our source code in the upstream git.

The 389-ds-base RPM requires both Python 2 and Python 3.

Except in very special circumstances, there is no need for one package
to drag in both Python stacks. Usually, this is a packaging error: for
example, a stray "/usr/bin/python" shebang in a Python 3 package can
introduce a Python 2 dependency.

Please split your package, or remove the stray dependencies.
There is a section on shebangs in the Python RPM Porting Guide [0]
which covers this issue.

It's ok to do this in Rawhide only, however, it would be greatly
appreciated if you could push it to Fedora 24 as well.


If anything is unclear, or if you need any kind of assistance, you can
ask on IRC (#fedora-python on Freenode), or reply here. We'll be happy
to help investigating or fixing this issue!


[0] http://python-rpm-porting.readthedocs.io/en/latest/application-modules.html#are-shebangs-dragging-you-down-to-python-2

389-ds-base-1.3.5.15-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-8660c7656f

389-ds-base-1.3.5.15-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-8f9d466bcc

389-ds-base-1.3.5.15-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-8660c7656f

389-ds-base-1.3.5.15-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-8f9d466bcc

389-ds-base-1.3.5.15-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

389-ds-base-1.3.5.15-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

The issue is still valid.

389-ds-base RPM still depends on both Python 2 and Python 3:

$ dnf --enablerepo=rawhide repoquery --requires 389-ds-base | grep python
/usr/bin/python2
/usr/bin/python3
...

Please remove the stray dependency. Let us know if you need any assistance or help.

I've seen this issue before. It's comes from the fact that one of our python scripts has a python2 in the #! line, and rpm is pulling it in automatically.

This may be difficult to correct, as we must support RHEL in addition to fedora, and python 3 is not in a state where we can depend on it in RHEL.

This leads to the issue that it's very hard to actually develop python tools that can seamlessly change between python 2 and python 3, because of the need to alter the #! line. Without using some kind of autotools substitution, sed, or other hacky solution, this is not easy to achieve.

So we are in a position where we can not without a "hack" support our needs of RHEL and Fedora.

If you could provide some advice or material on how to make this change "better", that would be excellent, but if not, we may need to continue to rely on python 2 during the 1.3.x lifecycle.

(In reply to wibrown from comment #18)
> I've seen this issue before. It's comes from the fact that one of our python
> scripts has a python2 in the #! line, and rpm is pulling it in automatically.
> 
> This may be difficult to correct, as we must support RHEL in addition to
> fedora, and python 3 is not in a state where we can depend on it in RHEL.
> 
> This leads to the issue that it's very hard to actually develop python tools
> that can seamlessly change between python 2 and python 3, because of the
> need to alter the #! line. Without using some kind of autotools
> substitution, sed, or other hacky solution, this is not easy to achieve.
> 
> So we are in a position where we can not without a "hack" support our needs
> of RHEL and Fedora.
> 
> If you could provide some advice or material on how to make this change
> "better", that would be excellent, but if not, we may need to continue to
> rely on python 2 during the 1.3.x lifecycle.

Hello. Ideally that would be only a downstream modification using sed in the SPEC file to change the shebang to point to python3.

Unfortunately there is not an elegant solution for an upstream project to support both python2 and python3 with a single shebang line for both Fedora and RHEL distros. Some packagers in order to make this happen, provide different subpackages for each python interpreter, with differently named binaries, even though the binaries provide the same functionality. It's something that must be solved at the downstream level. Hopefully at some point in the future, python will mean python3 everywhere, but it's not the case at the moment. 

Nevertheless I did some testing and the dependency on usr/bin/python2 can be removed by changing this configure line [0] to usr/bin/python3

[0] https://pagure.io/389-ds-base/blob/master/f/configure.ac#_423

Will provide a patch.

Created attachment 1280377 [details]
Make binaries pointing to python3

Created attachment 1280378 [details]
Make binaries point to python3

Amended the previous patch to change the libsemanage-python dependency to libsemanage-python3 as it was dragging python2 as well.

Output of 'rpm -qp --requires 389-ds-base-1.3.6.4-2.fc27.x86_64.rpm | grep python'

/usr/bin/python3
libsemanage-python3
policycoreutils-python-utils

Created attachment 1282192 [details]
Make binaries pointing to python3

Actually scratch the previous patch. Just adding the --with-pythonexec option at the configure will make this work. Providing another patch.

I applied your patch and rebuilt rawhide and f26.  Do you need me to do this to other branches?

(In reply to mreynolds from comment #24)
> I applied your patch and rebuilt rawhide and f26.  Do you need me to do this
> to other branches?

Thanks a lot! Rawhide and F26 are fine, no other branches required.

This is likely going to break our RHEL7 builds :( 

I have been attempting to integrate python3 for a long time, and it's so far proven impossible it maintain both python2 and python3 that works in RHEL7 and Fedora.

So far I have chosen to support RHEL7 because that is a more valuable target. I think that this may cause us some trouble to try and make our scripts blindly work with python3 only, because they will no longer work on RHEL7.

Also, just changing the pythonexec flag doesn't change the #! in the scripts, so this may still pull in the python2 files.

(In reply to wibrown from comment #26)
> This is likely going to break our RHEL7 builds :( 
> 
> I have been attempting to integrate python3 for a long time, and it's so far
> proven impossible it maintain both python2 and python3 that works in RHEL7
> and Fedora.
> 
> So far I have chosen to support RHEL7 because that is a more valuable
> target. I think that this may cause us some trouble to try and make our
> scripts blindly work with python3 only, because they will no longer work on
> RHEL7.

What do you mean with that? This is clearly a downstream change only for Fedora. Maintaining the same SPEC for RHEL and Fedora is just impossible as the distros have diverted way too much.

(In reply to wibrown from comment #27)
> Also, just changing the pythonexec flag doesn't change the #! in the
> scripts, so this may still pull in the python2 files.

It is true that it doesn't, however due to the scripts not being executable, rpmbuild will not add python2 as a dependency due to these scripts.

I've posted above how you can test the dependencies of the resulting rpm's.

(In reply to wibrown from comment #26)
> This is likely going to break our RHEL7 builds :( 
> 
 
This is upstream spec file change only - it has nothing to do with RHEL.

(In reply to wibrown from comment #26)
> This is likely going to break our RHEL7 builds :( 
> 
> I have been attempting to integrate python3 for a long time, and it's so far
> proven impossible it maintain both python2 and python3 that works in RHEL7
> and Fedora.
> 
> So far I have chosen to support RHEL7 because that is a more valuable
> target. I think that this may cause us some trouble to try and make our
> scripts blindly work with python3 only, because they will no longer work on
> RHEL7.

So, actually, we have 3 spec files:
RHEL: http://pkgs.devel.redhat.com/cgit/rpms/389-ds-base
Fedora: http://pkgs.fedoraproject.org/cgit/rpms/389-ds-base.git
Upstream: https://pagure.io/389-ds-base/blob/master/f/rpm/389-ds-base.spec.in

Since these are separate entities, we can maintain python2 for RHEL, python3 for Fedora, and whatever works for upstream. 

I'd like to generate working rpms from upstream bits for both RHEL and Fedora though.

389-ds-base-1.3.6.6-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-1a351f960e

389-ds-base-1.3.6.6-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-1a351f960e

389-ds-base-1.3.6.6-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Reopening the issue, as 389-ds-base-1.3.7.1-1.fc27 started to depend on both Python 2 and Python 3 again after the latest update.

The problem is using `#!/usr/bin/python` shebang in `ldap/admin/src/scripts/ds-replcheck` file. This should be fixed if you change shebangs preprocessing [0] to also match this file, e.g:

> sed -r -i '1s|^#!\s*/usr/bin.*python.*|#!%{__python3}|' ldap/admin/src/scripts/{*.py,ds-replcheck}


Please remove the stray dependency. Let us know if you need any assistance or help.

[0] https://src.fedoraproject.org/cgit/rpms/389-ds-base.git/tree/389-ds-base.spec#n239

New build done:  389-ds-base-1.3.7.1-2

This bug is currently reported against a Fedora version which is already unsuported.
I am changing the version to '27', the latest supported release.

Please check whether this bug is still an issue on the '27' release.
If you find this bug not being applicable on this release, please close it.

Reopening as with update to 389-ds-base-0:1.4.0.10-1, this can be reproduced again:

$ dnf --repo=rawhide repoquery --requires 389-ds-base | grep python
/usr/bin/python2
/usr/bin/python3
...

There are a couple more scripts that need to be checked:
$ grep -r  python2
usr/bin/readnsstate:#!/usr/bin/python2
usr/libexec/dirsrv/ds_selinux_port_query:#!/usr/bin/python2
usr/libexec/dirsrv/ds_selinux_enabled:#!/usr/bin/python2

Should now be fixed in 389-ds-base-1.4.0.10-2.fc28

389-ds-base-1.4.0.11-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-a00a3e9b36

389-ds-base-1.4.0.11-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-a00a3e9b36

389-ds-base-1.4.0.11-2.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.