Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1793563

Summary: espresso fails to build with Python 3.9
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: espressoAssignee: Christoph Junghans <junghans>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: cstratak, dakingun, jgrad, junghans, mhroncok, orion
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-11 11:48:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1803234, 1785415    

Description Miro Hrončok 2020-01-21 15:08:06 UTC
espresso fails to build with Python 3.9.0a2.

 6/68 Test  #6: MpiCallbacks_test ................***Failed    0.03 sec
Invalid error code (-2) (error ring index 127 invalid)
Invalid error code (-2) (error ring index 127 invalid)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
      Start  7: ParallelScriptInterface_test
 7/68 Test  #7: ParallelScriptInterface_test .....***Failed    0.04 sec
Invalid error code (-2) (error ring index 127 invalid)
Invalid error code (-2) (error ring index 127 invalid)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)

14/68 Test #14: ParticleCache_test ...............***Failed    0.03 sec
Invalid error code (-2) (error ring index 127 invalid)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Invalid error code (-2) (error ring index 127 invalid)
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
      Start 15: Particle_test

62/68 Test #62: gather_buffer_test ...............***Failed    0.03 sec
Invalid error code (-2) (error ring index 127 invalid)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Invalid error code (-2) (error ring index 127 invalid)
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
      Start 63: scatter_buffer_test
63/68 Test #63: scatter_buffer_test ..............***Failed    0.03 sec
Invalid error code (-2) (error ring index 127 invalid)
Invalid error code (-2) (error ring index 127 invalid)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
      Start 64: all_compare_test
64/68 Test #64: all_compare_test .................***Failed    0.03 sec
Invalid error code (-2) (error ring index 127 invalid)
Invalid error code (-2) (error ring index 127 invalid)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)

67/68 Test #67: sendrecv_test ....................***Failed    0.03 sec
Invalid error code (-2) (error ring index 127 invalid)
Invalid error code (-2) (error ring index 127 invalid)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(401): 
MPID_nem_tcp_init(373).............: gethostbyname failed, 309422f13c874ef58f6092b10b85d929 (errno 0)
      Start 68: PdbParser_test


For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01170271-espresso/

For all our attempts to build espresso with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/espresso/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Christoph Junghans 2020-01-21 18:44:18 UTC
INTERNAL ERROR: invalid error code fffffffe (Ring Index out of range) in MPID_nem_tcp_init:373
Fatal error in PMPI_Init: Other MPI error, error stack:

That look like an MPI problem, not a python problem.

Comment 2 Orion Poplawski 2020-01-29 03:56:53 UTC
Interesting.  espresso is currently green https://koschei.fedoraproject.org/package/espresso so something is different in the copr.

This is with mpich.  Google doesn't seem to shed much light on the error.  I suppose it would be worth contacting the mpich folks.

Comment 3 Ben Cotton 2020-02-11 17:23:48 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 4 Christoph Junghans 2020-02-14 22:40:59 UTC
MPICH issue patched here: https://src.fedoraproject.org/rpms/mpich/pull-request/2

Comment 5 Miro Hrončok 2020-02-23 20:57:21 UTC
Unfortunately espresso still fails to build with Python 3.9.0a3 and mpich-devel x86_64 3.3.2-4.fc33:

The following tests FAILED:
	  6 - MpiCallbacks_test (Failed)
	  7 - ParallelScriptInterface_test (Failed)
	 14 - ParticleCache_test (Failed)
	 62 - gather_buffer_test (Failed)
	 63 - scatter_buffer_test (Failed)
	 64 - all_compare_test (Failed)
	 67 - sendrecv_test (Failed)

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01249881-espresso/

For all our attempts to build espresso with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/espresso/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 6 Christoph Junghans 2020-02-23 23:37:27 UTC
It fails due to the mpich error again, that go fixed in last package bump, can you trigger a build with mpich-3.3.2-4.

Comment 7 Miro Hrončok 2020-02-24 00:15:03 UTC
As said, this is with mpich-devel x86_64 3.3.2-4.fc33. Do I need to rebuild something else with that mpich prior rebuilding espresso?

Comment 8 Christoph Junghans 2020-02-24 00:49:06 UTC
You are correct, I looked at the wrong log file, now the error in the mpich build is:
Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(586)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(404): 
MPID_nem_tcp_init(375).............: gethostbyname failed, 100a54659f594f819cb20da8e07351e2 (errno 0)

Comment 9 Christoph Junghans 2020-02-24 02:02:15 UTC
I don't think this error has anything to do with espresso nor python3.9, I could reproduce this by running a simple MPI hello world on copr, see: https://github.com/junghans/mpi-hello/blob/master/mpi-hello.spec and https://download.copr.fedorainfracloud.org/results/junghans/mpi-hello/fedora-rawhide-x86_64/01250237-mpi-hello/

Comment 10 Christoph Junghans 2020-02-25 20:58:04 UTC
I can reproduce the issue locally with "mock -r fedora-rawhide-x86_64 --no-clean mpi-hello-1.0-1.fc33.src.rpm". Adding "--isolation=simple" makes it work again.

Comment 11 Christoph Junghans 2020-03-01 22:04:29 UTC
@miro, one can workaround the above MPI issue by enabling network connectivity.

Comment 12 Christoph Junghans 2020-03-09 17:40:36 UTC
@miro link to upstream fix added.

Comment 13 Miro Hrončok 2020-05-25 13:02:36 UTC
This comment is mass posted to all bugs blocking the Python 3.9 tracker, sorry if it is not 100 % relevant. When in doubt, please ask.


The Python 3.9 rebuild is in progress in a Koji side tag.

If you fix this bug, please don't rebuild the package in regular rawhide, but do it in the side tag with:

    $ fedpkg build --target=f33-python

The rebuild is progressing slowly and it is possible this package won't have all the required build dependencies yet. If that's the case, please just leave the fix committed and pushed and we will eventually rebuild it for you.

You are not asked to go and try rebuild all the missing dependencies yourself. If you know there is a bootstrap loop in the dependencies, let me know and we can untangle it together.

If you want to test your fix or reproduce the failure, you can still use the Copr repo mentioned in the initial comment of this bug: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Comment 14 Miro Hrončok 2020-05-29 07:12:18 UTC
Python 3.9 update: The f33-python side tag is currently being merged.

New builds in f33-python are no longer possible, but python3 is not yet updated to Python 3.9 in rawhide. You can check when Python is Python 3.9 with:

    $ koji wait-repo f33-build --build python3.9-3.9.0~b1-3.fc3

And build the packages normally after that.

Comment 15 Miro Hrončok 2020-06-11 11:48:50 UTC
This is a bulk close of Python 3.9 bugzillas of packages that successfully built.
If this remained open for a reason, I am sorry and feel free to reopen.