Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1889726

Summary: unbound fails to build with Python 3.10: undefined reference to `PyParser_SimpleParseFile'
Product: [Fedora] Fedora Reporter: Tomáš Hrnčiar <thrnciar>
Component: unboundAssignee: Paul Wouters <pwouters>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: aegorenk, akhaitov, mhroncok, pemensik, pj.pandit, pwouters, thrnciar, vstinner
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 1.13.1-2.fc35 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-16 14:59:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1890881    

Description Tomáš Hrnčiar 2020-10-20 13:13:41 UTC
unbound fails to build with Python 3.10.0a1.

This report is automated and not very verbose, but we'll try to get back here with details.

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01713325-unbound/

For all our attempts to build unbound with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/unbound/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2020-10-20 13:20:12 UTC
The problem is:

/builddir/build/BUILD/unbound-1.10.1/unbound-1.10.1/pythonmod/pythonmod.c:371: undefined reference to `PyParser_SimpleParseFile'

I don't see this explicitly mentioned in https://docs.python.org/3.10/whatsnew/3.10.html but I see it mentioned in in https://docs.python.org/3.9/whatsnew/3.9.html#deprecated

Comment 2 Miro Hrončok 2020-11-10 12:41:04 UTC
Marking as high severity, because unbound is needed by gnutls.

Comment 3 Petr Menšík 2020-11-10 18:21:00 UTC
Hi Tomáš,

I lack python expertise to see obvious fix. It seems offending line [1] is executed only to obtain Exception to print into log.

There is already PyRun_SimpleFile, which runs the script. It would print execption to stderr.
Second time, it runs the same code to include it in log. It should prefix the second message with:

./unbound-checkconf unbound.conf 
  File "./pythonmod/examples/log.py", line 63
    print "-"*100
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("-"*100)?
[1605031806] unbound-checkconf[2797510:0] error: pythonmod: can't parse Python script ./pythonmod/examples/log.py
[1605031806] unbound-checkconf[2797510:0] error: pythonmod: python error:   File "./pythonmod/examples/log.py", line 63
    print "-"*100
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("-"*100)?

This part is repeated just to be catched also in logs under unbound. Could we instead modify PyRun_SimpleFile call to produce just one exception,
then print it to stderr once and once into the log? We could then remove deprecated function just for that purpose.

[1605031806] unbound-checkconf[2797510:0] error: pythonmod: can't parse Python script ./pythonmod/examples/log.py
[1605031806] unbound-checkconf[2797510:0] error: pythonmod: python error:   File "./pythonmod/examples/log.py", line 63

But it seems PyRun_SimpleFile does not throw Exception. Can you recommend variant or flags, which would make it raise an Exception, which log_py_err() would then to log file?
After commenting out PyParser_SimpleParseFile it reports None, so it did not already raise an exception.

[1605032274] unbound-checkconf[2798279:0] error: pythonmod: python error: NoneType: None

I could not find any information about exceptions in documentation [2]. Is there better place to look into?

Reproducing the error is quite simple. Enter unbound build directory and unbound-1.* in it.

cd unbound-1.*/unbound-1.*

cat <<EOF > unbound.conf
server:
	module-config: "python iterator"

python:
	python-script: ./pythonmod/examples/log.py
EOF

./unbound-checkconf unbound.conf

Because examples are written for python2, they emit error.

1. https://github.com/NLnetLabs/unbound/blob/master/pythonmod/pythonmod.c#L371
2. https://docs.python.org/3.9/c-api/veryhigh.html#c.PyParser_SimpleParseFile

Comment 4 Miro Hrončok 2020-11-10 19:30:40 UTC
Victor is better suited to answer your question.

Comment 5 Victor Stinner 2020-11-12 15:19:00 UTC
> I don't see this explicitly mentioned in https://docs.python.org/3.10/whatsnew/3.10.html but I see it mentioned in in https://docs.python.org/3.9/whatsnew/3.9.html#deprecated

I asked to explicitly document removed functions:
https://bugs.python.org/issue40939#msg380826

unbound used the removed function in pythonmod/pythonmod.c. Extract of unbound-1.12.0.tar.gz:

   if (PyRun_SimpleFile(script_py, pe->fname) < 0) {
      log_err("pythonmod: can't parse Python script %s", pe->fname);
      /* print the error to logs too, run it again */
      fseek(script_py, 0, SEEK_SET);
      /* we don't run the file, like this, because then side-effects
       *    s = PyRun_File(script_py, pe->fname, Py_file_input, 
       *        PyModule_GetDict(PyImport_AddModule("__main__")), pe->dict);
       * could happen (again). Instead we parse the file again to get
       * the error string in the logs, for when the daemon has stderr
       * removed.  SimpleFile run already printed to stderr, for then
       * this is called from unbound-checkconf or unbound -dd the user
       * has a nice formatted error.
      */
      /* ignore the NULL return of _node, it is NULL due to the parse failure
       * that we are expecting */
      (void)PyParser_SimpleParseFile(script_py, pe->fname, Py_file_input);
      log_py_err();
      PyGILState_Release(gil);
      fclose(script_py);
      return 0;
   }


> But it seems PyRun_SimpleFile does not throw Exception. Can you recommend variant or flags, which would make it raise an Exception, which log_py_err() would then to log file?

I'm not aware of any replacement for PyParser_SimpleParseFile() :-( I asked on the usptream issue:
https://bugs.python.org/issue40939#msg380828

Comment 6 Miro Hrončok 2020-12-11 14:42:47 UTC
Victor, how do we move this forward?

Comment 7 Victor Stinner 2020-12-16 16:17:08 UTC
Miro:
> Victor, how do we move this forward?

I reported the issue to upstream: https://github.com/NLnetLabs/unbound/issues/378

Comment 8 Miro Hrončok 2021-02-02 16:41:56 UTC
We need to have this fixed in order to be able to do the actual upgrade in Fedora. 

There are now 2 upstream issues, but no movement. Victor, could you please help with the fix?

Comment 9 Ben Cotton 2021-02-09 16:11:46 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 10 Victor Stinner 2021-02-15 15:22:14 UTC
Oh, it seems like Wouter Wijngaards fixed the issue 6 days ago, without updating my issue https://github.com/NLnetLabs/unbound/issues/378

I created a PR to backport his change: https://github.com/NLnetLabs/unbound/issues/378

Comment 11 Victor Stinner 2021-02-15 15:24:00 UTC
> I created a PR to backport his change: https://github.com/NLnetLabs/unbound/issues/378

Oops, the correct URL is: https://src.fedoraproject.org/rpms/unbound/pull-request/6

Comment 12 Miro Hrončok 2021-02-16 14:59:56 UTC
Thank You!