1914818 – python-configargparse fails to build with Python 3.10: argparse output changed

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1914818 - python-configargparse fails to build with Python 3.10: argparse output changed

Summary: python-configargparse fails to build with Python 3.10: argparse output changed

Keywords:
Status:	CLOSED RAWHIDE
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	python-configargparse
Sub Component:
Version:	rawhide
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	---
Assignee:	Felix Schwarz
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1968938 (view as bug list)
Depends On:
Blocks:	1821189 PYTHON3.10 F35FTBFS, RAWHIDEFTBFS F35FailsToInstall, RAWHIDEFailsToInstall 1968784 1968938 1969174 1969186
TreeView+	depends on / blocked

Reported:	2021-01-11 09:44 UTC by Tomáš Hrnčiar
Modified:	2021-07-03 17:36 UTC (History)
CC List:	11 users (show)
Fixed In Version:	python-configargparse-1.4.1-1.fc35
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-06-21 21:15:33 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Tomáš Hrnčiar 2021-01-11 09:44:20 UTC

python-configargparse fails to build with Python 3.10.0a4.

=================================== FAILURES ===================================
_______________________ TestBasicUseCases.testBasicCase2 _______________________

self = <tests.test_configargparse.TestBasicUseCases testMethod=testBasicCase2>
use_groups = False

    def testBasicCase2(self, use_groups=False):
    
        ## Test command line, config file and env var values
        default_config_file = tempfile.NamedTemporaryFile(mode="w", delete=True)
        default_config_file.flush()
    
        p = self.initParser(default_config_files=['/etc/settings.ini',
                '/home/jeff/.user_settings', default_config_file.name])
        p.add_arg('vcf', nargs='+', help='Variant file(s)')
        if not use_groups:
            self.add_arg('--genome', help='Path to genome file', required=True)
            self.add_arg('-v', dest='verbose', action='store_true')
            self.add_arg('-g', '--my-cfg-file', required=True,
                         is_config_file=True)
            self.add_arg('-d', '--dbsnp', env_var='DBSNP_PATH')
            self.add_arg('-f', '--format',
                         choices=["BED", "MAF", "VCF", "WIG", "R"],
                         dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT",
                         default="BED")
        else:
            g = p.add_argument_group(title="g1")
            g.add_arg('--genome', help='Path to genome file', required=True)
            g.add_arg('-v', dest='verbose', action='store_true')
            g.add_arg('-g', '--my-cfg-file', required=True,
                      is_config_file=True)
            g = p.add_argument_group(title="g2")
            g.add_arg('-d', '--dbsnp', env_var='DBSNP_PATH')
            g.add_arg('-f', '--format',
                      choices=["BED", "MAF", "VCF", "WIG", "R"],
                      dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT",
                      default="BED")
    
        # make sure required args are enforced
        self.assertParseArgsRaises("too few arg"
                                   if sys.version_info.major < 3 else
                                   "the following arguments are required: vcf, -g/--my-cfg-file",
                                   args="--genome hg19")
        self.assertParseArgsRaises("not found: file.txt", args="-g file.txt")
    
        # check values after setting args on command line
        config_file2 = tempfile.NamedTemporaryFile(mode="w", delete=True)
        config_file2.flush()
    
        ns = self.parse(args="--genome hg19 -g %s bla.vcf " % config_file2.name)
        self.assertEqual(ns.genome, "hg19")
        self.assertEqual(ns.verbose, False)
        self.assertIsNone(ns.dbsnp)
        self.assertEqual(ns.fmt, "BED")
        self.assertListEqual(ns.vcf, ["bla.vcf"])
    
        self.assertRegex(self.format_values(),
            'Command Line Args:   --genome hg19 -g [^\\s]+ bla.vcf\n'
            'Defaults:\n'
            '  --format: \\s+ BED\n')
    
        # check precedence: args > env > config > default using the --format arg
        default_config_file.write("--format MAF")
        default_config_file.flush()
        ns = self.parse(args="--genome hg19 -g %s f.vcf " % config_file2.name)
        self.assertEqual(ns.fmt, "MAF")
        self.assertRegex(self.format_values(),
            'Command Line Args:   --genome hg19 -g [^\\s]+ f.vcf\n'
            'Config File \\([^\\s]+\\):\n'
            '  --format: \\s+ MAF\n')
    
        config_file2.write("--format VCF")
        config_file2.flush()
        ns = self.parse(args="--genome hg19 -g %s f.vcf " % config_file2.name)
        self.assertEqual(ns.fmt, "VCF")
        self.assertRegex(self.format_values(),
            'Command Line Args:   --genome hg19 -g [^\\s]+ f.vcf\n'
            'Config File \\([^\\s]+\\):\n'
            '  --format: \\s+ VCF\n')
    
        ns = self.parse(env_vars={"OUTPUT_FORMAT":"R", "DBSNP_PATH":"/a/b.vcf"},
            args="--genome hg19 -g %s f.vcf " % config_file2.name)
        self.assertEqual(ns.fmt, "R")
        self.assertEqual(ns.dbsnp, "/a/b.vcf")
        self.assertRegex(self.format_values(),
            'Command Line Args:   --genome hg19 -g [^\\s]+ f.vcf\n'
            'Environment Variables:\n'
            '  DBSNP_PATH: \\s+ /a/b.vcf\n'
            '  OUTPUT_FORMAT: \\s+ R\n')
    
        ns = self.parse(env_vars={"OUTPUT_FORMAT":"R", "DBSNP_PATH":"/a/b.vcf",
                                  "ANOTHER_VAR":"something"},
            args="--genome hg19 -g %s --format WIG f.vcf" % config_file2.name)
        self.assertEqual(ns.fmt, "WIG")
        self.assertEqual(ns.dbsnp, "/a/b.vcf")
        self.assertRegex(self.format_values(),
            'Command Line Args:   --genome hg19 -g [^\\s]+ --format WIG f.vcf\n'
            'Environment Variables:\n'
            '  DBSNP_PATH: \\s+ /a/b.vcf\n')
    
        if not use_groups:
>           self.assertRegex(self.format_help(),
                'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE\n?'
                '\\s+\\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n' +
                9*r'(.+\s+)'+  # repeated 8 times because .+ matches atmost 1 line
                'positional arguments:\n'
                '  vcf \\s+ Variant file\\(s\\)\n\n'
                'optional arguments:\n'
                '  -h, --help \\s+ show this help message and exit\n'
                '  --genome GENOME \\s+ Path to genome file\n'
                '  -v\n'
                '  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n'
                '  -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n'
                '  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n')
E           AssertionError: Regex didn't match: 'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE\n?\\s+\\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)(.+\\s+)positional arguments:\n  vcf \\s+ Variant file\\(s\\)\n\noptional arguments:\n  -h, --help \\s+ show this help message and exit\n  --genome GENOME \\s+ Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n  -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n' not found in "usage: pytest-3.10 [-h] --genome GENOME [-v] -g MY_CFG_FILE [-d DBSNP]\n                   [-f FRMT]\n                   vcf [vcf ...]\n\nArgs that start with '--' (eg. --genome) can also be set in a config file\n(/etc/settings.ini or /home/jeff/.user_settings or /tmp/tmpfj58thk7 or\nspecified via -g). Config file syntax allows: key=value, flag=true,\nstuff=[a,b,c] (for details, see syntax at https://goo.gl/R74nmi). If an arg is\nspecified in more than one place, then commandline values override environment\nvariables which override config file values which override defaults.\n\npositional arguments:\n  vcf                   Variant file(s)\n\noptions:\n  -h, --help            show this help message and exit\n  --genome GENOME       Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n  -d DBSNP, --dbsnp DBSNP\n                        [env var: DBSNP_PATH]\n  -f FRMT, --format FRMT\n                        [env var: OUTPUT_FORMAT]\n"

tests/test_configargparse.py:250: AssertionError
_________________ TestBasicUseCases.testBasicCase2_WithGroups __________________

self = <tests.test_configargparse.TestBasicUseCases testMethod=testBasicCase2_WithGroups>

    def testBasicCase2_WithGroups(self):
>       self.testBasicCase2(use_groups=True)

tests/test_configargparse.py:290: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_configargparse.py:264: in testBasicCase2
    self.assertRegex(self.format_help(),
E   AssertionError: Regex didn't match: 'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE\n?\\s+\\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+positional arguments:\n  vcf \\s+ Variant file\\(s\\)\n\noptional arguments:\n  -h, --help \\s+ show this help message and exit\n\ng1:\n  --genome GENOME \\s+ Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\ng2:\n  -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n' not found in "usage: pytest-3.10 [-h] --genome GENOME [-v] -g MY_CFG_FILE [-d DBSNP]\n                   [-f FRMT]\n                   vcf [vcf ...]\n\nArgs that start with '--' (eg. --genome) can also be set in a config file\n(/etc/settings.ini or /home/jeff/.user_settings or /tmp/tmpmlvqubln or\nspecified via -g). Config file syntax allows: key=value, flag=true,\nstuff=[a,b,c] (for details, see syntax at https://goo.gl/R74nmi). If an arg is\nspecified in more than one place, then commandline values override environment\nvariables which override config file values which override defaults.\n\npositional arguments:\n  vcf                   Variant file(s)\n\noptions:\n  -h, --help            show this help message and exit\n\ng1:\n  --genome GENOME       Path to genome file\n  -v\n  -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\ng2:\n  -d DBSNP, --dbsnp DBSNP\n                        [env var: DBSNP_PATH]\n  -f FRMT, --format FRMT\n                        [env var: OUTPUT_FORMAT]\n"
_________________ TestBasicUseCases.testMutuallyExclusiveArgs __________________

self = <tests.test_configargparse.TestBasicUseCases testMethod=testMutuallyExclusiveArgs>

    def testMutuallyExclusiveArgs(self):
        config_file = tempfile.NamedTemporaryFile(mode="w", delete=True)
    
        p = self.parser
        g = p.add_argument_group(title="group1")
        g.add_arg('--genome', help='Path to genome file', required=True)
        g.add_arg('-v', dest='verbose', action='store_true')
    
        g = p.add_mutually_exclusive_group(required=True)
        g.add_arg('-f1', '--type1-cfg-file', is_config_file=True)
        g.add_arg('-f2', '--type2-cfg-file', is_config_file=True)
    
        g = p.add_mutually_exclusive_group(required=True)
        g.add_arg('-f', '--format', choices=["BED", "MAF", "VCF", "WIG", "R"],
                     dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT",
                     default="BED")
        g.add_arg('-b', '--bam', dest='fmt', action="store_const", const="BAM",
                  env_var='BAM_FORMAT')
    
        ns = self.parse(args="--genome hg19 -f1 %s --bam" % config_file.name)
        self.assertEqual(ns.genome, "hg19")
        self.assertEqual(ns.verbose, False)
        self.assertEqual(ns.fmt, "BAM")
    
        ns = self.parse(env_vars={"BAM_FORMAT" : "true"},
                        args="--genome hg19 -f1 %s" % config_file.name)
        self.assertEqual(ns.genome, "hg19")
        self.assertEqual(ns.verbose, False)
        self.assertEqual(ns.fmt, "BAM")
        self.assertRegex(self.format_values(),
            'Command Line Args:   --genome hg19 -f1 [^\\s]+\n'
            'Environment Variables:\n'
            '  BAM_FORMAT: \\s+ true\n'
            'Defaults:\n'
            '  --format: \\s+ BED\n')
    
>       self.assertRegex(self.format_help(),
            r'usage: .* \[-h\] --genome GENOME \[-v\]\s+ \(-f1 TYPE1_CFG_FILE \|'
            ' \\s*-f2 TYPE2_CFG_FILE\\)\\s+\\(-f FRMT \\| -b\\)\n\n' +
            7*r'.+\s+'+  # repeated 7 times because .+ matches atmost 1 line
            'optional arguments:\n'
            '  -h, --help            show this help message and exit\n'
            '  -f1 TYPE1_CFG_FILE, --type1-cfg-file TYPE1_CFG_FILE\n'
            '  -f2 TYPE2_CFG_FILE, --type2-cfg-file TYPE2_CFG_FILE\n'
            '  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n'
            '  -b, --bam\\s+\\[env var: BAM_FORMAT\\]\n\n'
            'group1:\n'
            '  --genome GENOME       Path to genome file\n'
            '  -v\n')
E       AssertionError: Regex didn't match: 'usage: .* \\[-h\\] --genome GENOME \\[-v\\]\\s+ \\(-f1 TYPE1_CFG_FILE \\| \\s*-f2 TYPE2_CFG_FILE\\)\\s+\\(-f FRMT \\| -b\\)\n\n.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+.+\\s+optional arguments:\n  -h, --help            show this help message and exit\n  -f1 TYPE1_CFG_FILE, --type1-cfg-file TYPE1_CFG_FILE\n  -f2 TYPE2_CFG_FILE, --type2-cfg-file TYPE2_CFG_FILE\n  -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n  -b, --bam\\s+\\[env var: BAM_FORMAT\\]\n\ngroup1:\n  --genome GENOME       Path to genome file\n  -v\n' not found in "usage: pytest-3.10 [-h] --genome GENOME [-v]\n                   (-f1 TYPE1_CFG_FILE | -f2 TYPE2_CFG_FILE) (-f FRMT | -b)\n\nArgs that start with '--' (eg. --genome) can also be set in a config file\n(specified via -f1 or -f2). Config file syntax allows: key=value, flag=true,\nstuff=[a,b,c] (for details, see syntax at https://goo.gl/R74nmi). If an arg is\nspecified in more than one place, then commandline values override environment\nvariables which override config file values which override defaults.\n\noptions:\n  -h, --help            show this help message and exit\n  -f1 TYPE1_CFG_FILE, --type1-cfg-file TYPE1_CFG_FILE\n  -f2 TYPE2_CFG_FILE, --type2-cfg-file TYPE2_CFG_FILE\n  -f FRMT, --format FRMT\n                        [env var: OUTPUT_FORMAT]\n  -b, --bam             [env var: BAM_FORMAT]\n\ngroup1:\n  --genome GENOME       Path to genome file\n  -v\n"

tests/test_configargparse.py:361: AssertionError
=========================== short test summary info ============================
FAILED tests/test_configargparse.py::TestBasicUseCases::testBasicCase2 - Asse...
FAILED tests/test_configargparse.py::TestBasicUseCases::testBasicCase2_WithGroups
FAILED tests/test_configargparse.py::TestBasicUseCases::testMutuallyExclusiveArgs
================= 3 failed, 18 passed, 14 deselected in 2.88s ==================

bpo-9694: Argparse help no longer uses the confusing phrase, “optional arguments”. It uses “options” instead.

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01870103-python-configargparse/

For all our attempts to build python-configargparse with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/python-configargparse/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2021-01-11 09:59:02 UTC

Argparse output has changed from:


  optional arguments:


to:


  options:


https://bugs.python.org/issue9694
https://bugs.python.org/issue42870

Comment 2 Ben Cotton 2021-02-09 15:39:15 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 3 Miro Hrončok 2021-06-04 20:13:00 UTC

This is a mass-posted update. Sorry if it is not 100% accurate to this bugzilla.


The Python 3.10 rebuild is in progress in a Koji side tag. If you manage to fix the problem, please commit the fix in the rawhide branch, but don't build the package in regular rawhide.

You can either build the package in the side tag, with:

    $ fedpkg build --target=f35-python

Or you can the build and we will eventually build it for you.

Note that the rebuild is still in progress, so not all (build) dependencies of this package might be available right away.

Thanks.

See also https://fedoraproject.org/wiki/Changes/Python3.10

If you have general questions about the rebuild, please use this mailing list thread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/G47SGOYIQLRDTWGOSLSWERZSSHXDEDH5/

Comment 4 Miro Hrončok 2021-06-06 22:24:37 UTC

Just a note that this blocks the certbot stack to rebuild with Python 3.10.

Comment 5 Miro Hrončok 2021-06-07 22:57:55 UTC

The f35-python side tag has been merged to Rawhide. From now on, build as you would normally build.

Comment 6 Miro Hrončok 2021-06-08 11:23:05 UTC

*** Bug 1968938 has been marked as a duplicate of this bug. ***

Comment 7 Miro Hrončok 2021-06-15 20:23:59 UTC

Hello,

This is the first reminder (step 3 from https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/#_package_removal_for_long_standing_ftbfs_and_fti_bugs).

If you know about this problem and are planning on fixing it, please acknowledge so by setting the bug status to ASSIGNED. If you don't have time to maintain this package, consider orphaning it, so maintainers of dependent packages realize the problem.

Comment 8 Felix Schwarz 2021-06-16 06:17:55 UTC

I think this PR is supposed to provide Python 3.10 compatibility: https://src.fedoraproject.org/rpms/python-configargparse/pull-request/4

Upstream issue: https://github.com/bw2/ConfigArgParse/issues/225

Comment 9 Major Hayden 🤠 2021-06-16 12:19:17 UTC

The PR should fix bug 1928695, too. 😉

Comment 10 Felix Schwarz 2021-06-21 21:15:33 UTC

fixed by Major's pull request: https://src.fedoraproject.org/rpms/python-configargparse/pull-request/4

new build in rawhide.

Comment 11 Major Hayden 🤠 2021-07-03 13:51:36 UTC

Thanks, Felix! 🎉

Comment 12 Felix Schwarz 2021-07-03 17:36:35 UTC

Well, you did all the heavy lifting, I just pressed a button :-) So thank you!

Note You need to log in before you can comment on or make changes to this bug.