Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1952351
Summary: | python-parso fails to build with Python 3.10: Changed enum repr causes test failure | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Tomáš Hrnčiar <thrnciar> |
Component: | python-parso | Assignee: | Miro Hrončok <mhroncok> |
Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | rawhide | CC: | carl, mhroncok, python-sig, thrnciar |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-04-23 10:43:47 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1890881 |
Description
Tomáš Hrnčiar
2021-04-22 06:26:28 UTC
Note that the INTERNALERROR is tracked in https://github.com/pytest-dev/pytest/pull/8227 but it IMHO only happens for tests that fail. Error with patched pytest: =================================== FAILURES =================================== _ test_ambiguities[foo: bar | baz\nbar: NAME\nbaz: NAME\n-foo is ambiguous.*given a PythonTokenTypes\\.NAME.*bar or baz] _ grammar = 'foo: bar | baz\nbar: NAME\nbaz: NAME\n' error_match = 'foo is ambiguous.*given a PythonTokenTypes\\.NAME.*bar or baz' @pytest.mark.parametrize( 'grammar, error_match', [ ['foo: bar | baz\nbar: NAME\nbaz: NAME\n', r"foo is ambiguous.*given a PythonTokenTypes\.NAME.*bar or baz"], ['''foo: bar | baz\nbar: 'x'\nbaz: "x"\n''', r"foo is ambiguous.*given a ReservedString\(x\).*bar or baz"], ['''foo: bar | 'x'\nbar: 'x'\n''', r"foo is ambiguous.*given a ReservedString\(x\).*bar or foo"], # An ambiguity with the second (not the first) child of a production ['outer: "a" [inner] "b" "c"\ninner: "b" "c" [inner]\n', r"outer is ambiguous.*given a ReservedString\(b\).*inner or outer"], # An ambiguity hidden by a level of indirection (middle) ['outer: "a" [middle] "b" "c"\nmiddle: inner\ninner: "b" "c" [inner]\n', r"outer is ambiguous.*given a ReservedString\(b\).*middle or outer"], ] ) def test_ambiguities(grammar, error_match): with pytest.raises(ValueError, match=error_match): > generate_grammar(grammar, tokenize.PythonTokenTypes) test/test_pgen2.py:357: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ bnf_grammar = 'foo: bar | baz\nbar: NAME\nbaz: NAME\n' token_namespace = <enum 'PythonTokenTypes'> def generate_grammar(bnf_grammar: str, token_namespace) -> Grammar: """ ``bnf_text`` is a grammar in extended BNF (using * for repetition, + for at-least-once repetition, [] for optional parts, | for alternatives and () for grouping). It's not EBNF according to ISO/IEC 14977. It's a dialect Python uses in its own parser. """ rule_to_dfas = {} start_nonterminal = None for nfa_a, nfa_z in GrammarParser(bnf_grammar).parse(): # _dump_nfa(nfa_a, nfa_z) dfas = _make_dfas(nfa_a, nfa_z) # _dump_dfas(dfas) # oldlen = len(dfas) _simplify_dfas(dfas) # newlen = len(dfas) rule_to_dfas[nfa_a.from_rule] = dfas # print(nfa_a.from_rule, oldlen, newlen) if start_nonterminal is None: start_nonterminal = nfa_a.from_rule reserved_strings: Mapping[str, ReservedString] = {} for nonterminal, dfas in rule_to_dfas.items(): for dfa_state in dfas: for terminal_or_nonterminal, next_dfa in dfa_state.arcs.items(): if terminal_or_nonterminal in rule_to_dfas: dfa_state.nonterminal_arcs[terminal_or_nonterminal] = next_dfa else: transition = _make_transition( token_namespace, reserved_strings, terminal_or_nonterminal ) dfa_state.transitions[transition] = DFAPlan(next_dfa) > _calculate_tree_traversal(rule_to_dfas) parso/pgen2/generator.py:278: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ nonterminal_to_dfas = {'bar': [<DFAState: bar is_final=False>, <DFAState: bar is_final=True>], 'baz': [<DFAState: baz is_final=False>, <DFAState: baz is_final=True>], 'foo': [<DFAState: foo is_final=False>, <DFAState: foo is_final=True>]} def _calculate_tree_traversal(nonterminal_to_dfas): """ By this point we know how dfas can move around within a stack node, but we don't know how we can add a new stack node (nonterminal transitions). """ # Map from grammar rule (nonterminal) name to a set of tokens. first_plans = {} nonterminals = list(nonterminal_to_dfas.keys()) nonterminals.sort() for nonterminal in nonterminals: if nonterminal not in first_plans: _calculate_first_plans(nonterminal_to_dfas, first_plans, nonterminal) # Now that we have calculated the first terminals, we are sure that # there is no left recursion. for dfas in nonterminal_to_dfas.values(): for dfa_state in dfas: transitions = dfa_state.transitions for nonterminal, next_dfa in dfa_state.nonterminal_arcs.items(): for transition, pushes in first_plans[nonterminal].items(): if transition in transitions: prev_plan = transitions[transition] # Make sure these are sorted so that error messages are # at least deterministic choices = sorted([ ( prev_plan.dfa_pushes[0].from_rule if prev_plan.dfa_pushes else prev_plan.next_dfa.from_rule ), ( pushes[0].from_rule if pushes else next_dfa.from_rule ), ]) > raise ValueError( "Rule %s is ambiguous; given a %s token, we " "can't determine if we should evaluate %s or %s." % ( ( dfa_state.from_rule, transition, ) + tuple(choices) ) ) E ValueError: Rule foo is ambiguous; given a NAME token, we can't determine if we should evaluate bar or baz. parso/pgen2/generator.py:339: ValueError During handling of the above exception, another exception occurred: grammar = 'foo: bar | baz\nbar: NAME\nbaz: NAME\n' error_match = 'foo is ambiguous.*given a PythonTokenTypes\\.NAME.*bar or baz' > ??? E AssertionError: Regex pattern 'foo is ambiguous.*given a PythonTokenTypes\\.NAME.*bar or baz' does not match "Rule foo is ambiguous; given a NAME token, we can't determine if we should evaluate bar or baz.". test/test_pgen2.py:-1: AssertionError =========================== short test summary info ============================ FAILED test/test_pgen2.py::test_ambiguities[foo: bar | baz\nbar: NAME\nbaz: NAME\n-foo is ambiguous.*given a PythonTokenTypes\\.NAME.*bar or baz] There were some changes in repr() in latest alpha of Python 3.10. https://docs.python.org/3.10/whatsnew/changelog.html#python-3-10-0-alpha-7 bpo-40066: Enum: adjust repr() to show only enum and member name (not value, nor angle brackets) and str() to show only member name. Update and improve documentation to match. bpo-40066: Enum’s repr() and str() have changed: repr() is now EnumClass.MemberName and str() is MemberName. Additionally, stdlib Enum’s whose contents are available as module attributes, such as RegexFlag.IGNORECASE, have their repr() as module.name, e.g. re.IGNORECASE. https://bugs.python.org/issue40066 Patched in copr with https://github.com/davidhalter/parso/pull/186 Will give upstream some time for feedback before backporting. |