IDLE fails to detect corresponding opening parenthesis

[AlexeyBurdin]

BPO	41388
Nosy	@terryjreedy, @taleinat, @lrjball

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://proxy-ga.blitzz.co/proxy/123456/github.com/terryjreedy'
closed_at = None
created_at = <Date 2020-07-24.19:10:41.733>
labels = ['expert-IDLE', 'type-bug', '3.8', '3.9', '3.10']
title = 'IDLE fails to detect corresponding opening parenthesis'
updated_at = <Date 2020-07-26.23:35:02.207>
user = 'https://proxy-ga.blitzz.co/proxy/123456/bugs.python.org/AlexeyBurdin'

bugs.python.org fields:

activity = <Date 2020-07-26.23:35:02.207>
actor = 'Lewis Ball'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2020-07-24.19:10:41.733>
creator = 'Alexey Burdin'
dependencies = []
files = []
hgrepos = []
issue_num = 41388
keywords = []
message_count = 9.0
messages = ['374205', '374218', '374224', '374225', '374226', '374274', '374298', '374301', '374350']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'taleinat', 'Alexey Burdin', 'Lewis Ball']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://proxy-ga.blitzz.co/proxy/123456/bugs.python.org/issue41388'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[AlexeyBurdin]

answers_field_order=sorted(
    set(j for i in data['items'] for j in i),
    key=cmp_to_key(lambda x,y:(
        -1 if (x,y) in answer_order 
        else (0 if x==y else 1)))
        )

when the cursor is placed in line 5 col 31 (between ) and )) ) hitting Ctrl-0 (Show surrounding parens) produces an error sound, though there's no error, the script works fine.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
$ uname -a
Linux odd-one 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ python3.8
Python 3.8.0 (default, Oct 28 2019, 16:14:01) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec('idlelib.pyshell')
ModuleSpec(name='idlelib.pyshell', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f1ea8ee14c0>, origin='/usr/lib/python3.8/idlelib/pyshell.py')
>>> exit()
$ dpkg-query -S /usr/lib/python3.8/idlelib/pyshell.py
idle-python3.8: /usr/lib/python3.8/idlelib/pyshell.py
$ dpkg -l | grep idle-python3\.8
ii  idle-python3.8                             3.8.0-3~18.04                                    all          IDE for Python (v3.8) using Tkinter

[lrjball]

A minimal example of the same issue seems to be:

(1 if x
 else 0)

In this case the parentheses matching also fails. It only ever seems to fail when else is the first word on a newline inside brackets, and adding the line break elsewhere in the above statement will not cause the matching to fail.

I imagine it stops looking for the matching parentheses when it gets to the else statement, as an else when part of an if/else cannot be inside parentheses. Obviously the exception to this is the ternary operator.

Bizarrely, it only seems to fail the first time it encounters this issue. Any subsequent code with the same pattern seems to be matched fine. So in the following example, the second brackets get matched without an issue:

(1 if x
 else 0)

(1 if x
 else 0)

Hopefully someone more familiar with the source code can shed some light on this!

[terryjreedy]

(While I wrote the following, Lewis Ball posted some of the same conclusions.)

0. I verified the failure with 3.8.5 and current master.

1. I reduced the failing example to a minimum of 7 chars on 2 lines.
   (
   else)

1a. Deleting '\n' or any letter of 'else' fixes the problem.
1b. Adding 'else' before '(' fixes the problem.
1a is true for the initial 'else' on line 4 of the original example.
1b is true for 'else' on a new line before the first.

2. I tried multiple variations
   2a. Matching also fails when typing ')'.
   2b. Replacing '()' with '[]' or '{}' makes no difference.

2c. Replacing 'else' with 'elif', 'while', 'def', or 'class' makes no difference. This include the addition for 1b above. Replacing 'else' with 'if', 'for', or 'with' fixes the bug.

Bingo!!! Matching uses idlelib.hyperparser, which uses idlelib.pyparse, which has regex _synchre, which finds "what looks like the start of a popular statement". It matches 'elif', etcetera, but not 'if', etcetera. I noticed previously that these 'popular' line beginners were missing and have planned to add them.* But we need to make matched words not disable fence matching.

• Keyword 'with' was likely added to python after this re was last revised. Keywords 'if' and 'for' might have been removed when comprehensions were added. If so, 'else' should have been removed when conditional expressions were added and 'yield' when yield expressions were added. These latter two are the only keywords matched by _synchre that can properly appear in the middle of a statement but at the beginning of a continuation line.

2d. In the original and minimal examples, moving the offending 'else' to the end of the previous line fixes the problem. So I believe that 'else' at the beginning of the line is seen marking the beginning of the statememt, and matching does not look back further. If 'else' and 'yield' were left in _synchre and 'if' and 'for' were added back, the matcher would have to determine whether the line is a continuation or not. (And I believe that not having 'if' and 'for' degrades the use of pyparse at least for other uses.)

2e. The minimal failing example works in the shell. Why are _synchre keywords not poisonous here? Is it just because the current statement is known to begin just after the '>>> ' prompt, so that else beginning the second line is somehow not seen as beginning a new statement? We need to check if and how fence matching in shell is different other than the presence of the marker where code input begins.

Alexey: in (#65955 (comment)) I claimed that there is no issue with typing closers or ^0 on the same line just before the closer. Thank you for reporting this counter-example. I imagine that you are not the first IDLE user to write a conditional expression on 2 or more lines with else at the beginning of a line. Such expressions always need a closer.

[terryjreedy]

Without rereading hyperparser and pyparse, I don't understand why a previous _synchre match prevents subsequent failures. It does suggest that when matching parentheses, a fix would be to temporarily prepend 'else\n' to the text, or otherwise simulating the effect of that.

But the previous 'else' might only work before the 'current' statement. And we should determine whether any other uses of these modules have related bugs. Also, removing 'else' from the _synchre regex did not fix the issue, so there is something else I am missing.