typing.get_type_hints wrong namespace for forward-declaration of inner class

[netbnd]

BPO	37835
Nosy	@ilevkivskyi
Files	typing_check.py: A simple file to reproduce the error with inner classes. get_type_hints_for_inner_classes.zip metaclass_workaround.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-08-13.09:36:43.877>
labels = ['3.7', 'type-bug', 'library']
title = 'typing.get_type_hints wrong namespace for forward-declaration of inner class'
updated_at = <Date 2019-09-09.09:59:14.831>
user = 'https://bugs.python.org/netbnd'

bugs.python.org fields:

activity = <Date 2019-09-09.09:59:14.831>
actor = 'levkivskyi'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-08-13.09:36:43.877>
creator = 'netbnd'
dependencies = []
files = ['48539', '48552', '48562']
hgrepos = []
issue_num = 37835
keywords = []
message_count = 8.0
messages = ['349535', '349958', '349986', '349988', '350468', '350982', '351282', '351398']
nosy_count = 2.0
nosy_names = ['levkivskyi', 'netbnd']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37835'
versions = ['Python 3.6', 'Python 3.7']

[netbnd]

When evaluating the type-hints of an inner-class (a class defined inside a function scope), the forward declaration does not work correctly. In this case the typing.get_type_hints does not get the right namespace, thus the class-string specified is not found.

When using the same syntax for a normal class (defined in global space) it works, or when using another class instead a forward declaration, also everything works.

As a workaround one could pass the local namespace (locals()) from the function where the class has been defined in to typing.get_type_hints. However in normal situations the typing.get_type_hints call is deep in the call hierarchy to do some runtime type-checks and at this point only the reference to the function-object is existing and no-one is aware of the fact, that this is just a method defined in a inner class.

From the outside perspective one would expect, that typing.get_type_hints reacts the same, independent of the type of class.

[ilevkivskyi]

Thanks for reporting!

I spent some time thinking about this and I can't find any reasonable way of doing this, sorry. Anyway, let's keep this open, maybe someone will come up with a proposal.

[netbnd]

Thanks for your response. I was also thinking much about it and was not able to find a nice idea how to get this working.

The problem is, that we loose the local-namespace information as soon as we leave the context of the function, where the class was defined in. This information is not stored in the function object of the method, we want to get the type hints from. The only open question, I have in this context is:

Why can python resolve the reference to class A (also a inner class, but no forward declaration)? Is there any chance to use the same mechanism also for forward declared references?

Beside from this question, I will give you my thoughts, how I think this issue could be addressed:

• What we need is the local namespace information from the function, where the inner class was defined in. This is not stored from python in the function object of this class (but __global__ is stored).

• Maybe any upcoming python version could store this information in __local__ ? So maybe we could clone this ticket to the python core in order to address this?

• As long as python is not storing this information, a workaround could be used: We could define a function decorator, which adds the attribute __local__ to a function object. I think about this syntax:

class InnerClass():
    @store_namespace(locals())
    def method() -> 'InnerClass':
        ...

• The get_type_hints function is then checking for the __local__ attribute and if it exits it is passed to the function, which resolves the forward declaration.

This workaround is not beautiful, since it requires manual work for those methods (adding the function decorator to those methods). Furthermore - without knowing the internals of the get_type_hints function - this workaround is not straight forward and easy to understand. So it needs to be carefully documented and even then I expect confusing questions on StackOverflow or other communities.

However, it is a quite rare case that someone really needs to use inner classes. Normally one could simply put the class in the global namespace. But when this happens and there is really a need for it, than this workaround would make it possible. Furthermore, checking a __local__ attribute of the function object would be a nice preparation for a feature request to the python core, which should store the reference to the local namespace for every function object, without using any decorators.

What do you think?

[netbnd]

I tried my idea with the small example code above. However it does not work like expected:
(see zipped example project, attached to this comment)

At the moment where the function decorator is applied to the method of the inner class, the local namespace ("locals()") does not contain any inner class. Even not another inner class, define before the corresponding class.

So the only way to get it working is to add the __locals__ attribute manually after defining the class, which is even more ugly than my suggested workaround with the function decorator :-(

Any further ideas about this?

[netbnd]

I think I found a better workaround (by accident).

I was debugging another issue with get_type_hints for the case that this function is used inside the __new__ method of a metaclass and with methods of the class to define. Here the same issue happens: Forward declared type names are not inside the namespace of the function.

However, inside the __new__ method of the metaclass, you already know the name of the class you want to define and you know the class-object to define. With this information it is very easy to add the missing reference to the __globals__ list of all methods found in the namespace.

So the workaround is to use a metaclass, which adds the class-name and class-object to the global namespace of the methods of this class. It works also in case of classes, which are inherited, from a class, which uses this metaclass.

This leads also to a non-workaround solution: Why is the own class not always inside the __globals__ list of the methods? Is there a reason? Or is this just a missing feature of the python-core?

What are you thinking about this solution? Do you see any issues or corner-cases with that?

Working code is attached to this ticket (metaclass_workaround.py)

[ilevkivskyi]

(Sorry for typos, fixed now.)

Maybe any upcoming python version could store this information in __local__ ? So maybe we could clone this ticket to the python core in order to address this?

I would say it is a too big change, and it is unlikely to happen only for the reason like this issue.

Why is the own class not always inside the __globals__ list of the methods? Is there a reason? Or is this just a missing feature of the python-core?

Because this is a reference to the actual module global namespace. If the class is defined inside another function, it is not in the module namespace.

You can probably use the metaclass workaround, but note that it will actually modify the module globals (since __globals__ is not a copy).

[netbnd]

Thanks for your explanations.

It sounds, that there is no way to fix this issue officially. Only the meta-class workaround is left, which is - as you highlight - really just a workaround.

I anyway always wonder, why functions, which are methods, do not hold a reference to the class, which they belong to. It could help here and second, this would be a good way to differentiate if a function object is a method or not.

[ilevkivskyi]

I anyway always wonder, why functions, which are methods, do not hold a reference to the class, which they belong to.

This may indeed be a useful feature on its own, but it will also require a much more wider discussion.