Steps to reproduce:

Create the following three files (or download the attached zip file, which contains these files):

main.py

    import foo
    from foo import a
    from foo import b

    print(foo.b.my_function())


foo/a.py

    import importlib.util
    import sys

    # implementation copied from https://github.com/python/cpython/blob/master/Doc/library/importlib.rst#implementing-lazy-imports
    def lazy_import(name):
        spec = importlib.util.find_spec(name)
        loader = importlib.util.LazyLoader(spec.loader)
        spec.loader = loader
        module = importlib.util.module_from_spec(spec)
        sys.modules[name] = module
        loader.exec_module(module)
        return module

    b = lazy_import("foo.b")


foo/b.py

    def my_function():
        return "my_function"

and then run main.py



Expected results

my_function should be printed to the terminal


Actual results

The following traceback is printed to the terminal

    Traceback (most recent call last):
      File "F:\Documents\lazy_import\main.py", line 6, in <module>
        print(foo.b.my_function())
    AttributeError: module 'foo' has no attribute 'b'

If you comment out "from foo import a" from main.py, then the traceback doesn't occur and my_function gets printed.  Alternatively, if you move "from foo import a" after "from foo import b", then the traceback doesn't occur and my_function gets printed.  Adding "foo.b = b" before "print(foo.b.my_function())" will also fix the traceback.


A colleague of mine originally ran into this bug when writing unit tests for lazily imported code, since mock.patch("foo.b.my_function") triggers the same AttributeError.  I've reproduced this on Windows using both Python 3.8.3 and Python 3.9.0, and my colleague was using Python 3.8.3 on Mac.

Just checking: is this not because the lazy import should be in `__init__.py`? (the code provided works fine with `a.b.my_function` on my system)

An __init__.py shouldn't be necessary.  If I comment out the 'b = lazy_import("foo.b")' line in a.py (i.e. disable the lazy import), then the print statement works correctly as written without any other changes.

Also, I double checked with the colleague who originally ran into this issue, and it turns out he encountered the bug on Linux, not on Mac (still Python 3.8.3).

My colleague just tested this on Mac and confirms that the bug also occurs there using Python 3.8.3.

In short, the module isn't being added to the package's namespace because we are directly modifying sys.modules (hence why the behaviour would be the same if we imported using `import foo.b` as `from foo import b`).

I personally prefer to use the metapath instead of modifying sys.modules but I agree that the given example should work when the lazy import is not in `__init__.py`. The other solution is to modify the `LazyLoader` class to explicitly add the lazy module to the package's namespace (opinions?).

The way import works,

Brett, what do you mean by "the way import works"?  Is the difference between using LazyLoader and using a normal import intentional?

One possible solution here would be to update the documentation at https://github.com/python/cpython/blob/master/Doc/library/importlib.rst#implementing-lazy-imports to either note the limitation or to modify the lazy_import function so that it adds the module to the package's namespace.  That's basically the workaround that we've been using.

Sorry Brett to readd you to the nosy for this, but we only got half a sentence in msg380718 (which is surely not what you intended?).

While I agree with you that this is not a bug, I do feel at least a note in the docs would be helpful to explain the implications of *adding* to sys.modules (the existing docs only mention replacing the dictionary or removing items from it).

Again, sorry to readd you to the nosy but Kevin's msg380719 was specifically for you.

You can ignore the half sentence. I was contemplating closing this issue when I decided to leave it open in case someone wanted to propose something and another core dev wanted to take it on. But everything is working as I expect it to and you may want to do your own implementation on PyPI if you want fancier as it's already a delicate thing as it is and so adding complexity for this specific case is a tough sell.