[Feature] Support submodules

[sarpedondev]

Describe the bug
Custom python modules (folders) don't get compiled onto the hub. Everything works fine when I put every python file that gets imported by any other python file into the project root. However, when I try to import a file that's within a directory, this error occurs:

[tom@neon:~/Coding/2425-RobotCode]$ pybricksdev run ble main.py 
Searching for any hub with Pybricks service...
Traceback (most recent call last):
  File "/home/tom/Coding/2425-RobotCode/venv/bin/pybricksdev", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/cli/__init__.py", line 383, in main
    asyncio.run(subparsers.choices[args.tool].tool.run(args))
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/cli/__init__.py", line 207, in run
    await hub.run(script_path, args.wait)
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/connections/pybricks.py", line 572, in run
    mpy = await compile_multi_file(py_path, abi)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/compile.py", line 119, in compile_multi_file
    finder.run_script(path)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 411, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 367, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 165, in import_hook
    q, tail = self.find_head_package(parent, name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 221, in find_head_package
    q = self.import_module(head, qname, parent)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 308, in import_module
    fp, pathname, stuff = self.find_module(partname,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 489, in find_module
    return _find_module(name, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 69, in _find_module
    if spec.loader.is_package(name):
       ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'is_package'

When I add an init.py file to the directory, it successfully uploads the file to the hub but fails importing it:

[tom@neon:~/Coding/2425-RobotCode]$ pybricksdev run ble main.py 
Searching for any hub with Pybricks service...
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 316/316 [00:00<00:00, 2.14kB/s]
Traceback (most recent call last):
  File "main.py", line 1, in <module>
ImportError: can't import name actual_module

However, when I remove the actual_module file and put the functions into the init.py file and remove the actual_module part from the import statement, everything works fine (except for my IDE not picking up the function)

To reproduce
Steps to reproduce the behavior:

1. Create a pybricks project
2. Create main file
3. Create module
4. Import module in main

Expected behavior
It compiles every single file successfully, and it runs without any issues.

Screenshots

I hope my explanation of this issue was sufficient, and thanks in advance!

[dlech]

Can you share you main.py?

[sarpedondev]

from my_module import actual_module

actual_module.my_func()

[laurensvalk]

Everything works fine when I put every python file that gets imported by any other python file into the project root.

This is currently the only officially supported way.

Custom python modules (folders) don't get compiled onto the hub.

If this doesn't require firmware changes, I think we could consider supporting this in Pybricksdev.

I don't know if we can do full packages, but submodules might work.

[sarpedondev]

A possible workaround might be parsing all the python files and moving them to the project root and then obviously changing the imports in a script before running pybricksdev run, however it would be quite nice if pybricksdev would support this natively. Looking forward to what you figure out!

[laurensvalk]

Support for this might have been removed in: pybricks/pybricksdev@ff32826

[laurensvalk]

Just hacking around with Pybricksdev a bit to put some of pybricks/pybricksdev@ff32826 back in, it appears that the firmware still works for the following use case (see folder structure on the left).

pybricks/pybricksdev@ff32826 was probably done for good reason (as it does make the code cleaner), but maybe there's a way to achieve the same with the ModuleFinder introduced there.

[sarpedondev]

Seems like this would just work without any modifications required if this bug in python got fixed. python/cpython#84530 It's probably going to take forever until that is fixed though. Would you be able to provide me with a patch with what exactly you changed in pybricksdev to make it work?

[dlech]

Recapping, there are two "bugs" here, one in the CPython standard library and one is a firmware limitation. And both have workarounds.

1. Not being able to find imports when there is an implicit namespace package

   • This is modulefinder chokes on numpy - dereferencing None in spec.loader python/cpython#84530
   • Workaround is to include __init__.py so there isn't an implicit namespace package

2. The following code fails in Pybricks MicroPython

   from my_module import actual_module
   
   actual_module.my_func()

   with ImportError: can't import name actual_module

   • The module is actually found and compiled into the .mpy file like it is supposed to be.
   • This is probably a limitation of how we implemented imports in the firmware.
   • Workaround is to write the code like this instead:

   from my_module.actual_module import my_func
   
   my_func()