Add a Python menuconfig implementation

[ulfalizer]

This series adds a Kconfiglib-based menuconfig implementation, built
with the standard Python curses module. A new pymenuconfig target is
added to run it.

The C tools are kept for now. Removing them separately makes it possible
to test pymenuconfig alongside the C tools, and keeps changes small and
focused.

A feature is planned for later that shows all symbols -- including those
that aren't currently visible -- along with a search and "jump to"
feature. Loading of arbitrary .config files will be supported later as
well (as opposed to always loading .config/KCONFIG_CONFIG). Those
features are all connected implementation-wise.

For Windows, the wheels at
https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses provide the curses
implementation. They use the standard Python curses module
(_cursesmodule.c), linked against PDCurses.

Running python -VV gives the Python version and bitness, to know which
wheel to install. User documentation will be added once the C tools are
removed and the pymenuconfig target is moved over to menuconfig.

The CMake parts are originally by @SebastianBoe.

Description, taken from the menuconfig.py docstring:

Overview
========

A curses-based menuconfig implementation. The interface should feel
familiar to people used to mconf ('make menuconfig').

Supports the same keys as mconf, and also supports a set of
keybindings inspired by Vi:

  J/K     : Down/Up
  L       : Enter menu/Toggle item
  H       : Leave menu
  Ctrl-D/U: Page Down/Page Down
  G/End   : Jump to end of list
  g/Home  : Jump to beginning of list

The mconf feature where pressing a key jumps to a menu entry with
that character in it in the current menu isn't supported. A search
feature with a "jump to" function for jumping directly to a
particular symbol regardless of where it is defined will be added
later instead.

Space and Enter are "smart" and try to do what you'd expect for the
given menu entry.

Running
=======

menuconfig.py can be run either as a standalone executable or by
calling the menu.menuconfig() function with an existing Kconfig
instance. The second option is a bit inflexible in that it will
still load and save .config, etc.

When run in standalone mode, the top-level Kconfig file to load can
be passed as a command-line argument. With no argument, it defaults
to "Kconfig".

The KCONFIG_CONFIG environment variable specifies the .config file
to load (if it exists) and save. If KCONFIG_CONFIG is unset,
".config" is used.

$srctree is supported through Kconfiglib.

Other features
==============

  - Seamless terminal resizing

  - No dependencies on *nix, as the 'curses' module is in the Python
    standard library

  - Unicode text entry

  - Improved information screen compared to mconf:

      * Expressions are split up by their top-level &&/|| operands
        to improve readability

      * Undefined symbols in expressions are pointed out

      * Menus and comments have information displays

      * Kconfig definitions are printed

Limitations
===========

  - Python 3 only

    This is mostly due to Python 2 not having curses.get_wch(),
    which is needed for Unicode support.

  - Doesn't work out of the box on Windows

    Has been tested to work with the wheels provided at
    https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses though.

Building the wheel

(Compiled wheels are already available, but for documentation purposes.)

1. Unpack curses‑2.2‑source.zip from https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses somewhere.

2. Clone PDCurses (https://github.com/wmcbrine/PDCurses) into the same directory.

3. Build PDCurses by running this command in the wincon\ directory (the Windows console PDCurses backend):

   nmake -f Makefile.vc WIDE=y UTF8=y
   

   WIDE=y is needed for Unicode support, while UTF8=y forces UTF-8 as the encoding (not essential, but maybe it can avoid obscure Unicode issues).

4. Build the wheel by running this command in the top-level directory:

   setup.py bdist_wheel build_ext --include-dirs=pdcurses-3.6 --library-dirs=pdcurses-3.6\wincon
   

[ulfalizer]

CC @carlescufi

[SebastianBoe]

"Running on Windows requires first installing an appropriate wheel from
https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses. Running python -VV
gives the Python version and bitness, to know which wheel to install.
That will be documented later."

The doc's should be updated in this PR.

[ulfalizer]

See #5847 for some development screenshots.

[SebastianBoe]

What about 'menu' ?

[ulfalizer]

My plan was to rename it to 'menuconfig' later, once we get rid of the C tools. That's what people are used to.

[SebastianBoe]

Hmm, this would be incosistent with the other Kconfig-frontends I guess ...

[carlescufi]

I agree with renaming to menuconfig later

[carlescufi]

Let's not overcomplicate things. Let's replace mconf with pymenuconfig in this PR, and keep the "menuconfig" name

[ulfalizer]

"Running on Windows requires first installing an appropriate wheel from
https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses. Running python -VV
gives the Python version and bitness, to know which wheel to install.
That will be documented later."

The doc's should be updated in this PR.

I thought I could update them once we get rid of the C tools and all the naming and stuff is set. I could put together some docs that describe this "staging" version as well though if you want.

[ulfalizer]

CC @carlescufi

[SebastianBoe]

I thought I could update them once we get rid of the C tools and all the naming and stuff is set. I could put together some docs that describe this "staging" version as well though if you want.

Nope. We need doc's now to know how bad it will be for Windows users to install and for Windows users to be able to test this. Also, it enforces that we don't have features with undocumented dependencies in master.

[SebastianBoe]

This requires an update to kconfig-usage.cmake.

Alternatively, we could try to deprecate kconfig-usage.cmake entirely. It was ported from Kbuild, but I'm not sure if it is found to be useful.

[codecov-io]

Codecov Report

Merging #7174 into master will increase coverage by 3.51%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #7174      +/-   ##
==========================================
+ Coverage   55.02%   58.54%   +3.51%     
==========================================
  Files         477      464      -13     
  Lines       51749    47400    -4349     
  Branches     9951     8783    -1168     
==========================================
- Hits        28477    27748     -729     
+ Misses      19306    15821    -3485     
+ Partials     3966     3831     -135

Impacted Files	Coverage Δ
subsys/net/lib/http/http.c	0% <0%> (-27.36%)	⬇️
subsys/net/lib/app/server.c	25.25% <0%> (-27.28%)	⬇️
subsys/net/lib/app/net_app.c	20.47% <0%> (-22.69%)	⬇️
subsys/net/ip/net_stats.h	57.74% <0%> (-8.09%)	⬇️
subsys/net/lib/app/client.c	48.49% <0%> (-7.3%)	⬇️
kernel/timer.c	90.12% <0%> (-3.71%)	⬇️
subsys/net/lib/http/http_parser.c	26.74% <0%> (-2.98%)	⬇️
subsys/net/ip/tcp.c	52.42% <0%> (-2.78%)	⬇️
subsys/net/lib/app/init.c	66.66% <0%> (-1.29%)	⬇️
subsys/net/ip/net_core.c	70.37% <0%> (-1.17%)	⬇️
... and 52 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb6f203...ea82b6f. Read the comment docs.

[carlescufi]

@ulfalizer I agree that the docs should be part of this PR. In fact maybe we should go ahead with replacing mconf with pymenuconfig.py directly in here so we get it all in one go.

[ulfalizer]

@SebastianBoe @carlescufi
Alright, will do. I still think the oldconfig implementation could be added separately, just to break it up a bit. After that, we could get rid of the C tools.

I added instructions for installing the curses wheel on Windows. More documentation is needed as well, but it's a start.

[carlescufi]

remove the prompt c:\...\zephyr>

[ulfalizer]

IMO, that makes it less clear that what follows it is the output, but alright.

[carlescufi]

yes, but makes it harder to copy-paste and we have this convention all over the documentation

[ulfalizer]

Hmm... yeah, think I would want to split it up in that case.

I think I prefer for it to be immediately obvious rather than perfectly consistent. What do you think?

[carlescufi]

Then do it as follows:

Type the following command:

python -VV

and check for the output, which should be something like:;

Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 16:07:46) [MSC v.1900 32 bit (Intel)]

[carlescufi]

replace "repository" with "web page"

[carlescufi]

Let's not overcomplicate things. Let's replace mconf with pymenuconfig in this PR, and keep the "menuconfig" name

[ulfalizer]

@carlescufi
Updated it to use the standard menuconfig target, and got rid of some stuff related to mconf. Updated the wheel installation doc too.

Thinking of removing the C tools completely later, once oldconfig is in too (there's already a version of it for Kconfiglib).

[ulfalizer]

@SebastianBoe
What about trimming kconfig-usage.cmake down to the essentials, e.g. menuconfig and oldconfig? defconfig files can be generated from within the menuconfig interface now.

Do you know which of the other ones people use in practice?

[SebastianBoe]

"What about trimming kconfig-usage.cmake down to the essentials, e.g. menuconfig and oldconfig? defconfig files can be generated from within the menuconfig interface now."

Sounds good.

"Do you know which of the other ones people use in practice?"

I have never heard of anyone ever using anything other than xconfig and menuconfig. But it is of course impossible to know.

[ulfalizer]

Where do these values come from? A comment would be nice, that also gives an overview of the issue.

This is probably an RFC, but I got curious anyway.

[carlescufi]

on the Windows Command prompt:

color /?

[ulfalizer]

Something like if platform.system() == "Windows" might be better.

Even then it's probably mostly up to the terminal, so gets a bit iffy.

[carlescufi]

This is taken from another toolkit, feel free to change

[ulfalizer]

@SebastianBoe
Alright, I'll trim it down a bit then.

Do you know of any people that might be "heavy" conf/mconf users by the way? Should probably add them to the review in that case.

[carlescufi]

@dbkinder the documentation has been removed from this PR

[carlescufi]

@SebastianBoe mind taking another look? See my comment about the documentation, which we don't want to write twice. Since we want to include the wheels in PyPi later, we want to have this "undocumented" first in order to move forward with this.

[pfalcon]

we will provide the wheels in pypi, maintained by us

I didn't have chance to follow this thru, and it seems there're enough people on this, so I'm sure it was already said and/or considered, so please treat the below just as "+1's" in this case:

1. I guess it would be polite to ask the maintainer(s) of https://www.lfd.uci.edu/%7Egohlke/pythonlibs/#curses why they don't publish this package on PyPI, and express intention to do that if they confirm they aren't interested in that.

2. I'd suggest to use reasonable name on PyPI. Like, avoid generic "curses", and instead use more specific like "curses-windows".

3. I hope there will be reasonable testing that trying to install this package on Linux and other OSes won't lead to too much confusion of users (e.g., there will be clear message that this package is Windows-specific). (How pip should work is that it should try to download a .whl for the current platform, if it's available on PyPI, and if not, should download "source package" and try to build that. Building involves running setup.py, which can do arbitrary checks).

Again, sorry if all of the above is obvious and was already considered. Just my 2 cents. Thanks.

Documentation changes were removed, so nothing to review :(

[ulfalizer]

@dbkinder
Will get back to it later. Check out the "merger" of our versions that I made in the comment above. :)

[ulfalizer]

I didn't have chance to follow this thru, and it seems there're enough people on this, so I'm sure it was already said and/or considered, so please treat the below just as "+1's" in this case:

I guess it would be polite to ask the maintainer(s) of https://www.lfd.uci.edu/%7Egohlke/pythonlibs/#curses why they don't publish this package on PyPI, and express intention to do that if they confirm they aren't interested in that.

Might be to decrease the maintenance burden, since there are so many wheels on that page.

@cgohlke is the maintainer by the way.

I'd suggest to use reasonable name on PyPI. Like, avoid generic "curses", and instead use more specific like "curses-windows".

I hope there will be reasonable testing that trying to install this package on Linux and other OSes won't lead to too much confusion of users (e.g., there will be clear message that this package is Windows-specific). (How pip should work is that it should try to download a .whl for the current platform, if it's available on PyPI, and if not, should download "source package" and try to build that. Building involves running setup.py, which can do arbitrary checks).

Sounds like a good idea. Will need to check with @cgohlke whether he's okay with mirrors and custom-built versions of the wheels being hosted elsewhere too.

[mbolivar]

I hope there will be reasonable testing that trying to install this package on Linux and other OSes won't lead to too much confusion of users (e.g., there will be clear message that this package is Windows-specific).
Sounds like a good idea. Will need to check with @cgohlke whether he's okay with mirrors and custom-built versions of the wheels being hosted elsewhere too.

AFAIK, the Python dependency installation procedure on all platforms right now (for mandatory dependencies) is roughly:

1. Install Python 3 with pip (via system-dependent means)
2. pip3 install --user -r scripts/requirements.txt

It would be nice to be able to avoid having to modify that with "... and if you're on windows here's another set of requirements", i.e. try to keep just one requirements.txt.

If that proves impossible, perhaps a scripts/win-requirements.txt is in order to initially contain this curses-windows dependency?

[ulfalizer]

I hope there will be reasonable testing that trying to install this package on Linux and other OSes won't lead to too much confusion of users (e.g., there will be clear message that this package is Windows-specific).
Sounds like a good idea. Will need to check with @cgohlke whether he's okay with mirrors and custom-built versions of the wheels being hosted elsewhere too.

AFAIK, the Python dependency installation procedure on all platforms right now (for mandatory dependencies) is roughly:

Install Python 3 with pip (via system-dependent means)
pip3 install --user -r scripts/requirements.txt
It would be nice to be able to avoid having to modify that with "... and if you're on windows here's another set of requirements", i.e. try to keep just one requirements.txt.

If that proves impossible, perhaps a scripts/win-requirements.txt is in order to initially contain this curses-windows dependency?

Yeah, would be nicer if it was automatic, if we can get that reliable somehow. Manual is better than broken automatic at least.

Switching over to some other toolkit would be way less work than doing the menuconfig from scratch was, so if we discover a bunch of pain later, it won't be a big deal to port it. In the near term, there's no panic at least. Could pull it in and see what happens.

[mbolivar]

Upon reflection, I really don't feel comfortable merging this without a clear and convenient installation solution being available for Windows users. Preferably one that adds no extra steps.

As discussed at the TSC meeting, can we have (zephyrproject-rtos owned on GitHub) infrastructure ready for installing the Windows wheel via pip before merging?

Right now, the curses solution is more difficult for Windows users to install than solutions based on other libraries (blessings, python-prompt-toolkit, etc.). I know most of us aren't on Windows (and I'm not either), but this is a basic workflow item, and we've prioritized making Windows a first class citizen, so it just doesn't feel right to merge as-is.

[ulfalizer]

@mbolivar
At the moment it's a no-op if you don't use it at least. My thinking was that having it in would make it easy for people to test it out, and would make it easy to add fixes incrementally.

Then, once it's ready, including the installation parts, we make the final switch.

[mbolivar]

At the moment it's a no-op if you don't use it at least. My thinking was that having it in would make it easy for people to test it out, and would make it easy to add fixes incrementally.

I understand, and I made the same argument in one of my PRs (#6832) regarding flashing and debugging.

But I have become convinced that Zephyr has hit the point where we should slow down on eagerly merging things before they are ready. I've marked my own PR with the "do not merge" tag as well in an effort to be consistent about this.

I have felt the pain of Zephyr's historical pattern of API churn over the past 16 months of working with it, and while I previously accepted it as a consequence of immaturity, I'm glad to see increased willingness among the project maintainers to put the brakes on that, even if it inconveniences me personally.

[ulfalizer]

@mbolivar
What problems do you see with having it in there from a practical perspective? I've never been much of a process guy, so that's what I mostly care about personally. :)

Having the experimental menuconfig in makes it easy to test it and to develop it incrementally. I also think it's more valuable to have a menuconfig in there that requires manual wheel installation on Windows than to have none at all, while you work on improving it. There are no points-of-no-return here that I can see.

[mbolivar]

There are no points-of-no-return here that I can see.

It's not about that. It's about how to decide whether or not to merge something.

From my perspective, merging code which implements critical workflow functionality for all Zephyr users and developers should be done when that code solves the problem on all supported platforms in a way we're all happy with. I think the Windows issue is a blocker here.

[ulfalizer]

@mbolivar
Get a working experimental version in, improve it incrementally till it's good enough (hopefully), and then make it official, is how I imagine it.

It's a nicer workflow than adding a big change all at once, IMO, especially for something where it's nice for people to be able to test it easily.

[mbolivar]

I have felt the pain of Zephyr's historical pattern of API churn over the past 16 months of working with it

@ulfalizer I want to underscore this point.

My company has had to develop a fair amount of tooling (with several FTEs involved) just to keep our Zephyr applications working at all given the amount of churn that has gone on (and, to be fair, continues to go on -- the DTS changes around LEDs and GPIO keys are currently preventing us from merging master and are the reason why the latest Zephyr newsletter has not gone out, since I currently write those as part of our merge workflow).

If the project is hitting the point where it's generally useful in "enough" scenarios where we can start to worry about stability, I'm 100% in favor of calming down eager merges despite lack of "points of no return" in order to have a more stable mainline.

[ulfalizer]

@mbolivar
This change is a no-op unless you use it though, so it shouldn't affect stability.

The initial version of the PR was a point-of-no-return that removed the C tools. I agree with @SebastianBoe that it's better to do it slowly and incrementally, so I changed it to just add the experimental menuconfig.

With the experimental menuconfig in, the other parts needed before we make it "official" can be added in a nice and incremental fashion. Once people are happy with it, we then make the final switch.

From a practical perspective, I think that's a nicer and more efficient workflow compared to having one huge PR at the end: less rebasing, easier to test, and small and digestible incremental improvements.

[mbolivar]

nicer and more efficient workflow compared to having one huge PR at the end

I understood from the TSC meeting that the Windows installation changes would come in the form of an additional Git repository under zephyrproject-rtos on GitHub and inclusion of its build artifacts in PyPI. So the incremental changes to this PR would just be adding documentation (or requirements.txt changes) to pull that package in from PyPI. I don't see why that is "huge".

Am I missing something?

[ulfalizer]

@mbolivar
This won't be done until we switch away from the C tools and fix whatever issues come up, so it'll take more changes. Until then, it's a no-op unless you use it, only there to make testing and development nice and incremental.

It's not really productive to argue back and forth though, so unless you change your mind, I'll go work on some stuff that will make this PR invalid as-is anyway (because it assumes the Kconfiglib change). :)

[mbolivar]

This won't be done until we switch away from the C tools and fix whatever issues come up, so it'll take more changes. Until then, it's a no-op unless you use it, only there to make testing and development nice and incremental.

In my view, from a project perspective, encouraging wider testing by merging into mainline is merited only after demonstrating the design's feasibility for use on supported platforms. A good-enough Windows workflow for installing this program is a prerequisite for such a demonstration, especially since curses is just one of multiple potential backends still under discussion, others of which already have been shown to work on Windows with installation via pip.

It's not really productive to argue back and forth though, so unless you change your mind, I'll go work on some stuff that will make this PR invalid as-is anyway (because it assumes the Kconfiglib change). :)

That's fair and your decision to make. I'm just responding to the code under consideration in this PR here.

[ulfalizer]

I talked to Cristoph Gohlke, and he's fine with us mirroring the Windows curses wheels and their source code.

I created a repository at https://github.com/ulfalizer/windows-curses-wheels for rebuilding the wheels easily. It has an overview of how the wheels work, and doubles as build instructions.

In practice, there seems to be more backwards- and forwards-compatibility than that would imply, but I did the "proper" version in case we'd need it.

I'm wondering if it's possibly to automatically install a particular wheel depending on what Python version is installed. Otherwise, a simple script that runs pip should be able to do it.

[ulfalizer]

I'm looking at getting the wheels onto PyPI now. If things work like I suspect they do, it might be enough to just drop all the wheels there, and pip/PyPI will take care of installing the right one.

[ulfalizer]

I'm looking at getting the wheels onto PyPI now. If things work like I suspect they do, it might be enough to just drop all the wheels there, and pip/PyPI will take care of installing the right one.

Yep, looks like it's that simple. I built a windows_curses-2.2-cp36-cp36m-win32.whl wheel from the sources, added it to PyPI, and pip install --index-url https://test.pypi.org/simple/ windows_curses installs a working version for 32-bit Python 3.6 (the default Python on my machine).

When I just had a different version of the wheel on PyPI, pip complained about incompatibility.

I could extend the instructions to cover adding the wheels to PyPI as well.

Any complaints about windows_curses as the name by the way? Do you want the PyPI account to be owned by Zephyr?

[ulfalizer]

@mbolivar
I've made the wheels available on PyPI. This should work now:

pip install windows-curses

Python 2.7, 3.5, 3.6, and 3.7 (32- and 64-bit, for all of them) are supported.

Compiling for Python 3.4 is a bit of a pain, as the 64-bit version of Visual C++ 10.0 isn't free, plus installing Visual Studio 2010 when Visual Studio 2017 is already installed is messy (the opposite works better).

Python 3.5 was released in September 2015, so maybe we can live without 3.4 for now.

The wheels are built from https://github.com/zephyrproject-rtos/windows-curses. The project needed to be renamed anyway, as PyPI isn't too happy about "curses" as a name. :)

[carlescufi]

I've made the wheels available on PyPI. This should work now:
pip install windows-curses

I tried this locally with one of the supported combinations (the one on my computer) which is Python 3.6, 64-bit. Worked like a charm.

[mbolivar]

I've made the wheels available on PyPI. This should work now:
pip install windows-curses

I tried this locally with one of the supported combinations (the one on my computer) which is Python 3.6, 64-bit. Worked like a charm.

Sounds great! Do you think you could post a diff to the installation documentation as well just so I can understand the flow?

I'm not sure if this is a windows-only thing or if it can be added to the 'core' requirements.txt, and there are other open questions about whether we want a scripts/win-requirements.txt if this is Windows-only, etc.

Just trying to make sure we've nailed down the impact on users here.

Thanks!

[mbolivar]

Just trying to make sure we've nailed down the impact on users here.

@carlescufi explained to me offline that this is a Windows-only step that will need to be mentioned from the Windows getting started guide, but that should be done (via requirements file or manual pip install) to get the Windows installation to parity with other platforms where these tools are concerned. That answers my questions about how it affects users.

I'm comfortable with that summary and plan for how to proceed. OK for getting the PR in for now, with the proviso that I think we need to nail down the docs (I'm in favor of adding a scripts/win-requirements.txt for this initially) before 1.12.

[ulfalizer]

@mbolivar
I added commands for rebuilding all wheels for Python 2.7, 3.5, 3.6, and 3.7 to the README in https://github.com/zephyrproject-rtos/windows-curses.

There's a section that explains how to add support for a new Python version now too, with the patch included.

I thought we had a windows-requirements.txt file or similar. Doesn't look like it though. Would've been a good place to put windows-curses otherwise.

[mbolivar]

@mbolivar
I added commands for rebuilding all wheels for Python 2.7, 3.5, 3.6, and 3.7 to the README in https://github.com/zephyrproject-rtos/windows-curses.

There's a section that explains how to add support for a new Python version now too, with the patch included.

Great! That's good for the windows-curses package's documentation, but a first-time Zephyr user should not have to hunt that down when getting started IMO.

I thought we had a windows-requirements.txt file or similar. Doesn't look like it though. Would've been a good place to put windows-curses otherwise.

Yup -- and setting that up / documenting this in a clean way is the final part of setting up the removal of the C tools, I think.