pyenv 'command not found' with system python set

[cas--]

Describe the bug
Encountering error running py due to pyenv installed and set to system

$ py
pyenv: python3.9: command not found

The `python3.9' command exists in these Python versions:
  3.9.2

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.

To Reproduce

1. Install pyenv
2. pyenv install 3.9.2
3. cargo install python-launcher
4. py

Tried export PY_PYTHON as per FAQ but version file does not exist if global not set:

head: cannot open '~/.pyenv/version' for reading: No such file or directory

So tried creating file with pyenv global but that also failed on the special string system:

Error parsing 'system' as an integer: invalid digit found in string

Expected behaviour
Opens system python 3.8 prompt

Screenshots

$ pyenv versions
* system (set by ~/.pyenv/version)
  3.9.2

$ py --list
 3.9 │ ~/.pyenv/shims/python3.9 
 3.8 │ /usr/bin/python3.8                 
 3.6 │ /usr/bin/python3.6                 
 2.7 │ /usr/bin/python2.7 

System Details (please complete the following information):

• OS: Ubuntu 20.04
• Shell: bash
• Launcher Version: 1.0.0

[brettcannon]

Can you run with logging turned on?

PYLAUNCH_DEBUG=1 py

I'm not a pyenv user so I can only guess it's because python3.9 isn't an actual executable but some shell function or something which can't be directly executed via execv().

[brettcannon]

BTW, what does system represent? Is it just reading the version number from ~/.pyenv/version? And if so what is in that file?

[cas--]

Yeah the pyenv python3.9 is a shim and from what I understand is not meant to be run directly, perhaps skip any executable in pyenv shims dirs?

PYLAUNCH_DEBUG=1 py
Checking for VIRTUAL_ENV environment variable
Searching for a venv in /home/calum and parent directories
Checking /home/calum/.venv/bin/python
Checking /home/.venv/bin/python
Checking /.venv/bin/python
Checking the PY_PYTHON environment variable
PY_PYTHON not set
Checking PATH environment variable
PATH: ["/home/calum/.pyenv/shims", "/home/calum/.nvm/versions/node/v12.22.1/bin", "/home/calum/.pyenv/plugins/pyenv-virtualenv/shims", "/home/calum/.pyenv/shims", "/home/calum/.pyenv/bin", "/home/calum/.cargo/bin", "/home/calum/.poetry/bin", "/home/calum/.local/bin", "/usr/local/sbin", "/usr/local/bin", "/usr/sbin", "/usr/bin", "/sbin", "/bin", "/usr/games", "/usr/local/games", "/snap/bin"]
Found executables: ["/usr/bin/python2.7", "/usr/bin/python3.6", "/home/calum/.pyenv/shims/python3.9", "/usr/bin/python3.8"]
Executing /home/calum/.pyenv/shims/python3.9 with []
pyenv: python3.9: command not found

The `python3.9' command exists in these Python versions:
  3.9.2

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.

BTW, what does system represent? Is it just reading the version number from ~/.pyenv/version? And if so what is in that file?

system can be used in the ~/.pyenv/version file as a special string to denote that pyenv should use the default system python not a pyenv installed version.

[brettcannon]

perhaps skip any executable in pyenv shims dirs?

I don't want to add pyenv-specific logic into the Launcher. Does pyenv manipulate your PATH variable to put just "/home/calum/.pyenv/shims/python3.9" on it for use?

system can be used in the ~/.pyenv/version file as a special string to denote that pyenv should use the default system python not a pyenv installed version.

So would that mean if it's set to system then the Launcher should just follow its own logic to select what to run? At this point it seems like the Python Launcher is potentially not that useful to you if pyenv is managing what python/python3 is pointing at.

[brettcannon]

The other solution for dealing with system is to expand the shell snippet in the FAQ to recognize system in ~/.pyenv/version and to skip setting PY_PYTHON in that instance.

I realize that doesn't solve your shims/python3.9 issue.

[gaborbernat]

I don't think you can solve this without integrating against pyenv, see pypa/virtualenv#2130 that's related. An executable named python can literally point to any python with pyenv. You'd have to duplicate the logic of pyenv shim layer to get it right.

[jefftriplett]

Long-time pyenv user here, and while it's possible to use pyenv to manage your system python, it's not really built for managing your system-level python. I'd recommend running pyenv global 3.9.2 to use pyenv's installed python version which is out-of-scope of the report.

If you want py to support it, I would treat system the same as if there were no version specified and use whatever your system-level default is. I would one-off ignore it and fall back to whatever logic you are already using.

[brettcannon]

Would this work as the updated entry for the FAQ for bash and zsh?

pyenv_version=$(head -n 1 pyenv-version.txt | cut -d "." -f 1,2)
if [ $pyenv_version != 'system' ]; then
    export PY_PYTHON=$pyenv_version
fi;

[brettcannon]

And for fish:

set pyenv_version (head -n 1 pyenv-version.txt | cut -d "." -f 1,2)
if test "$pyenv_version" != "system"
    set -xg PY_PYTHON $pyenv_version
end

[brettcannon]

Otherwise we can add a disclaimer to the FAQ entry to say system is not a supported version (or both; simple sample or more complex sample depending on what people want).

[cas--]

Perhaps my original intent for this bug report has lost it's way. I was interested to try out python-launcher but since I happen to have pyenv installed it failed to work. I imagined that this could be a common use-case so reported the bug so that it could be logged, considered and fixed.

I don't want to add pyenv-specific logic into the Launcher.

I understand the sentiment but if pyenv has been installed it seems unavoidable if launcher is unwittingly picking up the pyenv shims.

pyenv global path

while it's possible to use pyenv to manage your system python, it's not really built for managing your system-level python

An install of pyenv with env vars added to ~/.profile is already managing the system python. When I tried setting the pyenv global version to system it was an attempt to workaround the original bug but encountered a further bug with that value not being a valid python version.

e.g. after pyenv install this is now the python3 path:

$ which python3
~/.pyenv/shims/python3

pyenv shims

From understanding-shims

Shims are lightweight executables that simply pass your command along to pyenv. So with pyenv installed, when you run, say, pip, your operating system will do the following:

• Search your PATH for an executable file named pip
• Find the pyenv shim named pip at the beginning of your PATH
• Run the shim named pip, which in turn passes the command along to pyenv

This means that we cannot escape handling pyenv with it added to PATH:

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"

Does pyenv manipulate your PATH variable to put just "/home/calum/.pyenv/shims/python3.9" on it for use?

No the directory of shims is added to PATH:

~/.pyenv/plugins/pyenv-virtualenv/shims:~/.pyenv/shims:~/.pyenv/bin

Maybe the best-effort here is to check for PYENV_ROOT and exclude any paths prefixed by that?

Otherwise we can add a disclaimer to the FAQ entry to say system is not a supported version (or both; simple sample or more complex sample depending on what people want).

I am not sure it makes sense to have that particular FAQ entry for pyenv since it is already going to be overriding the PATH python and launcher should be treating it as the OS python?

After considering all of this I can understand if you instead want to say that launcher is not compatible with pyenv 😄

[fredrikaverpil]

I set pyenv to system because I don't want to interfer with what the system (and potentially other apps) expects, when outside of my python project/git directories.

Then when creating a new git project, I always start with running e.g. "pyenv local 3.9.6" which creates a .python_version file. Now, when in this directory, the pyenv shim detects this file and sets the "python" executable into 3.9.6. So now I can move on with creating my venv, setup my deps etc by executing "python".

All of the above of course requires me to have previously executed "pyenv install 3.9.6" so this python version gets installed on my system.

Given my workflow, I'm wondering how useful the python launcher is to you (me) if you are already using pyenv...?

I don't know if this helps, but I figured I'd share my workflow.

[brettcannon]

Maybe the best-effort here is to check for PYENV_ROOT and exclude any paths prefixed by that?

That's still pyenv-specific and so I would rather not do that.

After considering all of this I can understand if you instead want to say that launcher is not compatible with pyenv.

I don't want to go that far as people like @jefftriplett have shown they get use out of the Launcher with pyenv (as has @treyhunner and other folks who helped write that FAQ entry). Maybe they can share how they make them work together?

Then when creating a new git project, I always start with running e.g. "pyenv local 3.9.6" which creates a .python_version file.

Then the Launcher isn't necessarily that useful. It still helps with shebangs and using the newest version of Python and using virtual environments automatically. But if you're already using pyenv to accomplish this then you don't need the Launcher (or you can obviously switch over if you want 😉).

[cas--]

I have gained a bit more understanding after reading the original problem that the FAQ entry was written for. In that scenario Python 3.10 is in the PATH (perhaps not provided by pyenv?) and Launcher is picking that first.

Perhaps could be solved by instead of trying to parse a pyenv specific version file (that may or may not exist) we use the pyenv command to get the version from the python it considers current default. My proposal would be to replace the FAQ entry for pyenv with the following:

export PY_PYTHON=$(pyenv exec python -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))") 

From my quick testing this solves my issue and original FAQ issue but would like some confirmation

Thanks for your patience Brett 😁

[jefftriplett]

@cas-- does this last script work for you when you have system set as your global version in python?

It seems to work for me, but I don't set PY_PYTHON.

[cas--]

It works for all combinations however for completeness need to include PY_PYTHON3 for py -3 command:

export PY_PYTHON=$(pyenv exec python -c "import sys; print('.'.join(map(str, sys.version_info[0:2])))") 
export PY_PYTHON3=$(pyenv exec python3 -c "import sys; print(*sys.version_info[0:2], sep='.')") 

* Didn't use print sep arg for python just in-case python2 is being used.

The final issue to note is that once env vars are set any changes to pyenv global will not update them until session reload. Could be fixed with alias for py prefixed with env vars set at runtime but might be overkill.

[brettcannon]

I left a comment on the review, but that's not quite the right use of PY_PYTHON as that env var is for use when no version is specifically requested, not what python may point at (depending on your OS you're doing what PY_PYTHON2 should potentially be).