Python fails to load after installing via (zsh's) pyenv (Python, R2018b, macos)

Straight to the point. I have Matlab 2018b, installed on mac, running BigSur (macos 11.4), with the default shell set as ZSH. In addition to the default python versions on the OS (2.7 & 3.8.2), Matlab 2018b compatibility is limited to Python3.5/3.6. Attempting to install 3.6.13 and other versions directly couldn't be performed on the OS - there are plenty of refs to this. Note, any reference to PYENV here is using the terminal as my Matlab isn't recent enough to support pyenv.m - I have to use pyversion.

Eventually, I found pyenv worked with more hacking (eg gitmemory, accessed 2021-07-01), and I now have 3.6.13 installed in my local directory (/Users/Pete/.pyenv/versions/3.6.13/*). However, I cannot get Matlab to call any of the libraries (I think this is the correct term).

I've gone through the various answers I can find and none of them work for me. As I've used PYENV to install python, it is now located in hidden directory on home path, so I've changed this by pyversion.

pyversion
       version: '3.6'
    executable: '/Users/pete/.pyenv/versions/3.6.13/bin/python3'
       library: ''
          home: '/Users/pete/.pyenv/versions/3.6.13'
      isloaded: 0

Running py to force a python load (hopefully!) then

py
py.numpy
Undefined variable "py" or class "py.numpy". 

Still not loaded.

getenv('PATH')
ans =
    '/usr/bin:/bin:/usr/sbin:/sbin'

This last one looked suspicious as Python executable isn't located here - according to terminal, python works fine with the pyenv install, and I'm able to call the common "numpy" library with ease.

TERMINAL OUTPUT (NOT Matlab):

    Last login: Thu Jul  1 11:07:29 on ttys000
pete@Petes-MacBook-Pro ~ % python3
Python 3.6.13 (default, Jun 29 2021, 13:19:52) 
[GCC Apple LLVM 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import sys
>>> sys.path
['', '/Users/pete/.pyenv/versions/3.6.13/lib/python36.zip', '/Users/pete/.pyenv/versions/3.6.13/lib/python3.6', '/Users/pete/.pyenv/versions/3.6.13/lib/python3.6/lib-dynload', '/Users/pete/.local/lib/python3.6/site-packages', '/Users/pete/.pyenv/versions/3.6.13/lib/python3.6/site-packages']
>>> np.array(5)
array(5)
>>> 

I've had a go at setting up the setenv('PATH'), but no luck with the python function being able to run...

setenv('PATH',fullfile('/Users/Pete/.pyenv/versions/3.6.13/lib'))
>> getenv('PATH')
ans =
    '/Users/Pete/.pyenv/versions/3.6.13/lib'
>> py
>> py.numpy
Undefined variable "py" or class "py.numpy". 
>> pyversion
       version: '3.6'
    executable: '/Users/pete/.pyenv/versions/3.6.13/bin/python3'
       library: ''
          home: '/Users/pete/.pyenv/versions/3.6.13'
      isloaded: 0
>> setenv('PATH',fullfile('/Users/Pete/.pyenv/versions/3.6.13/bin/'))
>> py
>> pyversion
       version: '3.6'
    executable: '/Users/pete/.pyenv/versions/3.6.13/bin/python3'
       library: ''
          home: '/Users/pete/.pyenv/versions/3.6.13'
      isloaded: 0
>> py.numpy
Undefined variable "py" or class "py.numpy". 

No matter how I have called this, the logical

      isloaded: 0

is always returned.

I'm obviously able to run Python via the terminal, but really wanted to have the option to run out of Matlab as this is what I'm more familiar with, and feel I can do more debugging this way, not to mention that ultimately, it should work once I get the setup right!

Any advice or things to try based on the above?

Thanks again in advance!

Answer 1

[Pete]

So following this terrible thread of getting nowhere fast, I took the plunge and removed BIG SUR, downgrading OSX to 10.13!

Took a few hours to get all files saved and then another few to downgrade.

Matlab reinstalled and...

>> py.list
ans = 
  Python list with no properties.
    []
>> pyversion
       version: '2.7'
    executable: '/usr/bin/python'
       library: '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib'
          home: '/System/Library/Frameworks/Python.framework/Versions/2.7'
      isloaded: 1

>> 

I'm going to make a time machine backup of this point and then try adding Python 3.6 the correct way!

Thanks Yongjian!

Answer 2

[Yongjian Feng]

Hello Pete,

Python path is different from the environment variable PATH. It is PYTHONPATH. Try to set PYTHONPATH instead.

Thanks,

Yongjian

[Pete]

Thanks for your comment Yongjian. Above my knowledge here though.

getenv('PYTHONPATH')
ans =
  0×0 empty char array

I assume

set('PYTHONPATH',fullfile('/Users/Pete/.pyenv/versions/3.6.13/bin/'))

is what I want here, or something different?

setenv('PYTHONPATH',fullfile('/Users/Pete/.pyenv/versions/3.6.13/bin/'))
>> getenv('PYTHONPATH')
ans =
    '/Users/Pete/.pyenv/versions/3.6.13/bin/'
>> py
>> pyversion
       version: '3.6'
    executable: '/Users/pete/.pyenv/versions/3.6.13/bin/python3'
       library: ''
          home: '/Users/pete/.pyenv/versions/3.6.13'
      isloaded: 0

Sorry for the lack of understanding, but playing around with paths and environments is nothing I've done before, so bear with me!

[Yongjian Feng]

PYTHONPATH is for python to look for packages (like numpy). PATH is for matlab to look for python executable.

So maybe PYTHONPATH should be the following instead? Do you have your numpy installed there?

'/Users/Pete/.pyenv/versions/3.6.13/lib'

[Pete]

numpy folder appears in

'/Users/Pete/.pyenv/versions/3.6.13/lib/python3.6/site-packages/'

which contains many folders (319 items), including others that have been installed via terminal using

python -m pip install pyautogui

Should the site-packages folder also be added? Still didn't work using the setenv command though. Should this line be added to startup.m? Or add something similar to ~/.zshrc?

I assume the command

setenv('PYTHONPATH',fullfile('/Users/Pete/.pyenv/versions/3.6.13/lib'))

is what I should be trying here?

When I say 'still didn't work', I'm usually trying the following in matlab:

py
pyversion

which keeps returning the loaded paths (as initial post) and the logical

      isloaded: 0

Entering

py.

then <TAB> I'm expecting to bring up a list of possible auto-completions, but nothing appears here as it does on a linux machine (including py.numpy). Entering either of the full expressions

py.numpy
% or
py.sys   

has kept returning the error

Undefined variable "py" or class "py.numpy".

Just so you know what I mean as 'it didn't work' is meaningless to anyone trying to help!

I'm assuming that "pyversion" is returning something that looks correct?

pyversion
       version: '3.6'
    executable: '/Users/pete/.pyenv/versions/3.6.13/bin/python3'
       library: ''
          home: '/Users/pete/.pyenv/versions/3.6.13'
      isloaded: 0

[Yongjian Feng]

try

which py

[Pete]

which py
/Applications/MATLAB_R2018b.app/toolbox/matlab/external/interfaces/python/py.m

from Matlab

And shell (via MBP terminal):

Last login: Thu Jul  1 13:15:59 on ttys000
pete@Petes-MacBook-Pro ~ $ which py  
py not found
pete@Petes-MacBook-Pro ~ $ which python 
/Users/pete/.pyenv/shims/python
pete@Petes-MacBook-Pro ~ $ 

Note in the above, the $ is replaced with % as it's ZSH, but this assumes ML code, so comments out the actual commands. So, that points differently, and similarly,

which python3
/Users/pete/.pyenv/shims/python3

so should this be the pyversion setup?

[Pete]

>> pyversion(fullfile('/Users/Pete/.pyenv/shims/python3'))
>> py
>> py.numpy
Undefined variable "py" or class "py.numpy". 
>> py.numpy
Undefined variable "py" or class "py.numpy". 
>> 

Dang

[Pete]

Hmmm, that didn't change pyversion

Even restarting matlab and first command (ensurig python doesn't get a chance to start)

>> pyversion('/Users/Pete/.pyenv/shims/python3')
>> pyversion
       version: '3.6'
    executable: '/Users/pete/.pyenv/versions/3.6.13/bin/python3'
       library: ''
          home: '/Users/pete/.pyenv/versions/3.6.13'
      isloaded: 0
>> 

[Yongjian Feng]

First make sure py is still good. Try

py.list

It should return an empty list

[]

[Pete]

Not working...

>> py.list
Undefined variable "py" or class "py.list". 
>> pyversion
       version: '3.6'
    executable: '/Users/pete/.pyenv/versions/3.6.13/bin/python3'
       library: ''
          home: '/Users/pete/.pyenv/versions/3.6.13'
      isloaded: 0
>> 

What paths are used on OSX? Both getenv('PATH') and getenv('PYTHONPATH'), as from the start I have felt there is something wrong with this stuff?

[Pete]

Can the 'LIBRARY' field be specified somehow using pyversion? Or will it be automatically found if the paths are correct?

[Yongjian Feng]

Our document recommends to use pyenv, instead of pyversion, at least for the latest R2021a.

https://www.mathworks.com/help/matlab/ref/pyversion.html

Please try pyenv(https://www.mathworks.com/help/matlab/ref/pyenv.html), even though you are using R2018

Answer 3

[Pete]

2018b doesn't include this though as a builtin function, so no surprises...

>> pyenv
Undefined function or variable 'pyenv'.

I would guess that asking for one to share such a function is against all rules as I don't have the subscription past 2018b!

I feel we're grasping at straws and not likely to have this solved?

[Yongjian Feng]

We can try more things. On my macbookpro it loads 2.7 successfully once started. Yours doesn't load python at all. Please try this one from matlab command line first

pyversion 3.6

Answer 4

[Pete]

I've not updated path/pythonpath before calling this. Just load ML and the following is ALL that is on the new session command window:

>> pyversion('3.6')
Error using pyversion
Cannot find specified version. 
>> pyversion(3.6)
Error using pyversion
Input must be a string scalar or character vector. 
>> pyversion('2.7')
Error using pyversion
Cannot find specified version. 
>> 

Consequently, the PATH is the ML default which is rather empty compared to the terminal version

>> getenv('PATH')
ans =
    '/usr/bin:/bin:/usr/sbin:/sbin'
>> 

Terminal/zsh:

echo $PATH
/Users/pete/.pyenv/shims:/Users/pete/.pyenv/bin:/Users/Pete/.pyenv/versions/3.6.13/bin/python3:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin

[Yongjian Feng]

I see. You are using python virtual env. The worse case scenario, we might need to install python not in the virtual env, if it is possible for you.

But before that, let see if we can get system working. Please try this:

setenv('PATH', ['/Users/pete/.pyenv/shims/', pathsep, getenv('PATH')]); 
system('python3 --version')

Answer 5

[Pete]

>> setenv('PATH', ['/Users/pete/.pyenv/shims/', pathsep, getenv('PATH ')]);
system('python3 --version ') 
env: bash: No such file or directory
ans =
   127
>> 

Hmmm, this appears to be using the bash rather than the OS default which is now ZSH. Is that what I read by the "env:bash" string and a potential issue?!

So this whole thing started with OSX Big Sur (MacOS 11.4), which restricted me from installing python in the root directories (/bin, /etc, and the write-protected locations, even using sudo). Hence using pyenv. BigSur caused other small issues (java robot via Matlab for example) and softwares to stop, so debated reverting back, and it's getting even more appealing with all this!

Obviously I can use Python outside of Matlab without issue, but the whole exercise was to try using it within Matlab!

Answer 6

[Pete]

Final closure on this... following a HDD wipe and downgrade to Yosemite (OSX10.10), I have several applications that require a minimum of OSX10.13 (High Sierra) so updated to that, hoping to stay with some 32-bit applications (e.g. FitsLiberator).

Successfully updated and all apps working as I'd hoped, including Matlab 'java.Robot' which I have used successfully prior to BigSur for moving the mouse around by code (Big Sur broke this!)

HomeBrew was the first issue that I had ran into with Big Sur - somewhere in it I had been unable to progress. Can't recall now, but it had caused me to require pyenv and so on.

Downloaded Python3.9 (current release at time of writing) and was worried ML might not be compatible as initially suggested by the ML compatibility chart. Updated the path in pyversion:

pyversion(fullfile('/usr/local/bin/python3'))

and expected to have all sorts of issues running from this.

>> pyversion
       version: '3.9'
    executable: '/usr/local/opt/python@3.9/bin/python3.9'
       library: '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/libpython3.9.dylib'
          home: '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'
      isloaded: 0
>> py.list
ans = 
  Python list with no properties.

    []
>> pyversion
       version: '3.9'
    executable: '/usr/local/opt/python@3.9/bin/python3.9'
       library: '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/libpython3.9.dylib'
          home: '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'
      isloaded: 1
>> 

YAY!

Obviously not the most elegant solution, but upgrading OSX to the most recent version as soon as possible has been a mistake for some of the features I require. High Sierra is also less intense on processor and battery life on the MBP has improved significantly also, not to mention it sounding quieter as the fans aren't going all the time as with later OS's.

Sorry there's not a 'solution', rather problem solved by drastic means, but everything appears to be working as desired.