Clarification regarding 2GB transfer limit between MATLAB and Python
Could someone please clarify what exactly is meant by ‘The size of variables passed between Python and MATLAB is limited to 2 GB when you call a Python function out-of-process. This limit applies to the data plus supporting information passed between the processes.’ on https://uk.mathworks.com/help/matlab/matlab_external/limitations-to-python-support.html?
I want to run a sensitivity study using the MATLAB-based UQLab library (https://www.uqlab.com/), which comprises 12 evaluations of a custom Python model spanning many files. The Python model (Python 3.9) is treated as a black box function, which is executed from MATLAB as follows (NOTE: pyenv(ExecutionMode = "OutOfProcess")):
pyrunfile(strcat("python_file.py ", input_variables), output_variable);
where
"python_file.py " is the master script managing my Python model
input_variables is a JSON string of model inputs
output_variable is a single float constituting the model output
Halfway through the 11th model evaluation, MATLAB returns a Python package-specific error ("Python Error: TypeError: Numerical inputs to HAPropsSI must be ints, floats, lists, or 1D numpy arrays."). Now, when I ONLY run the 11th model evaluation I don’t get an error, and the results of the individual evaluation are identical to those from the full set of evaluations up to the point where the latter failed. Moreover, when I run all 12 evaluations on a subset of my model (less computationally/memory intensive), I don’t get an error either. After repeating the aforementioned steps multiple times, the error message always occurred in the exact same location, halfway through the 11th model evaluation.
All this suggests to me that I am exceeding the 2GB transfer limit between MATLAB and Python, but I don’t understand what is meant by ‘size of variables passed between Python and MATLAB’? The inputs and outputs described above are only a few byte, but I wouldn’t know what falls under ‘supporting information’.Could someone please clarify what exactly is meant by ‘The size of variables passed between Python and MATLAB is limited to 2 GB when you call a Python function out-of-process. This limit applies to the data plus supporting information passed between the processes.’ on https://uk.mathworks.com/help/matlab/matlab_external/limitations-to-python-support.html?
I want to run a sensitivity study using the MATLAB-based UQLab library (https://www.uqlab.com/), which comprises 12 evaluations of a custom Python model spanning many files. The Python model (Python 3.9) is treated as a black box function, which is executed from MATLAB as follows (NOTE: pyenv(ExecutionMode = "OutOfProcess")):
pyrunfile(strcat("python_file.py ", input_variables), output_variable);
where
"python_file.py " is the master script managing my Python model
input_variables is a JSON string of model inputs
output_variable is a single float constituting the model output
Halfway through the 11th model evaluation, MATLAB returns a Python package-specific error ("Python Error: TypeError: Numerical inputs to HAPropsSI must be ints, floats, lists, or 1D numpy arrays."). Now, when I ONLY run the 11th model evaluation I don’t get an error, and the results of the individual evaluation are identical to those from the full set of evaluations up to the point where the latter failed. Moreover, when I run all 12 evaluations on a subset of my model (less computationally/memory intensive), I don’t get an error either. After repeating the aforementioned steps multiple times, the error message always occurred in the exact same location, halfway through the 11th model evaluation.
All this suggests to me that I am exceeding the 2GB transfer limit between MATLAB and Python, but I don’t understand what is meant by ‘size of variables passed between Python and MATLAB’? The inputs and outputs described above are only a few byte, but I wouldn’t know what falls under ‘supporting information’. Could someone please clarify what exactly is meant by ‘The size of variables passed between Python and MATLAB is limited to 2 GB when you call a Python function out-of-process. This limit applies to the data plus supporting information passed between the processes.’ on https://uk.mathworks.com/help/matlab/matlab_external/limitations-to-python-support.html?
I want to run a sensitivity study using the MATLAB-based UQLab library (https://www.uqlab.com/), which comprises 12 evaluations of a custom Python model spanning many files. The Python model (Python 3.9) is treated as a black box function, which is executed from MATLAB as follows (NOTE: pyenv(ExecutionMode = "OutOfProcess")):
pyrunfile(strcat("python_file.py ", input_variables), output_variable);
where
"python_file.py " is the master script managing my Python model
input_variables is a JSON string of model inputs
output_variable is a single float constituting the model output
Halfway through the 11th model evaluation, MATLAB returns a Python package-specific error ("Python Error: TypeError: Numerical inputs to HAPropsSI must be ints, floats, lists, or 1D numpy arrays."). Now, when I ONLY run the 11th model evaluation I don’t get an error, and the results of the individual evaluation are identical to those from the full set of evaluations up to the point where the latter failed. Moreover, when I run all 12 evaluations on a subset of my model (less computationally/memory intensive), I don’t get an error either. After repeating the aforementioned steps multiple times, the error message always occurred in the exact same location, halfway through the 11th model evaluation.
All this suggests to me that I am exceeding the 2GB transfer limit between MATLAB and Python, but I don’t understand what is meant by ‘size of variables passed between Python and MATLAB’? The inputs and outputs described above are only a few byte, but I wouldn’t know what falls under ‘supporting information’. python, data transfer MATLAB Answers — New Questions
