Matlab Engine Array Transfer Size Limitation

I am using engGetVariable() to transfer an mxArray from the Matlab Engine to my C++ program. The documentation says: "The limit for the size of data transferred is 2 GB."

Why is this limitation present? Is there a way to increase it without breaking up the array into separate arrays? I need to transfer large arrays, potentially up to 256GB.

I like the simplicity of the Matlab Engine, so I would rather avoid having to break up, transfer, and re-assemble large arrays.

[James Tursa]

256GB is a lot of memory to copy. Have you considered a mex routine instead, so you can avoid the copy altogether?

[Jonathan]

Yes, actually I wrote one. The problem is that I need to solve a large, sparse matrix (Ax=b) on a cluster using Intel Pardiso Sparse Cluster solver.

Unfortunately, integrating Matlab with MPI and using the unsupported Intel MPI compilers seems infeasible.

In fact, I wrote a test program that solves the equation without the MPI code, compiled with GCC using MEX, and it exhibits some weird memory behavior.

So from the above, I concluded that getting this to work in MEX, running on a cluster with MPI is perhaps infeasible.

With regard to transferring 256GB, since I am doing the transfer within a single node (not over a network), it shouldn't take more than a few seconds in theory.

However, are you suggesting that I write my own MEX function that bypasses Matlab Engine to transfer directly using Linux IPC routines?

[Walter Roberson]

You can use MPI with MATLAB. One use of it is specifically documented; see http://www.mathworks.com/help/mdce/use-different-mpi-builds-on-unix-systems.html

For an example of someone using MPI see http://www.mathworks.com/matlabcentral/newsreader/view_thread/308346

Answer 1

[Walter Roberson]

The 2 Gb limit is for compatibility with 32 bit versions of MATLAB.

You should be considering using memmapfile() or shared memory