C/Fortran callback to MATLAB

I have a large library (~100K lines of code) written in modern Fortran, out of which I need to call only one function with the following C-signature,

#include <stdint.h>
void runFoo (
            // nd
            int32_t *
            // getFuncFromMatlab(nd, Point(nd)): procedure pointer to the MATLAB function
            , double (*)(int *, double [])
            // input character vector
            , char []
            // the length of the input chracter vector
            , size_t *
            );

Note that this function contains a callback to MATLAB. In other words, MATLAB calls this function, and this DLL function has to call another function in MATLAB provided by the original calling MATLAB function. I can generate a DLL from this library, and it loads in MATLAB via lodlibrary() with the above C header. However, that is no help, and I believe I am seriously missing some fundamental step to make this callback work. What is the best approach to solve this problem? MATLAB mex file compilation is no help either, as it apprently cannot even understand the syntax of modern Fortran, for example, module keyword. appreciate you help very much.

As a side note, as much as I find MATLAB a great standalone computing tool, I also find it terrible for any interoperation with any other languages. Hope this issue gets fixed. Are there any interoperation capabilities in MATLAB that would be equivalent to the simple, yet extremely powerful ctypes package of Python?

[Walter Roberson]

I believe that you are mistaken about support for modules.

If you are using Windows or Mac then Fortran support is by way of Intel Parallel Studio XE Fortran. For that compiler, files with extension .f90 are assumed to be Fortran 90/95 free form. You can specifically request that extensions beyond that be marked as errors by adding -std90 or -std95

If you are using Linux then support is through gfortran. You might need to add -std=f95

[A.B.]

Walter, thanks for your comment. The code is Fortran 2008 compliant. It will not compile under older standards. Does that mean Mex cannot compile the library? I am on Windows, but I'd like to also use it in Linux. I am using Intel Parallel Studio XE Fortran 2019 SP4.

[Walter Roberson]

https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference-standard-semantics

[Bruno Luong]

Does that mean Mex cannot compile the library?

Mex No (actually a MEX file is an DLL - so library - under windows just callable only under MATLAB), but MCC yes, see my post.

[James Tursa]

Are you asking if the argument with this signature

double (*)(int *, double [])

which is a pointer to a function that takes an int pointer and double pointer as input arguments and returns a double, can point to an arbitrary MATLAB function like an m-file?

[A.B.]

yes!

[James Tursa]

Two quick comments:

1) Whether or not you can use modern Fortran features depends entirely on your compiler, not on MATLAB or mex. The only catch is you may need to go into the mex configuration files and remove the silly /fixed option that TMW keeps insisting on including.

2) I don't know of any way to directly interface a C function pointer with a MATLAB m-file. Maybe there is a way with some of the loadlibrary capabilities, but I don't use loadlibrary so can't advise on this. For mex routines, typically one would use mexCallMATLAB for this.

[A.B.]

Thanks James, your response could take me to the next step indeed. I will have to try find that /fixed option see if that resolve the issue.

[Bruno Luong]

Why can't you compile your library as *.lib then link it whith a C/C++/FORTRAN MEX interface function?

You need to figure out suitable name mangling, calling convention to suit both sides but that all you need to do (well beside of course programming the interface function).

[James Tursa]

"You need to figure out suitable name mangling ..."

If you are using C++, I would highly advise using the extern "C" { ... } feature to avoid all name mangling.

[A.B.]

@james. @Bruno, comrades, I am back again to this same old problem. It seems like MATLAB Mex file is the way to go for this problem. I have never used mex files. Can you write a very simple example C-MATLAB callback Mex file to show me how this can be done? You have no idea how much I appreciate your help.

More details: I need to write a C Mex function with the above signature which calls a MATLAB function taking two arguments: one integer myint, and one array of doubles of size myint. Then I want MATLAB call this function in the C library and the C library will call a MATLAB function that was passed to the C library when called from MATLAB main program.

[James Tursa]

Is this function pointer argument:

double (*)(int *, double [])

supposed to call back and actually call a MATLAB m-file function at run time? I.e., You compile your library, but you don't know at compile time what the function pointer will point to. Then at run time, you pass in a string with the name of the MATLAB function you want that function pointer to use for a call back, and you want to make that connection at run time. Is that what you are trying to do? If so, this sounds doable and mexCallMATLAB would be the method to use, and I can write up a short example showing how to do this if needed.

HOWEVER, I would need to know what the function behind this function pointer is doing with the arguments. Specifically, is the data behind the int * and double [ ] input argument pointers read-only or is it supposed to get modified by the function. I need a much wider context of how this function will be used in your code to determine if there will need to be data copying going on behind those data pointers.

Maybe you could write up a short example using m-code and pseudo code or English to show what you want the code to do.

[A.B.]

James Tursa Thank you for your offer of help. very much appreciated. Your assumption is correct:

double (*)(int *, double [])

is a MATLAB callback function, to be passed by the MATLAB main.m script which calls an external library and passes this callback function to it.

To explain the specifics of the problem I am trying to solve. Here is this dynamic (DLL or shared) library that is supposed to call this MATLAB callback-function:

libparamonte_dynamic_heap_release_intel_c_windows_x64_mt.zip (Windows)

libparamonte_dynamic_heap_release_intel_c_darwin_x64_mt.tar.gz (macOS)

libparamonte_dynamic_heap_release_intel_c_linux_x64_mt.tar.gz (Linux)

I want to be able to call this dynamic library from within MATLAB and pass a mathematical function (implemented in MATLAB) to this library. The library will then call this MATLAB-function to compute its value for the given input values from the C side. For the C-signature of this MATLAB-callback-function, look at the contents of the file paramonte.h in the above tar/zip files. All I want to achieve is,

to write a MATLAB main.m file equivalent to the C main.c file inside the above packages,

to write a MATLAB logfunc.m file similar to the C logfunc.c file inside the above packages,

to be able to call this library by running the MATLAB main.m script.

As for your question, the library assumes that the user will not change the values of the passed arguments

(int *, double [])

from within the MATLAB-callback-function.

Again your help is tremendously appreciated as I cannot find any relevant good resources on the MATLAB website.

[James Tursa]

One final question. These input arguments

double (*)(int *, double [])

are coming from native C/C++ memory via malloc or local variables etc. and not from MATLAB generated memory via mxMalloc etc.?

[A.B.]

@James, that is correct. The two arguments are allocated inside the dll, MATLAB is only supposed to access the memory and read data, not write to it or destroy the allocation.

[Walter Roberson]

[You deleted a question before my response could go up, and you do not permit contacts in your profile, so the below is in response to the question you asked, not to the above Question]

• move the dylib to the /usr/lib folder on macOS, which macOS did not allow

You have to sudo in order to do it, but it does allow that.

I am surprised to see that in the year 2020, MATLAB and Apple still cannot resolve their problems like grown-ups and come to an agreement to recognize and respect each other's software.

Whether they do that or not, the C DLL is not Apple's software and it is not Mathworks software either, it is your software. Mathworks might have permitted Apple to scan its software to ensure that it is "safe", but Mathworks could not possibly have given Apple permission to scan your software because your software did not exist at the time Mathworks had MATLAB certified.

As far as Apple is concerned, SIP did what it is supposed to do: Prevent an unknown untrusted DLL from being used. Apple's SIP does not know that you, the user, created the DLL yourself: as far as it knows, an untrusted browser extension might have silently downloaded a DLL and is trying to trick MATLAB into running it.

As far as Apple is concerned, anyone who wants to write software that creates DLLs that will be executed by a program covered by SIP, should become a registered Apple developer and go through the process of getting every DLL scanned and certified before use.

If this leads you to the suspicion that Apple believes that end users (not part of the Developer Program) should not be compiling software for their own use, then that is because that is Apple's goal. Apple is pushing for the position that you should either be paying Apple to develop the software you want, or paying a certified third-party developer to develop the software you want. Apple is fond of "walled gardens". It is not (for now) going to prevent you from compiling your own software, but it is going to ensure that (A) you are inconvenienced; and (B) that the software runs in a lower-security context.

[A.B.]

Thanks, Walter. I did finally manage to run the dylib, which is the reason for deleting my question. I find it hard to believe that Apple cannot trust MATLAB. This seems to be more of a corporation rivalry than anything else, just the same way Intel and Apple appear to deal with each other right now. But in the end, the users suffer from the lack of cooperation between these big names. This is not a problem with my dylib alone. MATLAB's intrinsic dylib files are not recognized either, when I try to run a compiled version of the same script on the command line. Now, this might be resolved if I find and set all the relevant paths correctly (which I have not done yet). I also cannot buy Apple's argument for SIP. So many people, including me, survive on Linux, and have never encountered a problem there (at least I am not aware of). I agree that too few people live on Linux, but I find even Windows to be easier to deal with than macOS. Bottom line: thank you for investing your time to write a multi-paragraph response to a deleted question. Very much appreciated.

p.s. sudo does not let writing to /usr/lib/ either, at least on my macOS.

[Walter Roberson]

With regards to /usr/bin :

https://stackoverflow.com/questions/57440334/sip-is-disabled-but-usr-bin-is-write-protected-anyway-macos-catalina-beta-10

I find it hard to believe that Apple cannot trust MATLAB. This seems to be more of a corporation rivalry than anything else

Apple cares little what Mathworks thinks. Mathworks is not buying many systems from Apple, and not so many people buy Apple systems specifically to run MATLAB. Mathworks is just another customer of the Developer's Program as far as Apple is concerned.

Apple blew off NVIDIA, which is a $US100B+ company; Mathworks is small potatos by comparison. It isn't a matter of corporate rivalry: Mathworks is too small for Apple to notice. Apple makes decision, and Mathworks just has to live with the decisions.

[A.B.]

That's fair. Incidentally, do you (or James, now that we ended up on this question,) happen to know how to redirect the stdout from the dylib to the MATLAB command prompt (on macOS)?

Answer 1

[James Tursa]

Maybe this can give you the framework you need. I have made some assumptions about how things are connected. You pass in a char string of the MATLAB function you want called, and the callback function inside the C code calls it using some C native memory as input, simple as that.

The code:

// C = callback_test(fun)
//     fun = char string containing the MATLAB function name
//     C   = the result of the callback

#include "mex.h"
#include <stdlib.h>
#include <stdint.h>

void runFoo (
            // nd
            int32_t *
            // getFuncFromMatlab(nd, Point(nd)): procedure pointer to the MATLAB function
            , double (*)(int *, double [])
            // input character vector
            , char []
            // the length of the input chracter vector
            , size_t *
            );

double callback(int *, double []); // The callback function

char *mname; // MATLAB function name

//-------------------------------------------------------------------------

void mexFunction( int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
    int32_t n = 7;
    char *thestring = "testing123";
    size_t len = 10;
    
    if( nrhs && mxIsChar(prhs[0]) ) {
        mname = mxArrayToString(prhs[0]);
        mexPrintf("MATLAB function = %s\n",mname);
        runFoo( &n, callback, thestring, &len );
    }
}

//-------------------------------------------------------------------------

void runFoo (
            // nd
            int32_t *nd
            // getFuncFromMatlab(nd, Point(nd)): procedure pointer to the MATLAB function
            , double (*getFuncFromMatlab)(int *, double [])
            // input character vector
            , char str[]
            // the length of the input chracter vector
            , size_t *len
            )
{
    int i, ind;
    double *d;
    double result;
    
    // Generate some fake data using C native memory
    d = (double *) malloc( *nd * sizeof(*d) );
    for( i=0; i<*nd; i++ ) {
        d[i] = (double) (i+1);
    }
    
    // Call the callback function
    ind = *nd; // Just in case an int32_t is not the same as an int
    result = getFuncFromMatlab( &ind, d );
    
    // Print the results
    mexPrintf("Result = %g\n",result);
    mexPrintf("String passed = ");
    for( i=0; i<*len; i++ ) {
        mexPrintf("%c",str[i]);
    }
    mexPrintf("\n");
    
    // Deallocate the C native memory
    free(d);
}

//-------------------------------------------------------------------------

double callback(int *nd, double d[])
{
    mxArray *rhs[1], *lhs[1];
    double *pr;
    double result;
    int i;
    
    mexPrintf("Inside callback for function %s\n",mname);
    // Create the input argument
    rhs[0] = mxCreateUninitNumericMatrix( *nd, 1, mxDOUBLE_CLASS, mxREAL );
    // Copy the data (need to get C native memory into MATLAB memory)
    pr = (double *) mxGetData( rhs[0] );
    for( i=0; i<*nd; i++ ) {
        *pr++ = *d++;
    }
    // Make the call
    if( mexCallMATLAB( 1, lhs, 1, rhs, mname ) ) {
        mexErrMsgTxt("Callback didn't work");
    }
    result = mxGetScalar( lhs[0] );
    // Free the temporary variables
    mxDestroyArray( lhs[0] );
    mxDestroyArray( rhs[0] );
    //Done
    return result;
}

And some sample runs:

>> mex callback_test.c
Building with 'Microsoft Visual C++ 2013 (C)'.
MEX completed successfully.
>> callback_test('sum')
MATLAB function = sum
Inside callback for function sum
Result = 28
String passed = testing123
>> callback_test('prod')
MATLAB function = prod
Inside callback for function prod
Result = 5040
String passed = testing123
>> callback_test('sin')
MATLAB function = sin
Inside callback for function sin
Result = 0.841471
String passed = testing123
>> sin(1)
ans =
    0.8415
>> callback_test('garbage')
MATLAB function = garbage
Inside callback for function garbage
Error using callback_test
Undefined function 'garbage' for input arguments of type 'double'. 

[James Tursa]

Alas, I have never used MPI. One of the things I always intended to learn someday but as yet haven't found the time. How did you include the compiler flag? Are you sure it is getting to the compiler? E.g., are you building the mex routine via the mex command and modifying the $COMPFLAGS stuff, or are you building the mex routine outside of MATLAB?

You also mention the MATLB engine. Are you running the mex routine within a MATLAB Engine? Are you trying to run multiple MATLAB Engines in parallel using MPI?

[A.B.]

I followed an example on a NASA page that explained how to compile Fortran mex with extra compile flags, so I extrapolated the approach to Intel icc compiler. For example,

mex CC=icc CFLAGS="/IC:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.4.066\compilers_and_libraries_2019\windows\mpi\intel64\include" main.c mydll.lib

That did not work though. Based on the examples I have seen on the web, the MPI-parallelism basically requires starting one MATLAB engine per MPI task. On a side note, I do not think MATLAB officially supports MPI parallelism. But there are several external old packages written for this purpose.

Answer 2

[Bruno Luong]

Not sure what you mean by "Callback". Callback to me is kind of event trigger function like an SW interruption.

Anyway if you have the compiler, you can compile any MATLAB function in C shared library, and call it from external C program.

https://fr.mathworks.com/help/compiler_sdk/gs/create-a-c-application-with-matlab-code.html

I have no idea about FORTRAN, but in the pass I have linked fortran (gcc,g77) with MSVSC/Intel-C, a bit tricky but doable.

[A.B.]

Hi Bruno, Thanks, the code being in Fortran is almost irrelevant here, since modern Fortran enabled virtually full interoperation with C, as if you are dealing with C code instead of Fortran. So here is the situation: The user (me) wants to call subroutine Foo() in this Fortran library, which has the above C interface. I have this library's DLL file compiled by the Intel compiler, but apparently that is not enough. So what other methods exist to call this DLL library function from inside MATLAB?

How can I call this function via MATLAB's loadlibrary()? thanks again.