You will need to use the Parallel Computing Toolbox, and start a parallel pool with one member (and you would probably want to increase numthreads in the pool.) Then you would parfeval(), creating a "future" and which would proceed to run independently. In the client, you would monitor to see whether the future had completed, or whether the 10 minutes had elapsed. If the time had elapsed, then cancel() the future.
Note: if the function is deep inside a library routine, it might take some time to cancel. Mathematical calls involving large arrays can encounter this, and some symbolic functions can encounter this.
