Failing to free GPU memory on cleared/re​placed/gat​hered variables: 2017a

Hello all, I am running into an out of memory error on my GPU (8 GB, GTX 1080). I have drilled down to the code segment that keeps throwing the error, and it appears to be an issue with Matlab failing to free memory on the gpu after calculation. Specifically, this is occurring after an interp2 command.

I am attaching a MWE as a PDF. Long story short, this shows that after the interp2 command, a sizeable block of GPU memory never gets freed up even after all of the workspace variables on the GPU being cleared, gathered, or replaced. This is a major issue if the result of the interp2 operation is large enough that gather(gpuData) causes an out of memory error; there is no way to pull the data off of the GPU without resetting the GPU, which wipes the data.

Is this a 2017a issue? Any help would be appreciated.

Answer 1

[Joss Knight]

It's difficult to say exactly what is going on here, however I'm fairly sure that it is the cublas context that is occupying memory on the first call. This is perfectly normal and doesn't constitute any kind of memory leak. You can run this computation repeatedly from this point on and the overall memory usage remains the same. I believe it's cublas because if I replace your function with a simple call to v.*v I don't see this behaviour; whereas if I call v*v (a cublas call) I do. I think cublas is pooling a lot of memory and kernels and this is not reported as free memory.

[D. Plotnick]

Thanks for your response.

Unfortunately this doesn't really help me move past the issue: the problem in my case is that the variable I need (vi_g in the example case) is sufficiently large, and the AvailableMemory sufficiently small, that vi_g = gather(vi_g) results in an OutOfMemory error.

I cannot get the data off of the GPU, even though the data is ~ 2.5 GB on a 8 GB card (7 GB free according to matlab, when everything is clean).

Is there a way to free up that cublas context, without using reset and losing the gpuArray I need, or is there another option beyond gather that will let me pull the array off of the card without hitting the memory error?

Thanks again, -Dan