Preserving gpuArray data across Multiple GPUs

We have 16 GPUs on a single compute node. I'd like to create data on each GPU and have it preserved across uses of gpuDevice (to switch from one GPU to another). From experimentation, it appears that data is destroyed when switching from one GPU to another.

I'd like to do something like:

Initialize all output and work arrays on all GPUs.
Do other initialization work.
Iterate over streaming input data:
  Have all GPUs work on the current group of data (using a parfor), generating its own output.
  Do a reduction operation across GPUs to gather the results from all of them to one place on the CPU.
end

I don't see how to do this presently when using gpuArray, which is what I would prefer to use. This, for example, does not work:

gpuDevice(1);
A = rand(100,100,'gpuArray');
gpuDevice(2);
existsOnGPU(A)
B = rand(100,100,'gpuArray');
existsOnGPU(A)
existsOnGPU(B)
gpuDevice(1);
existsOnGPU(A)
existsOnGPU(B)
gpuDevice(2);
existsOnGPU(A)
existsOnGPU(B)

existsOnGPU returns 0 when switching from one GPU to another for data that we previously on that GPU.

Is there a way to have gpuArray data preserved in a GPU when attention is (temporarily) focused on a different GPU?

Please advise.

Thanks so much.

Cheers. --Bracy

Answer 1

[Bracy Elton]

Added gpuArray tag

Answer 2

[Edric Ellis]

Each MATLAB process can access only a single GPU at a time. The only way to preserve data when switching GPU devices is to call gather to bring the data to the CPU.

However, I suspect this is not really what you want to do - if you're trying to take advantage of all GPUs simultaneously, you need to open a parallel pool, and have each worker use a different GPU device. There's a detailed blog article showing how to achieve that.