CPU functions slower when mixed with GPU code?

I don't have too much experience with GPU coding in Matlab, but I noticed something very odd. When I take a program written entirely for the CPU and upgrade the subroutines that are most GPU friendly, I notice the the parts that I did not convert over to the GPU run much more slowly than in the older code (~3x longer run time): even though they are exactly the same functions. Is this a common thing, maybe related to how the JIT compiler handles hybrid code?

More specifically, I have a very long and complicated program that can be roughly split into two parts. The first does a bunch of FFT based 3D image processing; the second does some feature refinement using lots of conditionals and while loops and so on, but the bulk of the time is spent solving for all the zeros of a numerically defined vector function. In the CPU only code, the first part accounts for most of the run time, and the second part is done almost instantly. The hybrid code does the first part on the GPU and the second part remains on the CPU. Overall the hybrid code runs faster then the CPU only code, but the zero finding subroutines take a lot longer to complete.

Is this a common observation?

[Edric Ellis]

This is unexpected, is there any way you can post a (simple) reproduction? I presume you're using 'gather' on all your GPUArrays to bring them back to the CPU for the second part.