GPU CUDA error "An unexpected error occurred trying to launch a kernel. The CUDA error was: invalid configuration argument" resolves automatically on second try

Hi,

I'm trying to run code on GPU and I get the following error:

An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument

When I insert a breakpoint at the line that gives this error and run the code manually I get the same error but if I immediately try running it like again the code runs without the error. This proves that my code is not buggy and it is something else that's causing it to crash.

I have seen the same CUDA error using the NaN and subref functions. If it helps at all, here are the exact lines both of which give the same error on first try, but not on second try:

% On first try:
K>> W0(:,ones(1,size(dWU0,3)),:)
Error using gpuArray/subsref
An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument

% Here it is on second try, immediately following the last one (no actions in between):
K>> W0(:,ones(1,size(dWU0,3)),:)

ans(:,:,1) =

    0.0087    0.0087
    0.0135    0.0135
    0.0202    0.0202
    0.0303    0.0303
    0.0436    0.0436
    0.0667    0.0667
    0.0973    0.0973
    0.1273    0.1273
    0.1582    0.1582
    0.1935    0.1935
    0.2348    0.2348
    0.2767    0.2767
    0.3009    0.3009
    0.2821    0.2821
    0.2036    0.2036
    0.0705    0.0705
   -0.0867   -0.0867
   -0.2266   -0.2266
   -0.3169   -0.3169
   -0.3459   -0.3459
   -0.3220   -0.3220
   -0.2701   -0.2701
   -0.2121   -0.2121
   -0.1611   -0.1611
   -0.1206   -0.1206
   -0.0881   -0.0881
   -0.0612   -0.0612
   -0.0389   -0.0389
   -0.0218   -0.0218
   -0.0103   -0.0103
   -0.0032   -0.0032
    0.0013    0.0013
    0.0054    0.0054
    0.0086    0.0086
    0.0112    0.0112
    0.0124    0.0124
    0.0128    0.0128
    0.0093    0.0093
    0.0054    0.0054
    0.0059    0.0059
    0.0048    0.0048
    0.0045    0.0045
    0.0040    0.0040
    0.0039    0.0039
    0.0039    0.0039
    0.0039    0.0039
    0.0039    0.0039
    0.0038    0.0038
    0.0037    0.0037
    0.0037    0.0037
    0.0037    0.0037
    0.0036    0.0036
    0.0037    0.0037
    0.0037    0.0037
    0.0038    0.0038
    0.0039    0.0039
    0.0038    0.0038
    0.0037    0.0037
    0.0034    0.0034
    0.0033    0.0033
    0.0030    0.0030


ans(:,:,2) =

    0.0620    0.0620
    0.0931    0.0931
    0.1302    0.1302
    0.1729    0.1729
    0.2109    0.2109
    0.2376    0.2376
    0.2496    0.2496
    0.2512    0.2512
    0.2390    0.2390
    0.1930    0.1930
    0.0968    0.0968
   -0.0537   -0.0537
   -0.2302   -0.2302
   -0.3746   -0.3746
   -0.4279   -0.4279
   -0.3749   -0.3749
   -0.2513   -0.2513
   -0.1257   -0.1257
   -0.0487   -0.0487
   -0.0268   -0.0268
   -0.0308   -0.0308
   -0.0354   -0.0354
   -0.0321   -0.0321
   -0.0256   -0.0256
   -0.0231   -0.0231
   -0.0242   -0.0242
   -0.0249   -0.0249
   -0.0207   -0.0207
   -0.0136   -0.0136
   -0.0073   -0.0073
   -0.0027   -0.0027
    0.0023    0.0023
    0.0103    0.0103
    0.0158    0.0158
    0.0197    0.0197
    0.0195    0.0195
    0.0170    0.0170
    0.0145    0.0145
    0.0147    0.0147
    0.0094    0.0094
    0.0094    0.0094
    0.0077    0.0077
    0.0048    0.0048
    0.0052    0.0052
    0.0047    0.0047
    0.0049    0.0049
    0.0048    0.0048
    0.0043    0.0043
    0.0033    0.0033
    0.0027    0.0027
    0.0026    0.0026
    0.0023    0.0023
    0.0025    0.0025
    0.0032    0.0032
    0.0044    0.0044
    0.0053    0.0053
    0.0060    0.0060
    0.0057    0.0057
    0.0045    0.0045
    0.0032    0.0032
    0.0016    0.0016


ans(:,:,3) =

   -0.0826   -0.0826
   -0.0976   -0.0976
   -0.1043   -0.1043
   -0.0936   -0.0936
   -0.0674   -0.0674
   -0.0228   -0.0228
    0.0267    0.0267
    0.1056    0.1056
    0.1959    0.1959
    0.2812    0.2812
    0.3282    0.3282
    0.3057    0.3057
    0.2109    0.2109
    0.0744    0.0744
   -0.0401   -0.0401
   -0.0647   -0.0647
    0.0084    0.0084
    0.1415    0.1415
    0.2627    0.2627
    0.3130    0.3130
    0.2663    0.2663
    0.1646    0.1646
    0.0571    0.0571
   -0.0321   -0.0321
   -0.0928   -0.0928
   -0.1287   -0.1287
   -0.1545   -0.1545
   -0.1726   -0.1726
   -0.1818   -0.1818
   -0.1817   -0.1817
   -0.1754   -0.1754
   -0.1644   -0.1644
   -0.1537   -0.1537
   -0.1374   -0.1374
   -0.1179   -0.1179
   -0.1006   -0.1006
   -0.0818   -0.0818
   -0.0607   -0.0607
   -0.0417   -0.0417
   -0.0325   -0.0325
   -0.0220   -0.0220
   -0.0142   -0.0142
   -0.0041   -0.0041
   -0.0028   -0.0028
   -0.0012   -0.0012
   -0.0020   -0.0020
   -0.0022   -0.0022
   -0.0012   -0.0012
    0.0008    0.0008
    0.0026    0.0026
    0.0037    0.0037
    0.0041    0.0041
    0.0036    0.0036
    0.0031    0.0031
    0.0029    0.0029
    0.0032    0.0032
    0.0031    0.0031
    0.0034    0.0034
    0.0032    0.0032
    0.0024    0.0024
    0.0019    0.0019

% Here's another line of my code showing the same strange behavior:
% First try:
K>> y = nan(s,'like',x)
Error using gpuArray/nan
An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument

% Second try:
K>> y = nan(s,'like',x)

y =

   NaN

This is very perplexing to me. Any thoughts?

[Ganesh Regoti]

Hi Sia,

Can I get the following information which would be helpful in finding the issue.

1. Which Driver version you are using?
2. Which toolkit and it version if you are using one.
3. The sizes of the data structures you are using.
4. Whether you are using GPUarray or GPUCoder
5. If you could share the script /function that would give more insight.

[Jai Bhagat]

Hi Sia, I had the same issue running kilosort2. Did you figure out a fix in the end? I saw something similar was posted as an issue to kilosort2's github, but Marius closed the issue due to inactivity.

[Y.Yang]

I have the same problem when using permute with gpuArray matrix.

[Clay Smyth]

Have there been any solutions? Running into the issue with kilosort 2.

[Udaysankar Chockanathan]

I'm also getting this issue, but only for certain recordings.

Answer 1

[Ben]

Maybe you can consider trying to reduce the size of W0. I encountered this problem before, and it won't happen again until I reduce the size.