Reproducibility convolutional neural network training with gpu

16 ビュー (過去 30 日間)

古いコメントを表示

Diego Alvarez Estevez 2018 年 9 月 20 日

2
リンク

この質問への直接リンク

https://jp.mathworks.com/matlabcentral/answers/419943-reproducibility-convolutional-neural-network-training-with-gpu

コメント済み: Hamza 2023 年 11 月 20 日

採用された回答: Joss Knight

MATLAB Online で開く

Hello,

I am training a CNN using my local GPU (to speed up training) for classification problems and would like to try different parameterizations. To avoid the variability effects due to different data and/or weights initialization I am resetting the random seeds each time before training:

% Initialize random seed (thus same dataset on same architecture would lead
% to predictable result)
rng(0);
%parallel.gpu.rng(0, 'CombRecursive');
randStream = parallel.gpu.RandStream('CombRecursive', 'Seed', 0);
parallel.gpu.RandStream.setGlobalStream(randStream);
% Train the CNN network
net = trainNetwork(TR.data,TR.reference,layers,options);

The problem is that when using GPU I am getting different results on each execution, even if initializing the GPU random seed to the same value. Strange thing is if I use CPU instead, then I do get the reproducible results. I am doing something wrong with GPU random seed initialization? Is there a know problem for this situation or something I am missing?

Thanks beforehand.

PS: I am using Matlab R2017b

0 件のコメント
-2 件の古いコメントを表示-2 件の古いコメントを非表示

サインインしてコメントする。

サインインしてこの質問に回答する。

採用された回答

Joss Knight 2018 年 9 月 20 日

1
リンク

この回答への直接リンク

https://jp.mathworks.com/matlabcentral/answers/419943-reproducibility-convolutional-neural-network-training-with-gpu#answer_337673

Use of the GPU has non-deterministic behaviour. You cannot guarantee identical results when training your network, because it depends on the whims of floating point precision and parallel computations of the form (a + b) + c ~= a + (b + c).

Most of our GPU algorithms are in fact deterministic but a few are not, for instance, backward convolution.