Save intermediate model in matlab while training a deep learning model and resume training from that stage later

Hi,

I am training a 3 pipeline deep learning model in matlab which takes a lot of time to train. I need to store intermediate variable values while training, stop the training process and then resume training at a later point from the stage at which the training was stopped previously. Does Matlab have any options to do this? Any help in this regard would be highly appreciated.

Thanks in Advance

Answer 1

[Mahesh Taparia]

Hi

You can set the checkpoint path in trainingOptions as suggested in the above answer. The trained weights will be saved into the specified path as a mat file. To resume the training process, you can load those weights in the net variable and start the training process. For example, you can refer this documentation.

Hope it will help!

[Jeet Agrawal]

Can you help me understanding load model more.

Do i need to create another file and load model and train from their?

suppose i am saving model after 50 iteration using

checkpointfrequency =50 and checkpointunit = iteration

lastly it has saved checkpoint at 750 now how to run again from 751?

As per link i can reduce my epoch but how can i reduce iteration?

suppose my data set has 20000 files and minibatchsize is 40 for one epoch it will take 500 iteration. I am not able to wait for 500 iteration to complete because it will take around 5 hours. so, want to save at 50 epochs only.

could you please share snipet?

Answer 2

[yanqi liu]

sir，may be use CheckpointPath，such as

options = trainingOptions('sgdm', ...
    'MaxEpochs', 5, ...
    'MiniBatchSize', 1, ...
    'InitialLearnRate', 1e-3, ...
    'CheckpointPath', tempdir);