How can I make Grad-cam of CNN net

I made CNN net based on 1-D data (1*204)

layers=[
        imageInputLayer([1 204]);
        convolution2dLayer([1 3],64,'Stride',1,'Padding',[0 1]);
        convolution2dLayer([1 3],64,'Stride',1,'Padding',[0 1]);
        batchNormalizationLayer
        reluLayer();
        maxPooling2dLayer([1 2],'Stride',[1 2]);
        convolution2dLayer([1 3],128,'Stride',1,'Padding',[0 1]);
        convolution2dLayer([1 3],128,'Stride',1,'Padding',[0 1]);
        batchNormalizationLayer
        reluLayer();
        maxPooling2dLayer([1 2],'Stride',[1 2]);
        convolution2dLayer([1 3],256,'Stride',1,'Padding',[0 1]);
        convolution2dLayer([1 3],256,'Stride',1,'Padding',[0 1]);
        batchNormalizationLayer
        reluLayer();
        maxPooling2dLayer([1 2],'Stride',[1 2]);
        fullyConnectedLayer(400);
        fullyConnectedLayer(400);
        fullyConnectedLayer(4);
        softmaxLayer();
        classificationLayer();
        ];

I get a good CNN net, But I would like to which part of data is important to decide result.

I think It will be good way to use Grad-cam.

Many examples of Grad-cam are focused on Image. But my case is 1-D data not Image file..

Can you tell me which method will be good in my case?

Can I just use Grad-cam?,or more easy way to check which part is critical on result?

Answer 1

[Kenta]

Grad-cam is mainly used for images. In your case, LIME might be helpful.

https://jp.mathworks.com/matlabcentral/fileexchange/77828-explainable-ai-interpreting-the-classification-using-lime