How to get shapley value for Neural Network trained on matlab? it keeps error...

Hi there,

I wanted to get shapley value of my pre-trained ANN.

it is regression model.

it's input's shape is 7*5120 double

and output is 1*5120 double.

I'm confused with idea of shapley.. sorry

[Ronit]

Hello,

In game theory, the Shapley value of a player is the average marginal contribution of the player in a cooperative game. That is, Shapley values are fair allocations, to individual players, of the total gain generated from a cooperative game. In the context of machine learning prediction, the Shapley value of a feature for a query point explains the contribution of the feature to a prediction (the response for regression or the score of each class for classification) at the specified query point.

The Shapley value corresponds to the deviation of the prediction for the query point from the average prediction, due to the feature. For each query point, the sum of the Shapley values for all features corresponds to the total deviation of the prediction from the average.

Please refer to the documentation of Shapley Values for Machine Learning Model which can be used during implementation:

https://www.mathworks.com/help/stats/shapley-values-for-machine-learning-model.html

Note: shapley function in MATLAB is available in Statistics and Machine Learning Toolbox.

I hope it helps you query!

[Angelo Yeo]

Can you be more specific about your model and the error message? It's the best if you can share your model (code and data) and the reproduction steps for the error.

[한용]

thanks!

I've tried and managed to make it work.

but I want to change predictor's classes name.

I'd copied table and manually changed and manually plotted. but i want to change originals.

here's my code and want some help. I appreciate your help very much. thank you.

clc;
clear;
% 모델과 데이터 로드
net_temp = load('net_model_F1.mat');
net = net_temp.net;
%% Model F1
train_in_total_temp = load('Train_in.mat');
train_out_total_temp = load('Train_out.mat');
train_in_total = train_in_total_temp.Train_in;
train_out_total = train_out_total_temp.Train_out;
input = train_in_total(:,1:7);  % model_S
output = train_out_total(:,1);
veri_in_total_temp = load('Veri_in.mat');
veri_out_total_temp = load('Veri_out.mat');
veri_in_total = veri_in_total_temp.Veri_in;
veri_out_total = veri_out_total_temp.Veri_out;
input_veri = veri_in_total(:,1:7);  % model_S
output_veri = veri_out_total(:,1);
%% 입력 및 출력 데이터 정규화
input_max = zeros(1, size(input, 2));
input_min = zeros(1, size(input, 2));
output_max = zeros(1, size(output, 2));
output_min = zeros(1, size(output, 2));
for i = 1:size(input, 2)
    input_max(i) = max(input(:,i));
    input_min(i) = min(input(:,i));
end
for i = 1:size(output, 2)
    output_max(i) = max(output(:,i));
    output_min(i) = min(output(:,i));
end
x_train_re = zeros(size(input));
y_train_re = zeros(size(output));
for i = 1:size(input, 2)
    x_train_re(:,i) = (input(:,i) - input_min(i)) / (input_max(i) - input_min(i));
end
for i = 1:size(output, 2)
    y_train_re(:,i) = (output(:,i) - output_min(i)) / (output_max(i) - output_min(i));
end
x_train = x_train_re';
t_train = y_train_re';
y_train = net(x_train);
%% 입력 및 출력 데이터 정규화
x_veri_re = zeros(size(input_veri));
y_veri_re = zeros(size(output_veri));
for i = 1:size(input_veri, 2)
    x_veri_re(:,i) = (input_veri(:,i) - input_min(i)) / (input_max(i) - input_min(i));
end
for i = 1:size(output_veri, 2)
    y_veri_re(:,i) = (output_veri(:,i) - output_min(i)) / (output_max(i) - output_min(i));
end
x_veri = x_veri_re';
t_veri = y_veri_re';
y_veri = net(x_veri);
test_y = t_veri;
test_y_val = y_veri;
%% Shapley 값 계산
f = @(x) net(x')';  % 인공신경망 모델 함수를 정의
x_veri_shapley = x_veri';  % 각 행이 하나의 샘플이 되도록 전치
x_train_shapley = x_train';  % 각 행이 하나의 샘플이 되도록 전치
% 샘플링 예시 (500개 데이터 포인트 샘플링)
num_samples = 500; % 샘플링할 데이터 수
idx = randperm(size(x_veri_shapley, 1), num_samples);
x_veri_shapley_sampled = x_veri_shapley(idx, :);
% % 병렬 처리 활성화
explainer = shapley(f, x_train_shapley, 'QueryPoints', x_veri_shapley_sampled,'UseParallel', true);
%%
plot(explainer)
%%
% MeanAbsoluteShapley table을 복사
shapley_table = explainer.MeanAbsoluteShapley;
% 변수 이름 변경
desired_variable_names = {'PGA', 'Dur_{sig}', 'Sa_{max}', 'Tm', 'CAV_{max}', 'Arias_{max}', 'f_{1}'};
shapley_table.Predictor = desired_variable_names(:);  % 새 변수 이름으로 교체
% Shapley 값과 변수 이름을 Shapley 값의 내림차순으로 정렬
[sorted_values, sort_index] = sort(shapley_table.ShapleyValue, 'ascend');
sorted_names = shapley_table.Predictor(sort_index);
% 막대 그래프 그리기 (큰 값부터 작은 값 순서로)
figure;
barh(sorted_values);
set(gca, 'YTickLabel', sorted_names);
xlabel('Shapley 절댓값의 평균');
ylabel('예측 변수');
title('Shapley 중요도 플롯');
%%
figure(10);
plot(explainer,QueryPointIndices=30);
figure(11);
plot(explainer);
figure(12);
swarmchart(explainer);

Answer 1

[Angelo Yeo]

I do not have the model and dataset, so I used a random samples. The key is to use yticklabels. Would this work for you?

clc;
clear;
%% Shapley 값 계산
% demo neural network
x = randn(7, 150);
t = randn(1, 150);
net = fitnet(10);
net = configure(net,x,t);
% view(net)

f = @(x) net(x')';  % 인공신경망 모델 함수를 정의
x_veri_shapley = x(:,101:end)';  % 각 행이 하나의 샘플이 되도록 전치
x_train_shapley = x(:, 1:100)';  % 각 행이 하나의 샘플이 되도록 전치
% 샘플링 예시 
num_samples = size(x_veri_shapley,1); % 샘플링할 데이터 수
idx = randperm(size(x_veri_shapley, 1), num_samples);
x_veri_shapley_sampled = x_veri_shapley(idx, :);
% % 병렬 처리 활성화
explainer = shapley(f, x_train_shapley, 'QueryPoints', x_veri_shapley_sampled,'UseParallel', false);
%%
% plot(explainer)
%%
% MeanAbsoluteShapley table을 복사
shapley_table = explainer.MeanAbsoluteShapley;
% 변수 이름 변경
desired_variable_names = ["PGA", "Dur_{sig}", "Sa_{max}", "Tm", "CAV_{max}", "Arias_{max}", "f_{1}"];

shapley_table.Predictor = desired_variable_names(:);  % 새 변수 이름으로 교체
% Shapley 값과 변수 이름을 Shapley 값의 내림차순으로 정렬
[sorted_values, sort_index] = sort(shapley_table.ShapleyValue, 'ascend');
sorted_names = shapley_table.Predictor(sort_index);
% 막대 그래프 그리기 (큰 값부터 작은 값 순서로)
% figure;
% barh(sorted_values);
% set(gca, 'YTickLabel', sorted_names);
% xlabel('Shapley 절댓값의 평균');
% ylabel('예측 변수');
% title('Shapley 중요도 플롯');
%%
close all;
figure(10);
plot(explainer,QueryPointIndices=30);
hAxes = gca;
hAxes.TickLabelInterpreter = "tex";
yticklabels(hAxes, sorted_names) % use "yticklabels" to change the YTickLabels

figure(11);
plot(explainer);
hAxes = gca;
hAxes.TickLabelInterpreter = "tex";
yticklabels(hAxes, sorted_names) % use "yticklabels" to change the YTickLabels

figure(12);
swarmchart(explainer);
hAxes = gca;
hAxes.TickLabelInterpreter = "tex";
yticklabels(hAxes, sorted_names) % use "yticklabels" to change the YTickLabels