neural networks, cross validation, seting traing,test and validation sets, all posibles subset of feature

Respected colleague

I want to investigate influence od different variables on my neural network model using all possible subset feature selection(there is 8191 posible subsets) I have problem to force model to use net.divideFcn with specific train, validation and test indices. I use net.divideFcn with specific indices, but my code always give me divide random functions. Its important for me to use predefined indices using divideind.In my code below i put only 50 posible cominations of variables (total numbesr of combinations is 8191) because of the speed of execution.

I dont know what is problem... I want order all variables infuence in my modelusing criteria mse or RMSE.

Thanks

clear

 load house_dataset
inputs = houseInputs;
 targets = houseTargets;
N=506;
ind=randperm(506);
index = dec2bin(1:8191);
index = index == '1';
index_transpose=transpose(index);
index_double=double(index_transpose);
results = index_double;
results(14,:) = zeros(length(results),1);
for m = 1:50
 foo = index_transpose(:,m);
inputs_i=inputs(foo,:);
k=10;
for i = 1:k
      rngstate(i) = rng;
M=50;
      valind = 1 + M*(i-1) : M*i; 
      if i==k
           tstind = 1:M;
           trnind = [ M+1:M*(k-1) , M*k+1:N ];
      else
           tstind = valind + M;
           trnind = [ 1:valind(1)-1 , tstind(end)+1:N ];
      end;

hiddenLayerSize = 3;

trnInd = ind(trnind);

valInd = ind(valind);

tstInd = ind(tstind);

Inputs_train=inputs_i(:,trnInd);

Inputs_valid=inputs_i(:,valInd);

Inputs_test=inputs_i(:,tstInd);

targets_train=targets(trnInd);

targets_valid=targets(valInd);

targets_test=targets(tstInd);

net = fitnet(hiddenLayerSize);

net.divideFcn='divideind';

net=train (fitnet(3) , Inputs_train , targets_train);

simulate=net(Inputs_test);

error=simulate-targets_test;

square_error=sum((error).^2);

RMSE(m,i)=((square_error)^0.5);

RMSE_finall=mean(RMSE,2);

end;

end;

results(14,1:50)=transpose(RMSE_finall);

results_finall=transpose(results);

best_model=sortrows(results_finall,14);

[Miljan Kovacevic]

I forget to say, I want using validation set to prevent overfiting using early stopping method.

Answer 1

[Greg Heath]

Ranking a large number of correlated inputs for a NN is usually a thankless task.

Since the simpler the model, the better the good inputs show up, I tend to rely on models that are linear in their coefficients (e.g., polynomials)

>> lookfor stepwise

also

 >> help sequentialfs
 >> doc sequentialfs

Hope this helps.

Thank you for formally accepting my anwser

Greg