Training Neural Network on Large Datasets

Hello, I have a huge dataset that I am training a feedforward neural network on. This causes out of memory errors during training (on a huge server) even when I switch from trainlm to trainrp.

I have been reading about the adapt function for incremental learning, but I don't think that it's the way to go as many people say it gives bad performance and it's more appropriate when the nature of the data changes over time.

One idea I had was breaking the training data into smaller portions and running the train() function on each smaller portion and passing the neural network back into it each time. However, I am concerned that train will lose the previous training and start training the network afresh on each new portion.

What is the best way to train this network on all this data?

[Greg Heath]

1. What is the purpose of the net?

2. Please quantify

 huge dataset

 huge server

 input dimensionality

 Output dimensionality

[hossam eldeen]

Dear Greg Heath I am having the same problem. huge training data to be used in training. and i am always out of memory. i am using 2 giga ram,the input vectors are 200*800000 and the output is 5*800000. what can i do ? the purpose is classification

[Greg Heath]

You have 5 classes. I assume the target columns are from eye(5).

The input dimensionality is 200. The first task is to reduce that dimensionality.

Typically, ~35 random samples per dimension are more than adequate to define each class. Accordingly, ~35,000 random samples should be adequate to define the 5 classes in a 200-D space.

At this point you might think about reducing the dimensionality using PLS (PCA may not work well for classification).

Although this is just a start, I expect this to point you in the right direction.

Greg

Let us know if you run into roadblocks.

Greg

Answer 1

[Greg Heath]

YOU ARE IN LUCK!!!

1. Randomly divide each of the c classes into s subsets.

2. Randomly combine one subset from each class and design s c-class neural net subclassifiers

3. Design a combiner that takes the outputs of the s subclassifiers and outputs a final class index.

I was the first to do this for U.S. Army radar target classification in the early 1980s.

Have fun!

Greg

[H.k]

Dear Greg, I'm a newer at training fitnet, and I've the same problem (2721*720 input and 1*720 output) please illustrate the above steps with an example for more declaration thanks in advance,,

[Greg Heath]

Dimensionality Reduction for classification

 help PLS

 doc PLS

Replace PLS with PCA if your task is regression and not classification.

Hope this helps.

Greg

[H.k]

I tried to divided data in my code but i still having a problem please mr,Grig take a look on the below code I don't know why it gives me the same output

   x=rand(720,2721);
  y=rand(720,1);
  net=fitnet(10);
  Nets=[];
  for i=1:720
  P=x(i,:)';
  T=y(i,:)';
  net=train(net,P,T)
  Nets=[Nets,{net}]
  end 
% at simulating results
h=[];
for i=1:720
net=Nets{i}
a=net(P)
h=[h,{a}]
end
%the variable h contains constant value

[Greg Heath]

This thread is a standard for confusion. Multiple authors on multiple topics (e.g., classification AND regression!)

1. You definitely should have started a new thread!

2. It is ridiculous to use an example that has both random inputs and random targets!!!

3. What you are doing makes no sense at all. You are training a net for each input???

4. Go back and look at the documentation examples

help fitnet doc fitnet.

5. Start a new question.

Hope this helps

Greg