how to divide a data set randomly into training and testing data set?

Hello guys, I have a dataset of a matrix of size 399*6 type double and I want to divide it randomly into 2 subsets training and testing sets by using the cross-validation.

i have tried this code but did get what i want https://www.mathworks.com/help/stats/cvpartition-class.html

Could anyone help me to do that?

Expected outputs:

training_data: k*6 double

testing_data: l*6 double

[chocho]

Any help plz???

Answer 1

[KSSV]

Let A be your data of size 399*6. To divide data into training and testing with given percentage:

[m,n] = size(A) ;
P = 0.70 ;
idx = randperm(m)  ;
Training = A(idx(1:round(P*m)),:) ; 
Testing = A(idx(round(P*m)+1:end),:) ;

[chocho]

@KSSV i have tried your code but I want to divide it randomly by cross vaidation

[KSSV]

Edited the answer.

[chocho]

@KSSV now it works but there is a problem when I sum up the training and testing sets there is one extra row my current data of 359*7 and I got training of 251*7 and testing of 109*7, 251+109=360?

[KSSV]

Edited the answer....you need to add one.

[chocho]

Thanx

[Jaspalsingh Virdi]

Thanks it worked

[Timothy Tizhe Fidelis]

Thanks also. It work for me as well.

[Faiq Ahmad khan]

Thanks , it worked for me too.

[doaa khalil]

i have this error for this code can you help me pleas

Array formation and parentheses-style indexing with objects of class 'matlab.io.datastore.ImageDatastore' is not allowed. Use objects of class 'matlab.io.datastore.ImageDatastore' only as scalars or use a cell array.

there is my code

%Load Data

unzip('CropSet.zip');

imds = imageDatastore('CropSet', ...

'IncludeSubfolders',true, ...

'LabelSource','foldernames');

%Use countEachLabel to summarize the number of images per category.

tbl = countEachLabel(imds)

%Divide the data into training and validation data sets

rng('default') % For reproduciblity

[m,n] = size(imds) ;

P = 0.70 ;

idx = randperm(m) ;

imdsTrain= imds(idx(1:round(P*m)), :);

imdsValidation = imds(idx(round(P*m)+1:end),:);

[k priya]

Thanks a lot for your work.

[Aliyuda Ali]

Many thanks @KSSV. The code worked for me.

[MAT-Magic]

Dear KSSV,

For training, we take first 70% of the data. After applying your code, splitting is done, but why the values of data in say training are shuffled? Why these values in training set are not in same order which is in the given data? Thanks

[BANDARU UMAMADHURI]

What if we need a training data of 70% , testing and validation of 15% each,Can we use the same command used in testing for validation as well

[KSSV]

data = rand(100,3) ;  % your data   
%
[m,n] = size(data) ; 
idx = randperm(m) ;  % shuffle the rows 
Traing = data(1:round(m*0.70),:) ;  
Testing = data(round(m*0.70)+1:round(m*0.85),:) ;
Validation = data(round(m*0.85)+1:end,:) ;

[Susan]

@KSSV How the cross-validation is implemented here? Thanks

[KSSV]

[r,c] = size(A) ; 
P1 = 0.70 ; P2 = 0.85 ; 
idx = randperm(r) ; 

m = round(P1*r) ; n = round(P2*r) ;
Training = A(idx(1:m),:) ; 
Validation = A(idx(m+1:n),:) ; 
Testing = A(idx(n+1:end),:) ;

[Susan]

@KSSV Thanks for your response. I think I need your help to underestand the definition of cross-validation here. Does this cross-validation have nothing to do with something like k-fold cross validation?

[Saraswathi S]

Thank you so much

[Gentil Andres Collazos Escobar]

Thank you!!

[Abhijit Bhattacharjee]

If it hasn't been covered already, you can also use cvpartition to split the dataset. See THIS answer for more details.

Answer 2

[Jeremy Breytenbach]

Hi there.

If you have the Deep Learning toolbox, you can use the function dividerand: https://www.mathworks.com/help/deeplearning/ref/dividerand.html

[trainInd,valInd,testInd] = dividerand(Q,trainRatio,valRatio,testRatio) separates targets into three sets: training, validation, and testing.

[Koosha]

Thank you

Answer 3

[ALDO]

you can use The helper function 'helperRandomSplit', It performs the random split. helperRandomSplit accepts the desired split percentage for the training data and Data. The helperRandomSplit function outputs two data sets along with a set of labels for each. Each row of trainData and testData is an signal. Each element of trainLabels and testLabels contains the class label for the corresponding row of the data matrices.

percent_train = 70;

[trainData,testData,trainLabels,testLabels] = ...

helperRandomSplit(percent_train,Data);

make sure to have the proper toolbox to use it.

[Lucrezia Cester]

could you please send a link to this function?

Answer 4

[sidra ashiq]

Training = A(idx(1:round(P*m)),:) ;

what is the A function??

[Mohamed Marei]

A is the vector or array indexed by the elements inside the bracket. It is not a function.

[madhan ravi]

A is a matrix

Answer 5

[Mehernaz Savai]

You can partition data in a number of ways:

Let X be your input matrix. You can also use similar workflow for Tables.

If you have the Statistics and Machine Learning Toolbox, you can use cvpartition as follows:

% Partiion with 40% data as testing 
hpartition = cvpartition(size(X,1),'Holdout',0.4); 

% Extract indices for training and test 
trainId = training(hpartition);
testId = test(hpartition);

% Use Indices to parition the matrix  
trainData = X(trainId,:);
testData = X(testId,:);

If you have the Deep Learning Toolbox, you can use dividerand as follows:

% Partiion with 60:20:20 ratio for training,validation and testing
% respectively
[trainId,valId,testId] = dividerand(size(X,1),0.6,0.2,0.2);

% Use Indices to parition the matrix  
trainData = X(trainId,:);
valData = X(valInd,:);
testData = X(testId,:);

Answer 6

[Pramod Hullole]

hello sir,

iI'm new to the neuralnetworks..now i am working on my projects which is leaf disease detections using image processing. i am done with feature extraction and now not getting what is the next step..i know that i should apply nn and divide it in training and testing data set.. but in practically how to procced that's what i am not getting .please help me through this... please send steps..each steps in details. .

[Savas Yaguzluk]

Dear Pramod,

Open a new topic and ask your question there. So, people can see your topic title and help you.

Answer 7

[Hossein Amini]

Hi there, it worked for me but I have problem in rest of the code. In newrb doc, it has been witten how to write the code but the more tried that I did, I got error like below.

Answer 8

[Hossein Amini]

[z,r] = size(X);

idx = randperm(z);
TrainX = (X(idx(1:round(Ptrain.*z)),:))';
TrainY = (Y(idx(1:round(Ptrain.*z)),:))';
TestX = (X(idx(round(Ptrain.*z)+1:end),:))';
TestY = (Y(idx(round(Ptrain.*z)+1:end),:))';

If I'm not mistaken, in newrb doc, the size of input data and output data should be same like (4x266 and 1x266), that's why I transposed that matrixes. But the error which I got is specifying zeros matrix. I don't know how to prepare that.

Answer 9

[ranjana roy chowdhury]

the dataset is WS Dream dataset with 339*5825.The entries have values between 0 and 0.1,few entries are -1.I want to make 96% of this dataset 0 excluding the entries having -1 in dataset.