Time-Frequency Feature Embedding with Deep Metric Learning

I have a piece of code on MathWorks documentation. The code for classification EEG using supervised contrastive learning. It starts by building a model responsible for generating embeddings (TFnet) for row data. The point I'm asking about is abou the process of calculating the test accuracy as mentioned in the code below. First, we generate embeddings from the test dataset (by passing the test raw data through TFnet), and then it uses those embeddings to train an SVM and calculate accuracy of the model. Is it acceptable to train a model and test it using same data. Could you help clarify it?

Time-Frequency Feature Embedding with Deep Metric Learning

finalEmbeddingsTable = helperEmbedTestFeatures(TFnet,testDataPS,testLabelsPS);
template = templateSVM('KernelFunction', 'gaussian', 'PolynomialOrder', 'KernelScale', 4, ...
                        'BoxConstraint', 1, 'Standardize', true);
classificationSVM = fitcecoc(finalEmbeddingsTable,  "EEGClass", 'Learners', template, 'Coding', 'onevsone');
predLabelsFinal = predict(classificationSVM,finalEmbeddingsTable);
testAccuracyFinal = sum(predLabelsFinal == testLabelsPS)/numel(testLabelsPS)*100

Answer 1

[Jaimin]

Hi @Ahmed

Training and testing a model on the same data can result in overfitting, where the model excels on familiar data but fails to generalize to new, unseen data, making this approach generally inadvisable for evaluating a machine learning model's performance.

Using the same dataset for both training and testing the SVM may not accurately reflect its performance on new data. To prevent this, consider employing techniques such as:

Train-Test Split: Divide the dataset into separate training and testing sets. Train the SVM on the training set and evaluate its performance on the testing set.

Cross-Validation: Use k-fold cross-validation to assess the model's performance. This involves dividing the data into k subsets, training the model on k-1 subsets, and validating it on the remaining subset. This process is repeated k times, with each subset used exactly once as the validation data.

For more information kindly refer following MathWorks documentation.

https://www.mathworks.com/discovery/cross-validation.html

I hope this will be helpful.