Dynamically read all files in a file process them and combine the processed data in one matrix

Hi, I have a folder with various datasets, I want to read all the datasets and process them. Each datasets have 4 columns (A, B, C , D),

each column represent a signal that I want to process, in this case to filter and calculate the rms value of the signal. The final result should be a table with columns A,B,C,D and row(i) should have the rms values of the signas(i), i.e the rms values of the first dataset . My main problem is that when using reshape, in the second iteration, I am getting:

Error using reshape

Number of elements must not change. Use [] as one of the size inputs to automatically calculate the appropriate size for that dimension.

files = dir('*.csv');

num_files = length(files);

results = cell(length(files), 1);

for i = 1:num_files
  results{i} = readmatrix(files(i).name);
end
% 
% results{2}

features = []
results_k = results{1}
for i = 1:num_files
   results_k = results{i}
   len = length(results_k)
   results_k = results_k(~isnan(results_k))';
   %len = len/4
   results_k = reshape(results_k,len,4)
   firstCol = results_k(:,1)
   secondCol = results_k(:,2)
   thirdCol = results_k(:,3)
   fourthCol = results_k(:,4)
   bpFilt = designfilt('bandpassiir','FilterOrder',4, ...
         'HalfPowerFrequency1',0.08,'HalfPowerFrequency2',1, ...
         'SampleRate',20,'DesignMethod','butter');

   firstCol = filtfilt(bpFilt,firstCol);
   secondCol = filtfilt(bpFilt,secondCol);
   thirdCol = filtfilt(bpFilt,thirdCol);
   fourthCol = filtfilt(bpFilt,fourthCol);

   rms1 = rms(firstCol)
   rms2 = rms(secondCol)
   rms3 = rms(thirdCol)
   rms4 = rms(fourthCol)

   new_features=[rms1,rms2,rms3,rms4]

   features = [features new_features]

end

[Stephen23]

I very much doubt that these lines will work:

len = length(results_k)
results_k = results_k(~isnan(results_k))';
results_k = reshape(results_k,len,4)

Assuming that results_k has four columns (which seems to match your code) then you are telling MATLAB to remove some (unknown number of) NaNs from results_k and then reshape the resulting vector back into a matrix with exactly the same number of elements as results_k had as before you removed the NaN elements.

How do you expect to remove the NaN elements and yet still have exactly the same number of elements?

Use of LENGTH is also a possible cause of an error, as are the format of the file and various other things. But because you did not upload a sample data file or write any code comments I cannot suggest how to fix this.

[Mathieu NOE]

hello

my 2 cents

It may be worth considering using fillmissing to remove NaN values and replace them with interpolated values

[Dexter]

Hi, I resolved the problem, Stephen you were right I should use length after the NaN removal.

Now I end up with a list with M values that contains the rms values of all the signals. I want to create a table or array from this list. More specifically, I want the first 4 values to be the first row, the 4 values after the second row etc.. Does anyone have any idea how to do it?

I tried reshape but doesnot work like I want to.

[Stephen23]

"I tried reshape but doesnot work like I want to. "

I am guessing that you need to RESHAPE to have four rows, and then TRANSPOSE.

[Dexter]

That worked, thank you.