Matching multiple file names

Hi

I have two types of data each in a seperate folder. Lets say I have magnetic field files in folder x and electric field files in folder y.

There are 365 magnetic field files each represnting one day of measurments. However electric field files are more than 365 and there are multiple files per day. For example, we have electric field files measured from 00:00:00 to 10:30:00 in one file for a day and then 11:00:00 to 23:59:59 in another file for the same day. Note that for some days there are more than two electric field files.

I want MATLAB to open magetic field file

"SW_OPER_MAGA_HR_1B_20140101T000000_20140101T235959_0505_MDR_MAG_HR_processed.mat" from it's folder

then open electric field files

"SW_EXPT_EFIA_TCT16_20140101T000000_20140101T103000_0302_processed.mat"

and

"SW_EXPT_EFIA_TCT16_20140101T110000_20140101T235959_0302_processed.mat"

which belong to the same day as magnetic field file as shown by data timestamp in file names. I was not succeful in using regexp and dir functions.

Thanks in advance,

Pouya.

Answer 1

[Chris]

You'll have to fill in some of the blanks because I don't know how you're opening files, but to find the corresponding filenames in the electric field folder...

magstr = "SW_OPER_MAGA_HR_1B_20140101T000000_20140101T235959_0505_MDR_MAG_HR_processed.mat";

pat = "_" + digitsPattern(8) + "T";
thedates = extract(magstr,pat);

% eFieldNames = {dir().name};
eFieldNames = {'SW_EXPT_EFIA_TCT16_20140101T000000_20140101T103000_0302_processed.mat',
    'SW_EXPT_EFIA_TCT16_20140102T110000_20140102T235959_0302_processed.mat',
    'SW_EXPT_EFIA_TCT16_20140101T110000_20140101T235959_0302_processed.mat'};
filesToOpen = contains(eFieldFnames,thedates(1))

filesToOpen = 3×1 logical array
   1
   0
   1

[Pouya]

Hi Chris,

Thank you for a quick response.

filesToOpen succefully recognizes the files corresponding to the selected magnetic field file. How can I use the logical array (fielsToOpen) to actually open those chosen files?

By the way here is the small block of code that I have so far for your refrence:

clc
clear


path = 'C:\Users\pouya\OneDrive\Desktop\SW\Plasma\Processed\2014';
 addpath 'C:\Users\pouya\OneDrive\Desktop\SW\Mag\Processed\2014\Alpha dB_e';


magstr = "SW_OPER_MAGA_HR_1B_20140101T000000_20140101T235959_0505_MDR_MAG_HR_processed.mat";
pat = "_" + digitsPattern(8) + "T";
thedates = extract(magstr,pat);
 eFieldNames = {dir(path).name};
% eFieldFnames = {'SW_EXPT_EFIA_TCT16_20140101T000000_20140101T103000_0302_processed.mat',
%     'SW_EXPT_EFIA_TCT16_20140102T110000_20140102T235959_0302_processed.mat',
%     'SW_EXPT_EFIA_TCT16_20140101T110000_20140101T235959_0302_processed.mat'};
filesToOpen = contains(eFieldNames,thedates(1));

Regards,

Pouya.

[Chris]

I would probably cd(path), to get to the appropriate directory. I would also amend one of the previous commands:

filesToOpen = find(contains(eFieldNames,thedates(1)));

cd(path)
for idx = 1:numel(filesToOpen)
    % May vary depending on how you open the file...
    temp = readmatrix(eFieldNames(filesToOpen{idx}));
    % Put the data somewhere more useful
    allEfields{idx} = temp;
end

[Stephen23]

Do not use CD to access data files. Using CD is slow (it forces MATLAB to rescan the directories for MATLAB files), makes debugging harder, and is superfluous.

All MATLAB function that import/export data files accept absolute/relative filenames. Using relative/absolute filenames is easy and efficient. You should use absolute/relative filenames, for which FULLFILE is very useful.

[Pouya]

Chris, Stefan,

Both return "Brace indexing is not supported for variables of this type." (when I replace cd with fullfile too)

[Chris]

@Pouya Pourkarim,

Is there more to the error message?

Does the fully constructed path appear correct? You might put a breakpoint inside the for loop and inspect the filename you are trying to open/import.

Additionally, I would rename your "path" variable, as path() is a Matlab function that overlaps with the things you are trying to do here.

[Pouya]

I can add that it refers to temp in the for loop. Basically:

Brace indexing is not supported for variables of this type.

Error in analysisstarttest (line 24)

temp = readmatrix(eFieldNames(filesToOpen{idx}));

(analysisstarttest is my script's name)

[Chris]

I suspect you're missing a backslash.

You can either insert a breakpoint, as I suggested, or display the filename before trying to open it:

disp("File Name: " + eFieldNames(filesToOpen{idx}))

[Pouya]

Chris,

Thank you for continous help.

Unfortunatly now I am getting the same error but this time for disp

clc
clear


path = 'C:\Users\pouya\OneDrive\Desktop\SW\Plasma\Processed\2014\';
 addpath ('C:\Users\pouya\OneDrive\Desktop\SW\Mag\Processed\2014\Alpha dB_e');


magstr = "SW_OPER_MAGA_HR_1B_20140102T000000_20140102T235959_0505_MDR_MAG_HR_processed.mat";
pat = "_" + digitsPattern(8) + "T";
thedates = extract(magstr,pat);
 eFieldNames = {dir(path).name};

filesToOpen = find(contains(eFieldNames,thedates(1)));

% f=fullfile('processed','2014','filesToOpen');

cd(path)
for idx = 1:numel(filesToOpen)
    % May vary depending on how you open the file...
    disp("File Name: " + eFieldNames(filesToOpen{idx}));
    
%     temp = readmatrix(eFieldNames(filesToOpen{idx}));
%     % Put the data somewhere more useful
%     allEfields{idx} = temp;
end

Which leads to:

Brace indexing is not supported for variables of this type.

Error in analysisstarttest (line 24)

disp("File Name: " + eFieldNames(filesToOpen{idx}));

[Pouya]

I'm suspecting the problem is originating from eFieldNames = {dir(path).name};

eFieldNames has two elements at it's start that are "." and ".."

[Chris]

filesToOpen = find(contains(eFieldNames,thedates(1)));

will not indicate those two files, because the pattern you are looking for is not found there. The smallest index possible in filesToOpen should be 3.

[Chris]

@Pouya Pourkarim I just looked back at your most recent code. When you perform the dir() command, apparently you cannot have the trailing backslash.

I would reconstruct the full filenames like so:

pathn = 'C:\Users\pouya\OneDrive\Desktop\SW\Plasma\Processed\2014';
for (...)
        % filesep is a Matlab command that generates the appropriate
        % separator for your OS
        % None of these are strings, so do char concatenation with []
        f = [pathn, filesep, eFieldNames(filesToOpen{idx})];
        disp("File Name: " + f)
end

[Pouya]

Hi Chris

Unfortunatly the same error still presists and this time is

Brace indexing is not supported for variables of this type.

Error in analysisstarttest (line 24)

f=[path, filesep, eFieldNames(filesToOpen{idx})];

Let's think about it in a different way. I'm thinking I can do this process for mag files that have multiple E files manually and let the code do it only for mag files that have only one corresponding E files. If you come up with ideas I will be grateful for the help but I don't want to consume too much of your time now.

Regards,

Pouya.

[Image Analyst]

@Pouya Pourkarim filesToOpen is a linear vector - a regular array that needs parentheses, not a cell array that needs braces. So whenever you see

filesToOpen{idx} % Won't work.

replace it with

filesToOpen(idx) % Should work.

Do NOT use path as the name of a variable since that's an important built in variable. Call it "folder" or something other than path.

Don't do this

f=[path, filesep, eFieldNames(filesToOpen{idx})];

Use fullfile() instead. And use descriptive variable names, otherwise your program will become an alphabet soup mess of cryptics one and two letter variables. And eFieldNames is a cell array so you need braces:

fullFileName = fullfile(folder, eFieldNames{filesToOpen(index)});

See the FAQ for a good discussion on where to use braces, brackets, or parentheses.

https://matlab.fandom.com/wiki/FAQ#What_is_a_cell_array.3F

[Pouya]

Thanks, it actually solved the problem with the Brace indexing error that I was getting.