How do I access an Invalid-named variable from an exported MAT file that MATLAB will not recognize?

Hi!

I am having a problem resolving how to read into MATLAB some exported data that I obtained from another program, which I have no control in how it names its files.

The acquisition software allows for a MATLAB export, and it contains two MATLAB files, one a time series, and the other the output data. Unfortunately, the output data contains an illegal MATLAB variable symbol (#) and cannot be read.

Is there anyway to change the illegal variable name to make it a valid one?

I copied and pasted below for convenience exactly the describe problem above from my MATLAB screen:

>> load 01_ExportData >> whos -file 01_ExportData Name Size Bytes Class Attributes

c001_Time                 1x1200000            9600000  double              
c002_Analog_Input_#4      1x1200000            9600000  double

>> c002_Analog_Input_#4 ??? c002_Analog_Input_#4 | Error: The input character is not valid in MATLAB statements or expressions.

>>

How would I change the c002_Analog_Input_#4 variable name to access it? Or just get around this problem! Remember, I cannot control the inclusion of this # symbol, since I have no say it out the program from which I exported my data names its files.

Thanks a bunch in advance!!! Jess Li

Answer 1

[Richard Thiessen]

I had the same problem.

Things that don't work

Matlab itself refuses to load data from a .mat file if the variable name involved is invalid.

There are multiple ways to call the load() function

load("myfile.mat")
load("myfile.mat","variable name with spaces")
%obviously fails since the variable names in the file are invalid
%and can't be created in the workspace.
struct=load("myfile.mat")
%structs can't contain invalid variable names either
valid_varname=load("myfile.mat","variable name with spaces")
%this fails too, Matlab validates the name before retrieving it
%from the file

None of them work of course. Even if there's nothing technically stopping the last command from working Matlab still refuses to touch invalid variable names.

I tried using the LOADFIXNAMES library but that did not work for me. After installing the compiler (MinGW-w64) the extension compiled but crashed matlab as soon as I called it with my files.

What did work

Anaconda (A python programming language bundle) includes the scipy library. The scipy.io library has the loadmat() and savemat() functions which read and write matlab data files. Loadmat produces a data structure containing the (name,value) pairs for all variables in the file. The names can be changed and the new (name,value) pairs put into a similar structure and fed to savemat to produce a fixed output file.

Here's the code I used: (Note that this is python code not matlab code)

import scipy.io as sio
import pathlib
import string

def fix_name(name):
    name="".join(c for c in name if c in string.ascii_letters+string.digits+"_")
    if name[0] not in string.ascii_letters:#name must start with a letter
        name="X"+name
    name=name[:63]#names may not be longer than 63 chars
    return name

ignore_names=["__header__","__version__","__globals__"]

def fix_struct(s):
    out_struct=dict()
    for name,value in s.items():
        if name not in ignore_names:
            out_struct[fix_name(name)]=value
    return out_struct

def struct_needs_fixing(s):
    for name in s.keys():
        if name not in ignore_names:
            if fix_name(name)!=name:
                return True
    return False

root=r"C:\someplace\folder_with_the_files"
root=pathlib.Path(root)
files=list(root.glob("*.mat"))#get all the .mat data files in the directory

Overwrite=False #overwrite files in place

for f in files:
    new_f=f.parent/("fixed_"+f.name)#save the modified file with a prefix
    if Overwrite:new_f=f
    struct=sio.loadmat(str(f))
    if struct_needs_fixing(struct):
        print("fixing",str(f))
        struct=fix_struct(struct)
        sio.savemat(new_f,struct)
    else:
        print("skipping",str(f))

This removes all unallowed chars and prefixes names with "X" if they don't already start with a letter.

Anaconda itself takes about 20 minutes to install. The code above can be run from jupyter notebook which should be available in the start menu after installing anaconda. The file path can be copied from the windows file explorer (right-click on file)-->properties-->(popup window)-->location The above script will fix all the files in a given folder.

[S.R. Prakash]

This solution worked like a charm with my python3+numpy+scipy setup.

Thanks a bunch!!

Answer 2

[Jan]

In Matlab 2009a this C-Mex file can copy the value of Variables from the workspace:

#include "mex.h"
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
  char *Name;
  mxArray *Var;

    Name = mxArrayToString(prhs[0]);
    Var  = mexGetVariable("caller", Name);
    if (Var != NULL) {
       plhs[0] = mxDuplicateArray(Var);
    } else {
       plhs[0] = mxCreateDoubleMatrix(0, 0, mxREAL);
    }
    mxFree(Name);
    return;
  }

EDITED: "mxReal"->"mxREAL" Thanks Matt!

I've stored this as CopyData.c, compiled it and a call like

A = CopyData('c002_Analog_Input_#4')

gets the value of the variable with the ugly name. By the way: I've created the ugly variable using:

mexPutVariable("caller", "c002_Analog_Input_#4", Var)

Good luck, Jan

[Matt Fig]

Jan, that worked well. However I did have to change mxReal to mxREAL to get it to compile.

[Jan]

"Worked well"?! If you thought EVAL is evil, how do you like mexPutVariable? I'd accept negative reputation points, if the answer works... Thanks for the considerate correction!

[Walter Roberson]

mexPutVariable is no worse than assignin()

[Jan]

Then try to ASSIGNIN a variable called '_ugly', '#' or ' '.

But to be exact, you are right: ASSIGNIN is worse, because users really use it for *allowed* ugly actions, while mexPutVariable is seldom used to perform *forbidden* actions, although it has the power to.

Answer 3

[James Tursa]

You can try this FEX submission:

http://www.mathworks.com/matlabcentral/fileexchange/42274-loadfixnames-loads-a-mat-file-fixing-invalid-names

[James Tursa]

Unfortunately, TMW has changed the API to disallow mex routines from reading and fixing invalid names, so this submission is now obsolete.

Answer 4

[Matt Fig]

Give this a try. What is the extension of the data file?

X = load('01_ExportData')  % May need the extension
D = struct2cell(X)

If that doesn't work, have a look at this thread:

http://www.mathworks.com/matlabcentral/newsreader/view_thread/257989

[Walter Roberson]

I agree, a small modification to Jame's mex code should work for this.

Answer 5

[Jan]

Dynamic field names must not be valid Matlab symbols. So instead of calling a C-Mex, you can use this:

X = load('01_ExportData', '-mat')
Value = X.('c002_Analog_Input_#4');

Even the empty string is working! See: http://www.mathworks.com/matlabcentral/newsreader/view_thread/292602

[Joel Duncan]

What about this situation I'm in?

X = load('PresTrace.15.mat')
Value = X.('c002_Analog_Input_#4')
% Error using load
% Invalid field name: '15PresTrace'

[Walter Roberson]

Unfortunately in a later release the above option was disallowed.

You could try https://www.mathworks.com/matlabcentral/fileexchange/42274-loadfixnames-loads-a-mat-file-fixing-invalid-names

Answer 6

[Jessica]

Hi. First thank you for the response. I tried the above, but here is what I get when I do this:

>> x = load('01_ExportData' , '-mat') ??? Error using ==> load Invalid field name: 'c002_Analog_Input_#4'.

>>

I cannot even get to the next line of code to try. MATLAB just reports the invalid field name and will not recognize x as a variable to continue with the next line.

Argh! I'm not exactly a newbie with MATLAB but not an expert either, but this has had even a MATLAB TA I know stumped!

Am I doing something wrong here with the above, or is this some strange bottleneck with the pound-sign that cannot be resolved. I can use the original program that I exported for the analysis if all else fails, but using MATLAB for the stats is just far more superior!

Thank you again in advance for your help!

-Jess

[Jan]

Which Matlab version do you use? As far as I can see Matlab has stopped the support of bad field names from LOAD after Matlab > 2009a. What a pitty. Do you have access to an <= 2009a version to convert the file?

Does load('01_ExportData.mat', 'c002_Analog_Input_#4') work?

Answer 7

[Jessica]

Hi. Yes, load('01_ExportData.mat', 'c002_Analog_Input_#4') does work. And I am using >2009a. But the funny thing is that even though I can load it with the load version you gave, it will not even recognize c002_Analog_Input_#4.

[Walter Roberson]

Perhaps if you combined this with Jan's earlier suggestion, to form:

X = load('01_ExportData.mat', 'c002_Analog_Input_#4');
X.('c002_Analog_Input_#4')

Answer 8

[Jessica]

Hi Walter. Thanks again (and to everyone else). I am afraid that none of the code works unfortunately to fix the bad field name. I am a bit disappointed that MATLAB has no easy way to remedy this (which would seem to be an obvious feature to have) and puzzled that they would have gotten rid of loading bad field names after 2009a. At least you could have implemented some of the above code to change the variable name more readily.

Well, I guess my only alternative is to use the cruddy analysis routines in the main program I exported the data out from the beginning. :(

Thanks a bunch anyhows guys! I appreciate the posts. -Jess :)

[Jan]

Don't give up. Try the Mex approach using mexGetVariable at first.

[Matt Fig]

Jan, I emailed Jessica and told her to look at your MEX function. She is doing so...

[Jan]

Thanks Matt! I'm still confused about the Answers mechanisms. In opposite to CSSM personal emails are helpful here sometimes.