File Exchange

image thumbnail

restore_idl

version 1.14.0.0 (6.43 MB) by Charles Pelizzari
Imports variables (scalar, string, array, structure) from IDL save files into Matlab workspace.

9 Downloads

Updated 27 Sep 2015

View License

Data exchange between Matlab and IDL is a longstanding challenge. Exporting to an intermediate format (CSV, HDF, etc) often does the job, but reading the native archive files of each language into the other would be much simpler.
This is a Matlab GUI that allows restoring variables from an IDL save file. Variables are restored into the Matlab base workspace. Optionally changes names of created variables to lowercase.
Operation:

Click "Choose IDL File" - use file browser to identify the IDL save file. Variables in the IDL file will be listed indicating their data type (scalar, array, structure, uint8, float32, etc); and for arrays, their dimensions.

Select one or more variables in the display listbox (shift-click, control-click, etc. for multiple selection)

Choose "Convert names to lowercase" if desired. Variables are saved with names (and structure field names) in ALL CAPS so if you find that annoying, click the handy checkbox (actually it's checked by default because I find it annoying).

Click "Restore Selected" - chosen variables will be created in the base Matlab workspace.

Capabilities:
Can process all basic IDL numeric, array, string and structure data types. Does NOT restore object references. Does NOT understand ulong64 offsets, thus cannot process files >4GB.

Structures with fields that are arrays or other structures are handled via recursion. This structures within structures feature has not been thoroughly tested - only to one level of nesting. Known to work properly with save files from IDL version 8.1. Since the file format description used to develop the code is several years old (see link below), probably will work with files from earlier versions as well. Does NOT handle compressed files, i.e. the "/COMPRESS" switch in the SAVE command cannot have been used.

Non-GUI version is a function "restore_idl" that takes as input arguments a full pathname for the save file (provides a browser if no valid file specified), and a couple other options: "lowercase" and "verbose". Returns all the variables it can understand from the file in an array of structs with two fields: a string "name", and a cell array "value" which contains whatever the variable happens to be.
Based on description of IDL save file format by Craig Markwardt:

http://www.physics.wisc.edu/~craigm/idl/savefmt/

C. Pelizzari, University of Chicago, October 2013

Comments and Ratings (32)

Fabio VArgas

Thank you very much Mr. Pelizzari. This was a time saver. I am grateful.

New version April 2018 - see "Restore IDL Save Files." The system won't let me update this one since my email changed.

Antonio

Antonio (view profile)

You saved my life, thanks! Great work :)

Ooooh, this just made my life so much easier!

Mohammad

Sun

Sun (view profile)

Great job!

Daniel,

"invalid size" would indicate either the variable trying to be restored is too big (could it be > 2GB?) or else the header parsing has gotten out of sync and read in a nonsense value for the size of what is supposed to be read next. possibly your file has some new quirk that hasn't been encountered before - wouldn't be the first time. If you contact me offline (should be able to do this through my profile here) maybe we can arrange to get me a copy of your file to figure it out. it could also be helpful if you check the "verbose" checkbox before restoring, and capture the output.

have you tried the command line version of the program as well as the GUI? They should be the same, but there may have been a fix that got into one and not the other.

-cp

Daniel Smith

Hi,
I have been using this to restore variables from an IDL save file. I have been able to do this for nearly all the variables in the file which mainly contains 1x1 structure arrays. I am able to restore these 1x1 structure arrays but I have some larger structure arrays (up to 2719x1) which I can not restore. I get the following error when I try to extract these.

Error using fread
Invalid size.

Error in restore_idl_gui>read_structure_descriptor (line 532)
strucdesc.tagtable(n).name=...

Error in restore_idl_gui>read_idl_array (line 718)
if isempty(strucdesc), strucdesc = read_structure_descriptor(fid,0);
end

Error in restore_idl_gui>RestoreButton_Callback (line 419)
thevar=read_idl_array(fid,arrdesc,typecode,dolower);

Error in gui_mainfcn (line 95)
feval(varargin{:});

Error in restore_idl_gui (line 89)
gui_mainfcn(gui_State, varargin{:});

Error in
@(hObject,eventdata)restore_idl_gui('RestoreButton_Callback',hObject,eventdata,guidata(hObject))


Error while evaluating uicontrol Callback

Which I know is an error another user has had but I couldn't find the solution. Does anyone know the cause of this error?

Hugo B

Hugo B (view profile)

Leda, I added a demo that restores each of the IDL save files included in the download, and creates a Matlab variable from each IDL variable. You can just copy the code from that to create your variables. Also added directions to the header in restore_idl.m

I still don't understand the source of the GUI error you and Hugo encountered.

Leda Sox

This is going to be a total lifesaver. Thank you for creating this. I am an IDL user that's trying to switch to Matlab, so Matlab is still unfamiliar territory.

I had the same error come up as Hugo B when I tried to use the gui. I then tried to use the non-gui function and it seemed to work. It spit out the name and size of the variable I was expecting it to. So now, and this is probably just has to do with my unfamiliarity with Matlab, how do I actually use the variable?

I tried printing one column from my array and it says "undefined variable".

Ale

Ale (view profile)

Even better then! Thanks again!

what you suggest does make more sense. made it the default.

Ale

Ale (view profile)

Hi Charles, thanks a lot! I did a quick check and it works seamlessly for a number of rather structured sav files I deal with. That's fantastic! I foresee many other users will take advantage from this tool. It will definitely make a difference for me.

Technical question: what is the rationale for separating names and values in the output, instead of defining fields with the actual variable names? Double naming?

Ale, your wish is granted. New version uploaded with a separate non-GUI function, and also a cosmetic change to the GUI. Thanks for the suggestion.

Ale

Ale (view profile)

Charles, thanks for considering my suggestion. Yes, I imagined that you might have started with a non-GUI function :)

Ale, as you can imagine this started out as a couple of non-GUI functions. There have been some important fixes as a result of the feedback here, but a non-GUI version with the current improvements seems like a good idea. Will see what I can do. I think I would like it be able to import only specified variables by name as an option, and maybe a couple other features. Maybe that's for a future version...

Hugo, I'm unable to reproduce your error with the current download version freshly installed on a Win7 system (64 bit, unlike yours - I doubt that is a problem).

Be sure you're using the latest version of both the .m and .fig files, the .fig file was just updated with a cosmetic change.

The error seems to have to do with trying to access the "verbose" checkbox to get its value. You could try replacing the offending statement

verbose = get(handles.VerboseCheckbox,'Value');

with
verbose=0;
or
verbose=1;

and see what happens. you can't just comment it out because the variable "verbose" needs to have a value. if that works, then you might need to replace several other occurrences of the same statement, as the state of the verbose checkbox is queried in a couple places.

Ale

Ale (view profile)

Very nice piece of code. Thanks a lot!

I find it extremely useful, being a Matlab user just adsorbed in a traditionally IDL work environment. So far I didn't encounter any problem, but a lot of relief :)

I have one wish/request. Would it be possible to include a version without the GUI? I'm thinking of a function that, for example, loads all the detected variables into an output structure. That would allow loading data from IDL files in Matlab codes that shouldn't require user interaction.
I could go ahead and try and de-GUI the code myself, but you might already have developed something like this :)

In any case, thank you so much for sharing this
amazing tool.

Hugo B

Hugo B (view profile)

OK thanks!

I'll install the download package on a Win7 system and see if I can reproduce this error.

Hugo B

Hugo B (view profile)

Hi Charles,

Thank you for this amazing contribution.

However, I'm having trouble getting the UI to work. Whenever I click "Choose IDL File" to open the file browser, I get the error listed below:

------
Struct contents reference from a non-struct array object.

Error in restore_idl_gui>FileSelectButton_Callback (line 146)
verbose = get(handles.VerboseCheckbox,'Value');

Error in gui_mainfcn (line 95)
feval(varargin{:});

Error in restore_idl_gui (line 89)
gui_mainfcn(gui_State, varargin{:});

Error in
matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)restore_idl_gui('FileSelectButton_Callback',hObject,eventdata,guidata(hObject))
Error while evaluating UIControl Callback
-----

What did I do wrong ? FYI I'm working on Win7 32bits

Sachin Dev

You are right, the file has many variables. How do I send you my file? I can email if I know the address and if it is OK with you.

what kind of variable is selected? is there more than one variable in the file? do any of them restore correctly? the best way to work this out is for you to let me see your file.

Sachin Dev

After hitting 'Restore Selected' getting following error messages:

Error using fread
Invalid size.

Error in restore_idl_gui>read_idl_array (line 624)
thevar=cast(fread(fid,lenagain,'uint8',0,'b'),mytype);

Error in restore_idl_gui>read_idl_structure_new (line 581)
thisfield=read_idl_array(fid,strucdesc.arrtable(narr),...

Error in restore_idl_gui>read_idl_array (line 652)
thevar{1}=read_idl_structure_new(fid,strucdesc,1,dolower);

Error in restore_idl_gui>RestoreButton_Callback (line 389)
thevar=read_idl_array(fid,arrdesc,typecode,dolower);

Error in gui_mainfcn (line 96)
feval(varargin{:});

Error in restore_idl_gui (line 79)
gui_mainfcn(gui_State, varargin{:});

Error in
@(hObject,eventdata)restore_idl_gui('RestoreButton_Callback',hObject,eventdata,guidata(hObject))


Error while evaluating uicontrol Callback

Your file ON2_2008_234m.sav is compressed, and this code doesn't know how to handle that (nor will it learn). If you have the ability to save without the /COMPRESS switch, it can read the file. I read it into IDL, saved to another file without compression and it does restore into Matlab. There are two scalars and two arrays of floats, plus an array of 15 structures which do get populated correctly.

D

D (view profile)

Not working for me. Please check with the following file

http://guvi.jhuapl.edu/levels/level3/guvi_on2/data/IDLsave/2008/ON2_2008_234m.sav

should be fixed now.

I'll try to reproduce the problem, but it might be most effective if I can get one of your save files to try out. Probably something simple in processing start and end codes in the array elements, or improper handling of byte alignment for odd length strings.

Misla

Misla (view profile)

This works for me completely fine when I select variables that are "single array" or "int32 array". But not for "string array".

I have tried with two different IDL savefiles. It does not give an error message in the command window, it simply does nothing.

trazz

trazz (view profile)

Got everything working now, and it's absolutely perfect!
Best thing I've ever gotten from the file exchange.

trazz

trazz (view profile)

What a wonderful idea!
Unfortunately, I'm having trouble getting the UI to work. Whenever I click "Choose IDL File" to open the file browser, I get the error listed below:

------
Attempt to reference field of non-structure array.

Error in restore_idl_gui>FileSelectButton_Callback (line 107)
verbose = get(handles.VerboseCheckbox,'Value');

Error in gui_mainfcn (line 96)
feval(varargin{:});

Error in restore_idl_gui (line 67)
gui_mainfcn(gui_State, varargin{:});

Error in
@(hObject,eventdata)restore_idl_gui('FileSelectButton_Callback',hObject,eventdata,guidata(hObject))


Error while evaluating uicontrol Callback
-----

I'm not sure if this is just a problem with the UI, but if so, is there a way to use this program just through the workspace/command line?

Updates

1.14.0.0

Added demo which restores all files in the distribution directory, and creates local Matlab variables to illustrate how.

1.13.0.0

changed output format of callable function to single struct whose field names are IDL variable names.

1.12.0.0

Added callable function version "restore_idl" in addition to GUI version "restore_idl_gui" that loads all variables in the save file into an array of structs.

1.11.0.0

17 Sept 2015: Restored missing checkbox background color

1.10.0.0

April 2015: fixed logic for structures containing arrays of structures. Added restore debug output option.

1.9.0.0

print warning and bail out if file seems to be compressed

1.8.0.0

Corrected logic for arrays of structures and processing of variables which contain null strings

1.7.0.0

Feb 2015: add code for string array processing; fix logic for zero length strings if found in save file.

1.6.0.0

31 Jan 2014 - fixed problem with improper handling of array start codes within structures that caused incorrect results for structure containing multiple arrays.

1.5.0.0

Save path of previously chosen file as default for next one.

1.2.0.0

Adjusted GUI font sizes to play well on both Mac and Windows

MATLAB Release Compatibility
Created with R2011b
Compatible with any release
Platform Compatibility
Windows macOS Linux
Tags Add Tags

Discover Live Editor

Create scripts with code, output, and formatted text in a single executable document.


Learn About Live Editor