Import data and text in huge csv files into matlab and convert to matfiles

Hello All,

Struggling with converting my csv files to matfiles. xlsread and csvread works great with small files but with bigger ones my matlab freezes forever. The file that I need to convert has about 2000 rows and 1200 columns. The first row contains text and the remaining rows contain data. Ultimately I want to be able write the data in each column into the text in the first row of the corresponding column. I also tried using xlsread1 that was shared online but I keep getting errors. I am using Matlab R2010a. Please help if you guys know how to do it.

[per isakson]

"remaining rows contain data" &nbsp does that mean pure numerical data, i.e. no datetime?

Answer 1

[Robert Cumming]

Use low level routines to read the file.

 fopen
 fgetl
 strread or textscan

Using this you are in complete control.

Answer 2

[Image Analyst]

Honestly, that's not huge. Even at double precision, that's only 19.2 megabytes - much smaller than a run of the mill digital photo. I deal with images that are like 20 GB - a thousand times larger than yours. Maybe try dlmread(). If that or fread() or fgetl(), textscan, sscanf(), etc. don't work then I'll look into it.

Answer 3

[per isakson]

Here are two alternative sets of code to read your file. That is, if "remaining rows contain data" &nbsp means pure numerical data, i.e. no text like datetime.

In this case textscan is twice as fast as dmlread

    >> cssm
    Elapsed time is 1.972024 seconds.
    Elapsed time is 4.248113 seconds.
    ans =
         1
    M(1:3,1:5)
    ans =
         1     2     3     4     5
         1     2     3     4     5
         1     2     3     4     5

where cssm is the script below

    hdr = repmat( 'colhead,', [1,1200] );
    str = sprintf( '%.1f,',   [1:1200] );
    str(end) = [];
    %%write a sample file
    fid = fopen( 'cssm.txt', 'w' );
    fprintf( fid, '%s\n', hdr );
    for jj = 1 : 2000
        fprintf( fid, '%s\n', str );
    end
    fclose( fid );
    tic
    fid = fopen( 'cssm.txt', 'r' );
    rw1 = fgetl( fid );
    num = textscan( fid, '%f', inf, 'Delimiter', ',\n', 'CollectOutput', true );
    fclose( fid );
    Out = transpose( reshape( num{:}, [1200,2000] ) );
    toc
    tic
    M = dlmread( 'cssm.txt', ',', 1,0 );
    toc
    all(M(:)==Out(:))
    M(1:3,1:5)