How to make datenum more efficient for large arrays?

I am using textscan() to create an matrix that is 3,081,680 rows long. The very first column, A{1}, is the timestamp column, and it's elements look like this:

"2011-01-23 18:38:30"

I have tried using datenum() to convert this, but it is either taking too long or not working at all (longer than ten minutes). Is there a way I can make this more efficient?

Thanks!

Answer 1

[Cedric]

What is the purpose ultimately? Do you need an accurate time stamp which accounts for the date and time? If you were computing daily averages for 2011 only, for example, we could find a way to generate month/day IDs probably faster than DATENUM.

Otherwise, there are FEX contributions, e.g. this one by Jan Simon: http://www.mathworks.com/matlabcentral/fileexchange/28093-datestr2num

as well as a few relevant threads, among which: http://undocumentedmatlab.com/blog/datenum-performance/

[Marian]

The data I have is a text file containing meteorological data over a series of months. For this script in particular, I need to average one-minute data and output it into a new text file. The datenum has been the only issue I'm having.

[Marian]

Sorry, I meant to add ... one-minute data for December 2011 only. So this text file contains multiple months over two years, and I will need to have the correct year, month, day, hour, minute, AND second. I will look into the links you posted, thank you.

[Cedric]

Ok, Kelley's suggestion is the first thing to try; if it's not fast enough, Jan's function on the FEX is certainly the second. I am unsure how you are binning the data then with the date serial number; if I had to do it, I would probably use DATEVEC, build a minuteID using year, month, day, hour, minute, and then use ACCUMARRAY to compute the average per minuteID.

[Marian]

Ah, datevec! I have been teaching myself MATLAB and did not know of this function, but I believe that I needed datevec to read arrays, as datenum or datestr2num would not work for type "cell". Thanks so much!

[Cedric]

You're welcome. Let me know if you go for an ACCUMARRAY approach for binning and need help with that.

Answer 2

[Kelly Kearney]

Are you specifying the date format? Particularly for large arrays,

datenum(A{1}, 'yyyy-mm-dd HH:MM:SS')

will be much faster than

datenum(A{1})

[Cedric]

+1 this is definitely the first thing to try.

[Marian]

Error using datenum (line 179) DATENUM failed.

Error in onemin (line 9) time=datenum(A{1}, 'yyyy-mm-dd HH:MM:SS');

Caused by: Error using dtstr2dtnummx Failed on converting date string to date number.

This is the error I get using that, and I believe it is because of the double quotations around it.

[Cedric]

Use the following format: '"yyyy-mm-dd HH:MM:SS"'