MATLAB Answers

0

An efficient (quick) way of entering complex data into Matlab workspace.

Yisrael Loecher さんによって質問されました 2019 年 5 月 13 日
最新アクティビティ James Tursa
さんによって コメントされました 2019 年 5 月 14 日
A HW device stores binary data on disk as an array of 32.5e6 unsigned short values which are then read into Matlab workspace, using either fread or memmapfile. The data is actually complex in nature, arranged as I/Q/I/Q/,,, Our goal is to convert the short values into a complex array so that we may perform FFT and other Matlab operations. The problem is that the conversions to single and then to complex are each very time consuming; please see below.
datasize = 32.5e6;
fileID = fopen('dataFile_uint16.bin', 'r');
temp = fread(fileID, dataSize, 'int16=>int16'); % read file as I & Q; memmapfile is faster...
fclose(fileID);
temp1 = single(temp); % Takes between 100-200 ms.
% NOTE: Matlab does not allow short
% data to be converted to complex
complexArray = temp1(1:2:dataSize) + 1i*temp1(2:2:dataSize); % takes ~550 ms
I have tried alternatives to the last line of code above such as complex(a,b) and reshape instead of indexing, but no real-time savings is noticed. Any suggestions?
A rigourous alternative is to write a mex-file which reads, converts, and returns the data to an mxComplexArray type. However, before enbarking on such a journey, I would like to know if the savings in real-time will justify the effort.
Thank you,
Yisrael Loecher

  0 件のコメント

サインイン to comment.

製品


リリース

R2017b

2 件の回答

James Tursa
回答者: James Tursa
2019 年 5 月 13 日
編集済み: James Tursa
2019 年 5 月 13 日
 採用された回答

Assuming you are using R2018a or later, you might try this FEX submission which can reinterpret real variables as interleaved complex variables without a deep copy:
This might save you some time in your second step, but of course won't save any time associated with that first step of converting the uint16 to single. But if the original data is stored as uint16 you are probably just going to have to live with the time it takes to convert it all to single.

  4 件のコメント

James Tursa
2019 年 5 月 14 日
"Assuming the source data was stored as single and using my wild imagination, could it possible to memory map the source data directly into Matlab in such a way that it would be compatible with an interleaved numerical complex array?"
Easy to read it directly in a mex routine. Assuming you know the size of the data in advance, inside the mex routine just mxCreateEtc the complex variable up front, then pass the pointer returned by mxGetComplexSingles to fread, and pass the resulting mxArray back to MATLAB via plhs[0]. Only one data copy involved throughout the whole process.
"Matlab uses column orientation as opposed to Fortran/C row orientation"
Actually, Fortran uses column orientation same as MATLAB. In fact, MATLAB was originally built around the linear algebra libraries written in the Fortran column order standard.
If your data is written in row order, then you will have to transpose it after reading it in.
Side Note: One would have thought that once MATLAB went to interleaved complex data, they would have updated their fread function at the same time with 'complexdouble', 'complexsingle', etc options, but I don't see any such thing in the doc. Maybe this would be an appropriate enhancement request.
Yisrael Loecher 2019 年 5 月 14 日
...inside the mex routine just mxCreateEtc the complex variable up front, then pass the pointer returned by mxGetComplexSingles to fread, and pass the resulting mxArray back to MATLAB via plhs[0].
I am assuming that since this is NOT a direct shared memory map but rather one copy need be performed, I need not worry about column/row array orientation differences. Also, while we at it, a cast from short to single can be performed on the fly, thereby saving disk space by keeping the source data at 16 bits.
...they would have updated their fread function at the same time with 'complexdouble', 'complexsingle', etc options, but I don't see any such thing in the doc. Maybe this would be an appropriate enhancement request.
May I add that the memmapfile.m function also should encorporate this enhancement request.
James Tursa
2019 年 5 月 14 日
"... I need not worry about column/row array orientation differences ..."
Well, whatever order was written to the file is the order you will get when you read it back in, unless you transpose on the fly as part of the read.
"... a cast from short to single can be performed on the fly ..."
Yes, you could also read 16-bit ints from the file and convert to single on the fly as well, thus having only one copy of the data as single in memory at one time.

サインイン to comment.


Jan
回答者: Jan
2019 年 5 月 13 日
編集済み: Jan
2019 年 5 月 13 日

What about avoiding the conversion from int16 to single by doing this on the fly:
temp = fread(fileID, dataSize, 'int16=>single');
Then maybe it is cheaper to import the real and imaginary data separately:
datasize = 32.5e6;
fileID = fopen('dataFile_uint16.bin', 'r');
realX = fread(fileID, dataSize, 'int16=>single', 2);
fseek(fid, 'bof', 2);
imagX = fread(fileID, dataSize, 'int16=>single', 2);
fclose(fileID);
X = realX + 1i * imagX;
Is this faster?
What about:
datasize = 32.5e6;
fileID = fopen('dataFile_uint16.bin', 'r');
tmp = fread(fileID, [2, datasize/2], 'int16=>single');
fclose(fid);
X = tmp * [1; 1i];

  4 件のコメント

Yisrael Loecher 2019 年 5 月 13 日
Jan,
Sorry-I didn't relate to your first suggestion above. The time killer is not the fread, but rather the conversion to single, and worst off, the conversion to complex. Therefore, splitting up the fread shouldn't really help. Your last comment is best.
P.S.
I altered the code according to your last suggestion and checked further with a few other permutations of where to place the cast from short to single. This is what I came up with, which is close to a 50% savings in real-time:
memMapPtr2 = memmapfile('dataFile_uint16.bin', 'Format', 'int16', 'Offset', 0);
temp = memMapPtr2.data;
temp = reshape(single(temp),2,[]);
temp4 = temp.' * [1; 1i]; % need to transpose here, since fread and
% memmap both output column-based arrays.
Thanks again.
Jan
2019 年 5 月 13 日
"Interleaved numerical arrays" sounds like the optimal solution, if you import the data in a Mex function.
Yisrael Loecher 2019 年 5 月 14 日
Jan,
I am only now aware of the change that went on under the Matlab hood in R2018a, concerning how complex data (I,Q) is stored in memory. James (below) seems to be an expert on what was before the change, and what is now after the change. I'll be looking into it...

サインイン to comment.



Translated by