how to obtain the data from the middle of the binary file using fread in MATLAB?

I have a binary file with ~90 GB data with only single dimension. I would like to make a line spectra of the dataset in several chunks.

How to ask matlab to take the binary file ranging from (1:3e6) , ((3e6)+1:(3e6)*2) , and so on ... until the end of the file.

I've used:

fileID = fopen('Binarydata');
data1 = fread(fileID,[1:3e8],'uint16'); 
fclose(fileID);
plot(data1)

fileID = fopen('Binarydata');
data2 = fread(fileID,[(3e8)+1:(3e8)*2],'uint16'); 
fclose(fileID);
hold on; plot(data2)

But it throws error!

Error using fread
Invalid size.

Any help is appreciated

Answer 1

[Les Beckham]

The second argument to the fread function is the size of the data to read, not an index, as you appear to be trying to use it.

So, your code should change to something like the code below. Note the change to *uint16 in the precision argument of fread, so that Matlab won't convert the data to double, which is the default. This will use one fourth of the memory that would be used without the *.

Note also that I'm re-using the data array so you aren't keeping the old data in Matlab's memory when you read the second chunk of data.

Adapt as needed.

fileID = fopen('Binarydata');
data = fread(fileID, 3e8, '*uint16'); % read 300 million uint16 values as uint16 rather than doubles
% fclose(fileID); << you don't need to close the file if you are going to read from it again
plot(data)

% fileID = fopen('Binarydata'); << you don't need to re-open the file if you don't close it
data = fread(fileID, 3e8,'*uint16'); % read the next 300 million uint16 values
fclose(fileID);
hold on; plot(data)

[Walter Roberson]

See also memmapfile

Answer 2

[Cris LaPierre]

The input to fread is the size of the output (i.e. the number of bytes to read), not a range of bytes. Try this.

fileID = fopen('Binarydata');
% read the first 3e8 bytes
data1 = fread(fileID,3e8,'uint16'); 
plot(data1)

% read the next 3e8 bytes
data2 = fread(fileID,3e8,'uint16'); 
hold on; 
plot(data2)
hold off

fclose(fileID);

[madhan ravi]

what's the difference between with *uint16 and uint16? I was not able to understand the difference from documentation

[Dyuman Joshi]

From the documentation -

For uint16, the input values are of the class uint16, whereas the output matrix is of class double.

For *uint16, the input values and the output matrix are of the class uint16.

[Walter Roberson]

The default when you specify a precision is that data is read in as that precision but converted to double automatically. But if you put * in front of the precision then data is read in as that precision and is kept in that precision instead of being converted to double.

So 'uint16' means to read unsigned 16 bit integers and convert them to double and return the doubles. But '*uint16' means to read unsigned 16 bit integers and leave them as unsigned 16 bit integers.

You can also use the '=>' specification to indicate what you want to be converted to. 'uint16' is equivalent to 'uint16=>double' and '*uint16' is equivalent to 'uint16=>uint16'.

[madhan ravi]

Thank you sir Walter!