How can i speed up the following for loop?

Hi everyone,

I have a large .bin file. I read the .bin file and swap the bytes by modulo 4 and write to txt file. For example consider the .bin files starts with bytes.

D0 40 03 BC 00 FB 03 00 04 01 00 02 0F 00 C1 38 00 39 00 0F 00 0F 00 EE FF BB CC DD ...

After I swap the bytes I get

BC 03 40 D0 00 03 FB 00 02 00 01 04 38 C1 00 0F 0F 00 39 00 EE 00 0F 00 DD CC BB FF and so on.

After writing to a txt file I get

BC 03 40 D0

00 03 FB 00

02 00 01 04

38 C1 00 0F

0F 00 39 00

EE 00 0F 00

DD CC BB FF

Here is my code:

FID_Huz_file        = fopen(Huz_file_path);
Huz_file_Data_total = fread(FID_Huz_file);
Huz_file_Data       = Huz_file_Data_total(1:huz_file_lenght); -- lenght is calculated somewhere before this line
Huz_file_hex        = dec2hex(Huz_file_Data);

Huz_data_text_dir   = fullfile(Huz_data_text);
Huz_data_text_fid   = fopen(Huz_data_text_dir,'w');
[row, ~] = size(Huz_file_hex);
tic;
for i = 1:row/4
    temp_array = Huz_file_hex((4*(i-1)+1):4*i,:);
    
    for j = 1:4
        temp_array_valid (j,:) =  temp_array(5-j,:);
    end
    
    for j = 1:4
        temp_array_tp((2*j-1:2*j)) = temp_array_valid(j,:);
    end
    
    fprintf(Huz_data_text_fid,'%s\n',temp_array_tp);
end
toc;
fclose(Huz_data_text_fid);

The problem is when Huz_file_Data is very big (greater then 40 MB) it takes to much to create txt file. Here are some performance statisitics for several .bin files.

Bin file length : 54525952 Byte

huz_file_lenght : 5112320 Byte

Elapsed time is 16.610973 seconds.

--------------------------------------------------

Bin file length : 54525952 Byte

huz_file_lenght : 10224124 Byte

Elapsed time is 16.610973 seconds.

--------------------------------------------------

Bin file length : 54525952 Byte

huz_file_lenght : 20447744 Byte

Elapsed time is 58.783637 seconds

--------------------------------------------------

Bin file length : 54525952 Byte

huz_file_lenght : 40894976 Byte

Elapsed time is 127.972875 seconds.

Answer 1

[Deepak]

Hi @Ümit Güden,

To improve the performance of byte-swapping and file-writing operations for large .bin files, we can leverage vectorized operations to optimize performance.

Instead of iterating over each row to swap bytes, reshape the data into a matrix with four columns and use matrix indexing to swap bytes efficiently. This approach minimizes loop overhead.

Additionally, accumulate the swapped data into a buffer and write it to the text file in larger chunks to reduce the frequency of file I/O operations, which can be a significant bottleneck.

Below is the MATLAB code to achieve the same:

% Open the input .bin file for reading
FID_Huz_file = fopen(Huz_file_path, 'r');
Huz_file_Data_total = fread(FID_Huz_file, 'uint8');
fclose(FID_Huz_file);

% Limit the data to the specified length
Huz_file_Data = Huz_file_Data_total(1:huz_file_lenght);

% Reshape the data into a matrix with 4 columns
numRows = floor(length(Huz_file_Data) / 4);
Huz_file_Data = Huz_file_Data(1:numRows * 4); % Trim excess bytes
reshaped_data = reshape(Huz_file_Data, 4, []).';

% Swap bytes using matrix indexing
swapped_data = reshaped_data(:, [4, 3, 2, 1]);

% Convert to hexadecimal
Huz_file_hex = dec2hex(swapped_data);

% Open the output text file for writing
Huz_data_text_fid = fopen(Huz_data_text_dir, 'w');

% Write the hex data to file
tic;
fprintf(Huz_data_text_fid, '%s %s %s %s\n', Huz_file_hex.');
toc;

fclose(Huz_data_text_fid);

Please find attached the documentation of functions used for reference:

reshape: www.mathworks.com/help/matlab/ref/double.reshape.html

dec2hex: www.mathworks.com/help/matlab/ref/dec2hex.html

fprintf: www.mathworks.com/help/matlab/ref/fprintf.html

I hope this will help in resolving the issue.

[Ümit Güden]

Thanks @Deepak