How to handle a huge amount of data with surface and stair plots to avoid memory spikes

When we try to handle very large amount of the data with batched bins (500) and averages (50) and stored in the matrices. It still consumes and have memory spikes when the ploting scenario reaches 20 minutes long.

Besides adjusting the bin and average sizes, is there a way to make this more efficient to plot a large timeframe of data by splicing multiple plots?

In both plots, X is referring to time with calcuated magnitude spectrum in dB.

[Jan]

The question is not clear. Please explain, what you are doing exactly: What is displayed how? Code? What are "memory spikes"? What takes 20 minutes? How large is "large"? What is "splicing"? Which "X" refers to what?

[Ivy Chen]

Sorry about the confusion and here are some clarifications.

• File size - Data collected in 20 seconds results about 265 MB
• 20 minutes referring to the raw data collected in the bin file during the 20 minutes timeframe. Ideally, we would like to process at least data collect up to 1 hour.
• We notice that the memory spikes when the plot opens.
• X is referring to time vs calcuated magnitude spectrum in dB
• Splicing meaning that we can have multiple plots respect to time (X) axis. For example, plot 1: time 0-20 minutes, plot 2: time 21-40 minutes, plot 3: time 41-60 minutes, etc.

Here is the code:

Determine RF data size based on defined batches (NsperBatch=500 in this case)

magSpectrumMAT_Sum=[];
magSpectrumMAT_Avg=[];                                                          
while ~feof(fid) 
    magSpectrum=0;
    for k=1:Kavg                                                             
        data=fread(fid,NsperBatch*2,'int16');
        dataIQ = complex(data(1:2:end,:), data(2:2:end,:));
        num_dataIQ = length(dataIQ);
        dataIQ_GnOff = (Gain * dataIQ) + (Offset+Offset*1i);                    % Apply mx+b for Gain and Offset specified in the URT User Manaul
        dataIQ_GnOff = dataIQ_GnOff * db2mag(Reference_L);                      % Normalize with respect to the reference level as indicated in the header (eg. -1.445 dbm = -31.445 dbw)  

        target_num_dataIQ = NsperBatch;
        if num_dataIQ ~= target_num_dataIQ
          fprintf('Note: complex data is not a multiple of %d samples long, padding\n', NsperBatch);
          dataIQ_GnOff(target_num_dataIQ) = 0;  % Fill zero automatically                              
        end;

        mag_dataIQ_GnOff = abs(fftshift( fft( reshape(dataIQ_GnOff, NsperBatch, []) ) ) / sqrt(NsperBatch)).^2;   % Compute RF data batched Matrix in FFT, then ABS
        magSpectrum = magSpectrum + mag_dataIQ_GnOff;
    end
    magSpectrumMAT_Sum = [magSpectrumMAT_Sum magSpectrum];    
    magSpectrum_avg = magSpectrum/Kavg;
    magSpectrumMAT_Avg = [magSpectrumMAT_Avg magSpectrum_avg];
end
magSpectrumMat_dB=pow2db(magSpectrumMAT_Avg);                                   %%

Define Frequncy Range for Batch Load

num_full_batches = size(magSpectrumMAT_Avg, 2);
dt = (NsperBatch*Kavg)/BW;                                                                    
tvec = ((1:num_full_batches)-1)*dt;
df=BW/NsperBatch;
freqvec=(-BW/2):df:(BW/2-df);

Plot 1: Spectrum Power

X=tvec;
Y=freqvec/1e6;
Z=magSpectrumMat_dB;
[X, Y]=meshgrid(X,Y);
h=surf(X,Y,Z);
set(h,'EdgeColor','none');
grid off
axis tight
view([90 90]);   %Set view angle as Top-Flat 2D plot                                                     
colormap(jet);
%%Plot 2: Sum Power
Power_Sum=sum(magSpectrumMAT_Sum)/Kavg;               
CPower_W=Power_Sum/NsperBatch;                   
CPower_dBW=10*log10(CPower_W);              
dt_sum = (NsperBatch*Kavg)/BW;
X_sum=((1:num_full_batches)-1)*dt_sum;
h_sum=stairs(X_sum,CPower_dBW);