Saving many variables as NetCDF makes file too large. How to reduce?

Hello all,

I would REALLY appreciate any help on this, as I looked online but didn't find straight answers. I am trying to save many variables of different sizes as NetCDF format in this way:

numrow = 576;

numcol = 48;

numlayers = 125;

numtime_save = numtime + 1;     %This is usually a value of '8'

%%Define the dimensions

netcdf.setDefaultFormat('NC_FORMAT_CLASSIC') ; 

ncid = netcdf.create(files_name,'NC_WRITE');

dimidrow = netcdf.defDim(ncid,'rows',numrow);

dimidcol = netcdf.defDim(ncid,'length',numcol);

dimidlay = netcdf.defDim(ncid,'layers',numlayers);

dimidtime = netcdf.defDim(ncid,'time',numtime_save);

%%Define the name of variables, with dimensions
%%Question: do these all really have to be double precision???

varmask = netcdf.defVar(ncid,'cloudmask','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_sigma = netcdf.defVar(ncid,'cloudmask_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_clear = netcdf.defVar(ncid,'cloudmask_clear','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_clear_sigma = netcdf.defVar(ncid,'cloudmask_clear_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_radar = netcdf.defVar(ncid,'cloudmask_radar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_radar_sigma = netcdf.defVar(ncid,'cloudmask_radar_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_both = netcdf.defVar(ncid,'cloudmask_both','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_both_sigma = netcdf.defVar(ncid,'cloudmask_both_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varmask_lidar = netcdf.defVar(ncid,'cloudmask_lidar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_lidar_sigma = netcdf.defVar(ncid,'cloudmask_lidar_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varheight = netcdf.defVar(ncid,'height','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varheight_sigma = netcdf.defVar(ncid,'height_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

varN = netcdf.defVar(ncid,'N','NC_BYTE',[dimidrow dimidcol dimidtime]);

%%end the variable naming and dimensions, then actually input
%%variables into netcdf file, then close the netcdf file

netcdf.endDef(ncid);

netcdf.putVar(ncid,varmask,cloudmask);

netcdf.putVar(ncid,varmask_sigma,cloudmask_sigma);

netcdf.putVar(ncid,varmask_clear,cloudmask_clear);

netcdf.putVar(ncid,varmask_clear_sigma,cloudmask_clear_sigma);

netcdf.putVar(ncid,varmask_radar,cloudmask_radar);

netcdf.putVar(ncid,varmask_radar_sigma,cloudmask_radar_sigma);

netcdf.putVar(ncid,varmask_both,cloudmask_both);

netcdf.putVar(ncid,varmask_both_sigma,cloudmask_both_sigma);

netcdf.putVar(ncid,varmask_lidar,cloudmask_lidar);

netcdf.putVar(ncid,varmask_lidar_sigma,cloudmask_lidar_sigma);

netcdf.putVar(ncid,varheight,height);

netcdf.putVar(ncid,varheight_sigma,height_sigma);

netcdf.putVar(ncid,varN,N);

netcdf.close(ncid);

clear mex

However, when I save it this way, the file size is 1.33 GB! That is huge...If I save the files as:

save(filename, 'cloudmask','cloudmask_sigma','cloudmask_clear','cloudmask_clear_sigma',...
    'cloudmask_radar','cloudmask_radar_sigma','cloudmask_both','cloudmask_both_sigma',...
    'cloudmask_lidar','cloudmask_lidar_sigma','height','height_sigma','N');

this file has a size of 16 MB. I would REALLY like to save in NetCDF format, because my boss needs it this way. I have many files to save, this is just an example but it is inside of a loop. PLEASE HELP!

[Walter Roberson]

netcdf does not use compression unless it is told to.

https://www.unidata.ucar.edu/blogs/developer/entry/netcdf_compression

[Macarena]

Thanks! So, I would have to compress using the terminal? I am not so comfortable doing that, but I guess I can try. I'll give the article a detailed-read.

[Walter Roberson]

https://www.mathworks.com/help/matlab/ref/netcdf.defvardeflate.html

[Macarena]

This worked! Thank you! I did this:

example:

comp = 5;      %compression size

varmask_radar = netcdf.defVar(ncid,'cloudmask_radar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

netcdf.defVarDeflate(ncid,varmask_radar,true,true,comp);

on all the variables, and now my file has been reduced to 6.3MB!!

[Walter Roberson]

You should post that as an Answer so we can vote for it. (You did the work of putting together the answer that would be of use to other people.)

Answer 1

[Macarena]

%Example for one variable. Have to do on all

numrow = 576;

numcol = 48;

numlayers = 125;

numtime_save = numtime + 1; %This is usually a value of '8'

comp = 5; %compression size, for example

%%Define the dimensions

netcdf.setDefaultFormat('NC_FORMAT_CLASSIC') ;

ncid = netcdf.create(files_name,'NC_WRITE');

dimidrow = netcdf.defDim(ncid,'rows',numrow);

dimidcol = netcdf.defDim(ncid,'length',numcol);

dimidlay = netcdf.defDim(ncid,'layers',numlayers);

dimidtime = netcdf.defDim(ncid,'time',numtime_save);

varmask_radar = netcdf.defVar(ncid,'cloudmask_radar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);

netcdf.defVarDeflate(ncid,varmask_radar,true,true,comp);

netcdf.endDef(ncid);

netcdf.putVar(ncid,varmask_radar_sigma,cloudmask_radar_sigma);

netcdf.close(ncid);

clear mex