csvwrite does not store data completely!

Hello,

I am generating data in MATLAB by storing them in separate .csv files using csvwrite(), each having the same number of rows and columns. The weird thing is that it nicely stores some of the data, however, for some others it stores data incompletely, meaning that the number of rows/columns does not match with what it should be! seems that the excel file is corrupted!

Note that, I store 3000 excel files and say 2995 of them are nicely stored, but just some of them are corrupted and I do not know what is the issue!

Does anyone experience something like this?

for i=1:1:3000
    ...
        
filename1 =sprintf('DATA1_%d.csv', i); 
csvwrite(filename1, x)
filename2 = sprintf('DATA2_%d.csv', i);
csvwrite(filename2, y)

end

[Walter Roberson]

How much data is being written? Is it possible that you are near one million rows?

[shiva moshtagh]

3000 .csv files, each having 4K rows and 200 columns.

As I try this process several times, I realized that the corruption happens randomely. It is not specific to a type of data or happens in a specific row.

[Walter Roberson]

Do you happen to be writing to a network file system such as OneDrive?

[shiva moshtagh]

Yes, I store the data in a network drive. Is that the issue?

[Walter Roberson]

It certainly could be the problem.

[shiva moshtagh]

Thanks so much!

[Walter Roberson]

By which I mean that there are reports of problems writing to network drives, especially OneDrive. When space permits it is faster and safer to write to a local drive and then copy the results to the network drive.

Answer 1

[Image Analyst]

Is it always the same array that is being written out short? What happens if you recall it right away to test.

for i=1:1:3000
    ...
        
    filename1 =sprintf('DATA1_%d.csv', i); 
    fprintf('Writing %s with %d rows and %d columns.\n', filename1, size(x, 1), size(x, 2));
    csvwrite(filename1, x)
    data = csvread(filename1);
    fprintf('    Recalled %s with %d rows and %d columns.\n', filename1, size(data, 1), size(data, 2));
    if ~isequal(size(data), size(x))
        message = sprintf('ERROR: File %s was written out with %d rows and %d columns but was recalled with only %d rows and %d columns!',...
            filename1, size(x, 1), size(x, 2), size(data, 1), size(data, 2))
        uiwait(errordlg(message));
    end
    
    filename2 = sprintf('DATA2_%d.csv', i);
    fprintf('Writing %s with %d rows and %d columns.\n', filename2, size(y, 1), size(y, 2));
    csvwrite(filename2, y)
    data = csvread(filename2);
    fprintf('    Recalled %s with %d rows and %d columns.\n', filename2, size(data, 1), size(data, 2));
    if ~isequal(size(data), size(x))
        message = sprintf('ERROR: File %s was written out with %d rows and %d columns but was recalled with only %d rows and %d columns!',...
            filename2, size(y, 1), size(y, 2), size(data, 1), size(data, 2))
        uiwait(errordlg(message));
    end
    
end

[shiva moshtagh]

The corruption happens randomely! it is not specific to a type of data.

Sometimes it does not even happen!

[Image Analyst]

The code I gave will detect if it happened. Did you even try it? If you did and you did not get the error message, then open in Excel the file that's bad and see how many columns it has, then look up in the list of all files and sizes that my code wrote out and see what my code said the size should have been.

[shiva moshtagh]

Thank you!

[Image Analyst]

You're welcome. But what did you observe? Was the recalled data different than the saved data for any of the files? If not, what are you basing the fact that some files are only partially saved on?

[Walter Roberson]

Network drive was likely the problem.