Large CSV table (2 dimension) imports into a single column

I have a large csv file (100001 x 1000), with no headers. When I import using either csvread or dlmread matlab produces a variable that is 100001000 x 1.

I have tried on multiple machines and same thing happens.

CSV file size is confirmed using script: awk -F, 'END {printf "Number of Rows : %s\nNumber of Columns = %s\n", NR, NF}' file.csv

Edit: if I read the csv file, reshape it to 100001x1000, write it to a new csv file, and then read the new file, the new file, when imported to matlab, is a single column vecotr.

[Andy L]

Can you provide the code you have used?

[Andrew]

Sure.

CSV is read and sized (in matlab) with

model_output = csvread('PEEEmodel9/Outputs/100000MODEL_OUTPUT.csv'); size(model_output)

reshape with

output_reshaped = reshape(model_output,100001,1000);

[per isakson]

It might have something to do with the exceptionally long lines. However, I cannot find in the documentation.

[Andy L]

No me neither! I cannot recreate it either, thanks to the memory limitation of my machine.

[dpb]

Which release, which OS?

As another says, can't produce as large a file here for memory constraints but no issue (other than takes a long time to write a big formatted file) with as large as cared to go w/ the re-read file being same dimension as written given my limited patience to wait.

If this is reproducible, I'd suggest halving down the size until it no longer happens (easy enough to do w/ just random data in a loop) and then working back until it does to see if can isolate the magic number/size.

Then, if that doesn't provide enough clues as to what's happening to solve the problem for some other reason, submit an official bug report to TMW at www.mathworks.com

[Image Analyst]

You forgot to attach the file! Please chop off all but 10 lines or so and attach the file with the paper clip icon so people can test code with your actual data.

[Andrew]

Thanks for your responses. I received an answer from elsewhere which has worked. I simply used the load command rather than a reader. Not elegant, but does the job!

The first 10 lines of data file is attached if anyone is interested in getting to the bottom of why csvread refused to work properly. The same problem occurred with this file.

[dpb]

... I simply used the load command rather than a reader.

If there's nothing otherwise peculiar in the file, it needs to be submitted as a bug report as noted above. One presumes there must be something unique in either newline and/or the delimiter to cause the problem but until actually attach the file can't tell anything more.

I'd wonder why you wouldn't use a stream or other much more compact form for file transfer for such large file sizes, however. Performance would have to go up greatly it would seem.