problem in preallocation

hi, I used data with huge size, so I did preallocation A(147611,30977)=zeros But , I got this message: ??? Maximum variable size allowed by the program is exceeded.

what i have to do in this case?

thanks in advance

Answer 1

[Titus Edelhofer]

Hi Huda,

before going on waht you can do other, you are aware, that a matrix of size 147611x30977 will need approx. 34GB of memory?

Titus

[huda nawaf]

thanks

what I have to do in this case?

what are alternatives?

[Titus Edelhofer]

Either use sparse matrices or process your data in chunks instead of one large matrix ... Other then this general advice would require some words where your huge data comes from and what you want to do with it ...

[huda nawaf]

my data is from social networks .

look

I opened folder with 1.92GB :

f=fopen('D:\social net\dataset\flixster\flixster\ratings.txt','r+')

this what cause the problem.

I just want to arrange my data in arrays.

these data regarding users in one of social networks.

I'm have not processed it yet.

okay regarding chunks , do u mean I have to splite my data each set is processed alone?

if so, what is sparse matrix?

please if have any information about it provide me.

thanks

[Titus Edelhofer]

Hi,

1.92GB is large, but still far away from 34. How did you get to the size 147611x30977? Are the numbers in the file only integers? If you read them from file, you don't need to preallocate (depending on how you read).

Titus

[huda nawaf]

in file there are integers and floating, but currently I read just integer.

I used just 1.92GB.

look please, in fact I convert that file into array

the structure of my file:

user id(integer) movie id(integer) rating(float)

in my file the id of users is frequent, so I try to place each user in row and draw his movie id and his rating .

So, the size of each row is different , and because i used preallocate i have to specify the size. I know the no. of rows but I don't know the no. of columns. So I give the largest(30977).

I'm on right or not?

please help me, I have in the same point since some days.

thanks

the array (147611*30977)

[Walter Roberson]

I do wonder whether the "social network"'s web site's Terms and Conditions allow this use of the data?

Anyhow, sounds to me like a good place to use cell arrays, or perhaps struct.