Mapreduce with tall array in matrix multiplication

Hi,

I am processing huge dataset (billions rows), so I used tall array in my code. While I found the tall array are not support general matrix multiplication. The '*' only works in a limited conditions.

I am wondering if we can combine mapreduce to perform matrix multiplication with tall arrays ? Below is a example shows the error.

A = tall(ones(200,2));
B = tall(ones(2,3));
values = A*B; 
gather(values)

Error using tall/mtimes>iVerifyAtLeastOneScalar

Matrix multiplication of two tall arrays requires one of them to be scalar.

Learn more about errors encountered during GATHER.

Error in * (line 31)

[X,Y] = iVerifyAtLeastOneScalar(X,Y,"MATLAB:bigdata:array:MtimesBothTall");

Answer 1

[Torsten]

Would this work for you ?

B = ones(2,3);
A = tall(ones(200,2));
values = A*B;
gather(values)

Evaluating tall expression using the Local MATLAB Session:
- Pass 1 of 1: 0% complete
- Pass 1 of 1: Completed in 0.21 sec
Evaluation completed in 0.37 sec

ans = 200x3

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

[chen zhe]

Unfortunately no. Actually, A and B ate both tall arrays obtained by some processings.

Your method is equivalent to gather（B），then multiply A and B. Finally need to gather（values） again. （2 gather（） in total）