Extract the certain rows from a tall array

7 ビュー (過去 30 日間)
Juan-Jie Sun
Juan-Jie Sun 2019 年 7 月 1 日
回答済み: Rajani Mishra 2019 年 7 月 19 日
I am working on a text file with 22 GB consisting of around 180 million rows and 17 columns. I want to extract the specific raws (e.g. 3rd to 100th row, and only the 2nd column) and put it in memory (i.e. gather) instead of leaving it as unevaluated data.
data='D:\...\SQLDataExport.txt';
ds = tabularTextDatastore(data);
t_array=tall(ds);
trip=gather(t_array(2:100,2:2));
This would take much longer than I expected. A wierd thing is that if I extract data from the beginning (e.g. 1st row to 100th row, and only the 2nd column). It takes much much shorter time to complete.
data='D:\...\SQLDataExport.txt';
ds = tabularTextDatastore(data);
t_array=tall(ds);
trip=gather(t_array(1:100,2:2));
From my understanding, they both put a matrix about 100 by 1 into memory. Why would the first take much much longer than the second?

回答 (1 件)

Rajani Mishra
Rajani Mishra 2019 年 7 月 19 日
Hi,
I think the difference between the time taken for rows 1:100 and 2:100 can be because for rows between 1 to 100, just first 100 rows are extracted. For rows between any N:M full numeric indexing algorithm is used. A lot more work is done as numeric indexing is geared up for tall index which in turn causing the delay.
You can try using head for speeding up, like this:
trip=gather(head(t_array(2:100,2:2),99));

カテゴリ

Help Center および File ExchangeBig Data Processing についてさらに検索

製品


リリース

R2018a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by