Handling Big Data in MatLab

4 ビュー (過去 30 日間)
Buddhi Wimarshana
Buddhi Wimarshana 2021 年 1 月 22 日
コメント済み: Buddhi Wimarshana 2021 年 1 月 24 日
Hi everyone,
I am working with large sized data and the followings are the steps I follow, but need your advice/help to correct the procedure? or make it better...
I receive the data as csv files (data is of mixed types like doubles, date-time ets.) and the whole data set spans across about 150 csv files each having more than 100MB of data. So, I combined them all into one matlab table so the table contains the whole data set. I can't (don't like to really) make matlab tables for each csv file, becuase of the continuity of the data across each csv file so I guess it would be harder to retrieve them later. A part of the table I created looks like below:
This Table is more than 2GB in size (26 columns and millions of raws). Then I saved this Table as a mat file so I can 'load' it back to workspace later and retrieve the necessary data segments as per requirements. For example, I may need to retrive and post-process data of 'CurrentA' and 'VoltageV' for Cycle_Index=1 & Step_Index=30.
My concerens are:
  1. this method is quite time taking, for example loading the mat file takes quite long, so wonder is there a better way? I saw using 'matfiles' we don't need to load the mat file to memory. But I am not sure this will allow me to retrieve data with much flexibility compared to original file loading to workspace method?
  2. not so sure about the total methodology I followed, is there a better way to handle these kind of large data in a better, efficient way ?
Many thanks in advance...

採用された回答

Steven Lord
Steven Lord 2021 年 1 月 22 日
Some of the tools and techniques described in the Large Files and Big Data category in the documentation may be of interest to you.
  1 件のコメント
Buddhi Wimarshana
Buddhi Wimarshana 2021 年 1 月 24 日
Thnk you so much Steven, went through this method before, then had a problem so went with the method I mentioned. I think I got more understanding on datastores now, seems it's the way to go!!

サインインしてコメントする。

その他の回答 (0 件)

カテゴリ

Help Center および File ExchangeWorkspace Variables and MAT-Files についてさらに検索

製品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by