Feeding in large amounts of data from an online database

1 回表示 (過去 30 日間)
Muneer
Muneer 2014 年 2 月 6 日
コメント済み: Muneer 2014 年 2 月 11 日
Would appreciate someone pointing me in the right direction:
Currently I am using urlread to feed in data from a password protected database. I read in all the data at once as a block and then start breaking it up with textscan. Currently, I place a limit on the amount of data I'm pulling in. You can see it below (password removed):
url = 'http://assa104/eurofot/csv';
block = urlread(url, 'Authentication', 'Basic', 'Post', {'login', '***', 'passwd', '***', 'trips', '1332244409', 'signals', '1.AVL_BRTORQ_SUM.AVL_BRTORQ_SUM_DVCH', 'limit', '300000','content_type', 'text'});
but when I remove this limit, I get the following errors:
Error using urlreadwrite (line 97)
Error downloading URL. Your network connection may be down
or your proxy settings improperly configured.
Error in urlread (line 36)
[s,status] =
urlreadwrite(mfilename,catchErrors,url,varargin{:});
My question is two-fold: 1. Any idea why I am returning such an error? 2. This method takes a while (almost a minute with my limit at 300k) and I want to read in even more data in the future. Is there an alternative to my current method? Maybe one that doesn't save the whole set in a block at once but one that reads in rows at a time?
I'm not very experienced with urlread, so any assistance would be greatly appreciated.
Thanks in advance.
  1 件のコメント
Muneer
Muneer 2014 年 2 月 11 日
Here is the message for the exception. Hopefully this sheds some light on the problem.
Java exception occurred:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:99)
at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:76)

サインインしてコメントする。

回答 (0 件)

カテゴリ

Help Center および File ExchangeData Import and Export についてさらに検索

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by