In expressions of the form T.X = B where T is a tall table, B must be a tall array.

Hello,

I am trying to convert a tall array with a column of strings to numbers. I tried this:

ds = datastore(fullfile(p,'tallFinal.csv'));            % CREATE A DATASTORE
ds.SelectedFormats={'%q','%C','%C','%s'};               % DEFINE COLUMN FORMATS
ds.VariableNames = {'DATE','TICKER','FIELD','VALUE'};   % DEFINE COLUMN NAMES
tds = tall(ds);                                         % CREATE A TALL ARRAY
tmp = str2double(tds.VALUE);                            % CONVERT TO STRING
[tds.fVALUE]=gather(tmp);

And recieved an error after over two hours of processing.

Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 2: Completed in 10 min 50 sec
- Pass 2 of 2: Completed in 2 hr 11 min
Evaluation completed in 2 hr 22 min
In expressions of the form T.X = B where T is a tall table, B must be a tall array.

Thank You,

Michael

[Michael]

I'm trying this now, but since I have to wait about two and a half hours to see if it works, please respond if you have advice.

tmp = tall( str2double(tds.VALUE));

Thank You,

Michael

[Walter Roberson]

Is there a reason you are not just using

ds.SelectedFormats={'%q','%C','%C','%f'};

so that the VALUE field becomes numeric as the data is read in?

[Michael]

Dear Mr. Roberson,

When I try that I get an error.

Thank You,

Michael

>> ds = datastore(fullfile(p,'tallFinal.csv'));            % CREATE A DATASTORE
>> ds.SelectedFormats={'%q','%C','%C','%f'};               % DEFINE COLUMN FORMATS
>> ds.VariableNames = {'DATE','TICKER','FIELD','VALUE'};   % DEFINE COLUMN NAMES
>> tds = tall(ds);                                         % CREATE A TALL ARRAY

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 8).

>> x=max(tds.VALUE);
>> X=gather(x);

Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 2: 0% complete
Evaluation 0% complete
Error using matlab.io.datastore.TabularTextDatastore/readData (line 77)
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 6923, field number 4) ==> IG8     \n
Learn more about errors encountered during GATHER.

Error in matlab.io.datastore.TabularDatastore/read (line 120)
            [t, info] = readData(ds);

Error in tall/gather (line 50)
[varargout{:}, readFailureSummary] = iGather(varargin{:});

Caused by:
    Reading the variable name 'VALUE' using format '%f' from file: 'tallFinal.csv' starting at offset -15.

[Michael]

PS, There are some string data in the column causing the error, but it would not cause an error using str2double.

[dpb]

What is the offending data in the column? Attach an illustrative section of the file...

[Walter Roberson]

The error message above says there is a 'IG8 \n' in input.

[Michael]

Hello,

1) I attached some data.

2) tall(str2double(... didn't work.

3) I don't understand what you mean by "'IG8 \n"

ds = datastore(fullfile(p,'tallFinal.csv'));            % CREATE A DATASTORE
ds.SelectedFormats={'%q','%C','%C','%s'};               % DEFINE COLUMN FORMATS
    ds.VariableNames = {'DATE','TICKER','FIELD','VALUE'};   % DEFINE COLUMN NAMES
    tds = tall(ds);                                         % CREATE A TALL ARRAY
tmp = tall( str2double(tds.VALUE));                     % CONVERT TO NUMBER
[tds.fVALUE]=gather(tmp);
Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 2: Completed in 11 min 49 sec
- Pass 2 of 2: Completed in 2 hr 5 min
Evaluation completed in 2 hr 17 min
In expressions of the form T.X = B where T is a tall table, B must be a tall array.