Dealing with NaNs in a code

Hi everyone, I'm struggling with a code that doesn't accept the NaNs. This is a part of a bigger code:

data = [y1 y2];

for i=1:5
  c(:,i) = mean(data(:,i));
  d(:,i)=(data(:,i));
  data_demeaned(:,i) = d(:,i) - c(:,i);
end

I've tried to do this:

    c(:,i) = nanmean(data(:,i));
    d(:,i)= data(:,i)(~isnan(data(:,i)));/ d(:,i)=~isnan(data(:,i));
    data_demeaned(:,i) = d(:,i) - c(:,i);

but it doesn't work. Any suggestion would be highly appreciated.

Answer 1

[Star Strider]

I believe you’re making this more difficult than it needs to be. I do not understand what you are doing with your ‘d’ matrix, since the code doesn’t make sense, so please explain it.

Try this:

data = rand(1,30);                              % Create Data
data(randperm(30,10)) = NaN;                    % Create Data (Insert ‘NaN’ Values)
data = reshape(data, [], 5);                    % Create Data
c = nanmean(data);
data_demeaned = data - c;

[Nicu Sprincean]

The idea is that, in order to further run the code, it should ignore NaNs (the whole code works only if there is no missing data). The above code is for data management. So, I have a matrix with 5 columns - 5 variables (in excel) and I want to ignore the NaNs. The 'd' matrix is indeed for no use; it is just for the last line of the code: data_demeaned(:,i) = d(:,i) - c(:,i). I should mention that my code's version is 2008, that is, I'm using Matlab R2008b.

[Star Strider]

I don’t know what you’re doing, so I don’t know if interpolating the NaN values to eliminate the missing values are acceptable options. If they are, John D’Errico’s inpaint_nans (link) or related functions would be the way to go.

[Nicu Sprincean]

This is the starting code:

clear all

warning off all

clc

close all

[data]=xlsread('sample.xlsx');

y1 = data(:,1);

y2 = data(:,2:5);

y3 = data(:,6:9);

y4 = data(:,10:end);

alpha = 0.05;

k = 0.08;

res = call_fct(y1,y2,y3,y4,k,alpha);

After this, it calls the call_fct.

[Star Strider]

Please see my previous Comment.

I still don’t know what you’re doing, what ‘call_fct’ is or does, or whether interpolating the NaN values are appropriate.

[Nicu Sprincean]

This code computes some systemic risk measures ( http://www.runmycode.org/companion/view/175). I have already done interpolation and at this stage is not necessary. I just want the code to ignore the NaNs. Here's how the authors did only for 2 variables, but in Matlab 2015 (the first column refers to market portfolio return, and the second to an asset return):

 data = data(~isnan(data(:,2).*data(:,1)),:); % select only elements which are not NaN

[Star Strider]

I’m still lost as to what you’re doing.

If you want to eliminate the rows with NaN values from all your data, create a logical vector, then apply it to every row of every subset of your data:

Example —

data = rand(1,50);                              % Create Data
data(randperm(50,5)) = NaN;                     % Create Data (Insert ‘NaN’ Values)
data = reshape(data, [], 5);                    % Create Data (Matrix)
Lidx = ~any(isnan(data),2);                     % Logical Index: ‘0’ If ‘any’ Element In A Row Is ‘NaN’

Then use ‘Lidx’ as the row index in your data.

[Nicu Sprincean]

To be more specific, I want to compute variance-covariance matrix for a DCC MGARCH model, based on that 5 variables, where data(:,1) - first column - refers to market portfolio and the other 4 variables, data(:,2:5) - refer to return of 4 assets (banks). But the problem is that if there is any missing data, it fails to compute the matrix. That's why I need the code to select only the elements which are not NaN.

[Star Strider]

In R2017a and other more recent releases, the cov (link) function has 3 possible options for the nanflag (link) parameter. One of these should work for you.

[Nicu Sprincean]

Thank you for your suggestions. I'm still struggling to find something like this

data = data(~isnan(data(:,2).*data(:,1)),:); % select only elements which are not NaN

which works only for one asset return. In the above comment I've mentioned that there are only 4 assets, but this was just an example, cause my data contain in fact hundreds of firms. The whole code is too complicated, so I'd rather prefer to apply something like this code, but for n assets. Thank you again for your suggestions.

[Star Strider]

My pleasure.

The MATLAB cov code is likely optimized to be efficient. Please read and understand the documentation I linked to, in order to understand how to apply the various ‘nanflag’ options to your problem. One of them should work with your data.

[Nicu Sprincean]

Hi, again. I have another issue. I have a MxN matrix (matrix A) and I run the following code:

B = A(~isnan(A));

but I get a Mx1 vector as a result. How could I manage it to get a matrix, with the same number of columns (N)? Or this happens due to the fact that the number of rows is not equal? Thank you.

[Star Strider]

My pleasure.

That is normal, and results from numeric arrays not being defined to have ‘empty’ elements, so the matrix structure disappears and produces a vector of the ‘non-NaN’ values. (Cell arrays can have empty elements.)

You can create the original matrix as a cell array with the num2cell function. That will allow ‘empty’ elements, although it introduces the complexities of calculating with cell arrays, and does not eliminate your original problem.

That it has (Mx1) elements indicates to me that it has (M-1)*N elements that were NaN.

Since you have functions available that automatically calculate with NaN values and handle them as you want them to, I would just use those functions and not deal with eliminating the NaN values or using cell arrays.