Low Pass filter for big experimental data set

Hello

I have some experimental data-set and I am using a low pass filter in order to remove the noise.

The problem is that when trying to run the code I am getting this warning : " Requested 59992x59992 (26.8GB) array exceeds maximum array size preference."

I hope you can help :)

[Adam Danz]

My bet is that you're "requesting a 59992x59992 array that exceeds maximum array size preference". ;)

You'll need to show us the code that produces this message.

[Ammar Alfaili]

I am not sure that this approach is correct, but this goes wrong in the initial steps where the warning apears.

s = Fx;
noise = 0.5*randn(size(tA));
x1 = noise + s;

figure()
plot(x1)  

Where Fx has a size of 59992 and tA has a size of (1 59992)

[Adam Danz]

" Fx has a size of 59992 "

Do you mean [59992 x 59992] or do you mean that Fx is a vector with 59992 elements?

Let's look at each possibility.

Fx is size [59992 x 59992] matrix

I cannot create Fx at that size because I get the same error you're describing. So i'm wondering how that variable was created in the first place.

Fx is size [1 x 59992] row vector

This code runs quickly and without error because x1 = noise + s; also produces a vector of size 1x59992 which is no big deal.

Fx is size [59992 x 1] column vector

This code produces the error in your message because x1 = noise + s; produces a matrix of size 59992x59992 which is a big deal (3.6 billion values).

So my quesitons are

1) what is the real size of Fx

2) what is the expected size of x1?

My bet is on the 3rd interpretation given your error message (unless that error was caused by creating Fx as described in the 1st interpretation). If that's the case, are you expecting x1 to be a vector or matrix (question #2 above).

Answer 1

[Adam Danz]

My hunch is that x1 should be a vector (not a matrix) and that Fx is a column vector.

If this hunch is correct, the solution is to transpose Fx (or 's' or 'noise').

x1 = noise + s.';
%             ^^ this makes 's' a row vector

If the orientation of a vector (row vs column) is unknown, you can force the vectors to be rows or columns using this method (assuming inputs are vectors of equal lenght).

% If you want a column output
x1 = noise(:) + s(:);

% If you want a row output
x1 = noise(:).' + s(:).';  % or x1 = (noise(:) + s(:)).';

[Ammar Alfaili]

Hi Adam

Fx is size [59992 1] column vector and and so is x1 is size [ 1 59992], row vector

[Ammar Alfaili]

It worked. Thanx a lot :)

[Adam Danz]

Glad I could help!