Detecting start and end of signal

I have a dataset which reflects force data over time. There are sections in the data where the force sensor is being repeatedly activated and sections at rest. I need to be able to detect the starting and end point of these "chunks" of data. Here is a small piece of my data to show an example of the points (in red) that I am trying to identify.

What I have tried so far is this:

%compile data
d = table();
fileNames = dir('*.csv');
 for ii = 1:numel(fileNames)
     d0 = readtable(fileNames(ii).name);
     d = [d; d0];
 end

%create new empty index column, the same size as table t
d.index=zeros(size(d, 1), 1);

%number index column
x=[];
index=0;
for i=1:size(d, 1) 
  index=index+1;
  x(end+1) = index;
end
d.index=x';
 
%create variables
T = d.(3)./100;
Fz = d.(2);

%find start and end of data chunks
cpt = findchangepts(Fz,'MaxNumChanges',160,'Statistic','rms');

%find time duration of each chunk
for k = 1:80
    
    sig(k) = T(cpt(2*k))-T(cpt(2*k-1));
    
end

%display time duration data in a column
sig = sig';

Some additional background, I have 80 "chunks" of data and therefore 160 changepoints I am searching for.

With the findchangepts function, using the mean is not sufficient to get the points I am looking for. Using the rms statistic works, but it takes a really long time. For some of the datasets it takes about 60 seconds to process and spit out the 80 values I am looking for. For other sets of the same size I have waited as long as 10 minutes before I finally give up and stop running the script. Then I get this. The findchangepts function always gets stuck at the same point.

I have also looked into smoothing functions but since there are many repeated signals within the "chunk" they don't seem to smooth correctly.

I've attached a portion of my data which contains 10 "chunks", and therefore 20 changepoints. In order to run the code with the attached data those parameters need to be adjusted (change 160 to 20 and change 80 to 10). This took about 60 seconds to process when I ran it.

If anyone has any ideas it would be greatly appreciated! Thank you :)

Answer 1

[Star Strider]

Try something like this —

M1 = readmatrix('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1251052/P05%20Trial%201%201-10.csv')

M1 = 48300×2

0.0100 0.0890 0.0200 0.0900 0.0300 0.0900 0.0400 0.0900 0.0500 0.0900 0.0600 0.0890 0.0700 0.0890 0.0800 0.0900 0.0900 0.0890 0.1000 0.0900

x = M1(:,1);
y = M1(:,2);

[eh,el] = envelope(y, 350, 'peaks');                % Calculate 'envelope' To Outline Data Groups

figure
plot(x, y)
hold on
% plot(x, eh, '-g')
plot(x, el, '-r')
hold off
grid

elv = el;
elv(elv>=-0.4) = -0.4;                                   % Threshold Envelope
elv(elv<=-0.5) = -0.5;                                   % Threshold Envelope
Lv21 = islocalmin(elv, 'FlatSelection','first');
% Q4 = nnz(Lv21)
Lv22 = islocalmin(elv, 'FlatSelection','last');
% Q5 = nnz(Lv22)

figure
plot(x, y)
hold on
plot(x, elv, '-', 'Color',[1.0 0.5 0.1], 'LineWidth',2.0)
plot(x(Lv21), elv(Lv21), '>r', 'MarkerFaceColor','r')
plot(x(Lv22), elv(Lv22), '<r', 'MarkerFaceColor','r')
hold off
grid
ylim([-2.5 0.25])

StartStop = table(find(Lv21),find(Lv22), 'VariableNames',{'Start_Indices','Stop_Indices'})

StartStop = 10×2 table
    Start_Indices    Stop_Indices
    _____________    ____________

         2657            5885    
         8168           11319    
        12796           16151    
        17172           20521    
        22229           22800    
        23911           26749    
        29689           33327    
        34447           37902    
        39917           43036    
        44153           47214    

figure
plot(x, y, 'DisplayName','Force Data')
hold on
plot(x(StartStop{:,1}), ones(size(StartStop{:,1}))*y(1), '>r', 'MarkerFaceColor','r', 'DisplayName','Start')
plot(x(StartStop{:,2}), ones(size(StartStop{:,2}))*y(1), '<r', 'MarkerFaceColor','r', 'DisplayName','Stop')
hold off
grid
xlabel('Time')
ylabel('Force')
title('Results Plot')
legend('Location','best')
axis([min(x) max(x) -2.5  0.25])

EDIT — (5 Jan 2023 at 16:04)

Minor plot format changes. Code unchanged.

.

[Kara Wilcox]

Thank you for getting back to me @Star Strider

The method you have suggested processes the data significantly faster, however, it doesn't provide the accuracy I am looking for. For example, one of the points your code has found is at 239 but it should be around 242, which is 3 seconds off.

To compare, the method I chose produces points here:

The answer may be as simple as the processing just takes a while. Any other ideas?

[Star Strider]

What method are you using? I got the impression that findchangepts wasn’t producing the result you wanted.

The problem is that the data are quite noisy, and that limits the processing and thresholding, since some of the noise spikes produce undesired results in the thresholding steps. Filtering helps, however it can’t eliminate enough of the noise spikes to make the thresholding as robust as I would like it to be.

This is my best effort —

M1 = readmatrix('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1251052/P05%20Trial%201%201-10.csv')

M1 = 48300×2

0.0100 0.0890 0.0200 0.0900 0.0300 0.0900 0.0400 0.0900 0.0500 0.0900 0.0600 0.0890 0.0700 0.0890 0.0800 0.0900 0.0900 0.0890 0.1000 0.0900

x = M1(:,1);
y = M1(:,2);

L = numel(x);
Fs = 1/x(1);
Fn = Fs/2;
NFFT = 2^nextpow2(L);
FTy = fft(y-mean(y), NFFT)/L;
Fv = linspace(0, 1, NFFT/2+1)*Fn;
Iv = 1:numel(Fv);

Fco = 0.30;

figure
plot(Fv, abs(FTy(Iv))*2)
grid
xline(Fco, '-k', sprintf('Lowpass Filter Cutoff (%.2f Hz)',Fco))
xlabel('Frequency (Hz)')
ylabel('Magnitude')
title('Fourier Transform Of Force Signals')
xlim([0 10])

yf = lowpass(y, Fco, Fs, 'ImpulseResponse','iir');

[eh,el] = envelope(yf, 100, 'peaks');                % Calculate 'envelope' To Outline Data Groups

figure
plot(x, yf, 'DisplayName','Filtered Force Signal')
hold on
plot(x, ones(size(x))*y(1), '-k', 'DisplayName','Baseline')
% plot(x, eh, '-g')
plot(x, el, '-r', 'DisplayName','Lower Envelope')
hold off
grid
legend('Location','best')
axis([150 250 -0.5  0.25])

figure
plot(x, yf, 'DisplayName','Filtered Force Signal')
hold on
plot(x, ones(size(x))*y(1), '-k', 'DisplayName','Baseline')
% plot(x, eh, '-g')
plot(x, el, '-r', 'DisplayName','Lower Envelope')
hold off
grid
legend('Location','best')
axis([min(x) max(x) -0.5  0.25])

elv = el;
elv(elv>=-0.20) = -0.20;                                   % Threshold Envelope
elv(elv<=-0.25) = -0.25;                                   % Threshold Envelope
Lv21 = islocalmin(elv, 'FlatSelection','first');
% Q4 = nnz(Lv21)
Lv22 = islocalmin(elv, 'FlatSelection','last');
% Q5 = nnz(Lv22)

figure
plot(x, y)
hold on
plot(x, elv, '-', 'Color',[1.0 0.5 0.1], 'LineWidth',2.0)
plot(x(Lv21), elv(Lv21), '>r', 'MarkerFaceColor','r')
plot(x(Lv22), elv(Lv22), '<r', 'MarkerFaceColor','r')
hold off
grid
axis([125 250 -0.5  0.25])

stidx = find(Lv21);
spidx = find(Lv22);
% dfidx = spidx-stidx

StartStop = table(stidx,spidx, 'VariableNames',{'Start_Indices','Stop_Indices'})

StartStop = 10×2 table
    Start_Indices    Stop_Indices
    _____________    ____________

         2697            6129    
         7968           11211    
        12956           15931    
        17410           20751    
        22067           22929    
        24082           26615    
        29893           33259    
        34642           37918    
        39959           42892    
        44234           47456    

figure
plot(x, y, 'DisplayName','Force Data')
hold on
plot(x(StartStop{:,1}), ones(size(StartStop{:,1}))*y(1), 'vg', 'MarkerFaceColor','g', 'DisplayName','Start')
plot(x(StartStop{:,2}), ones(size(StartStop{:,2}))*y(1), 'vr', 'MarkerFaceColor','r', 'DisplayName','Stop')
hold off
grid
xlabel('Time')
ylabel('Force')
title('Results Plot')
legend('Location','best')
axis([0 125 -0.5  0.25])

figure
plot(x, y, 'DisplayName','Force Data')
hold on
plot(x(StartStop{:,1}), ones(size(StartStop{:,1}))*y(1), 'vg', 'MarkerFaceColor','g', 'DisplayName','Start')
plot(x(StartStop{:,2}), ones(size(StartStop{:,2}))*y(1), 'vr', 'MarkerFaceColor','r', 'DisplayName','Stop')
hold off
grid
xlabel('Time')
ylabel('Force')
title('Results Plot')
legend('Location','best')
axis([125 235 -0.5  0.25])

figure
plot(x, y, 'DisplayName','Force Data')
hold on
plot(x(StartStop{:,1}), ones(size(StartStop{:,1}))*y(1), 'vg', 'MarkerFaceColor','g', 'DisplayName','Start')
plot(x(StartStop{:,2}), ones(size(StartStop{:,2}))*y(1), 'vr', 'MarkerFaceColor','r', 'DisplayName','Stop')
hold off
grid
xlabel('Time')
ylabel('Force')
title('Results Plot')
legend('Location','best')
axis([235 385 -0.5  0.25])

figure
plot(x, y, 'DisplayName','Force Data')
hold on
plot(x(StartStop{:,1}), ones(size(StartStop{:,1}))*y(1), 'vg', 'MarkerFaceColor','g', 'DisplayName','Start')
plot(x(StartStop{:,2}), ones(size(StartStop{:,2}))*y(1), 'vr', 'MarkerFaceColor','r', 'DisplayName','Stop')
hold off
grid
xlabel('Time')
ylabel('Force')
title('Results Plot')
legend('Location','best')
axis([385 max(x) -0.5  0.25])

figure
plot(x, y, 'DisplayName','Force Data')
hold on
plot(x(StartStop{:,1}), ones(size(StartStop{:,1}))*y(1), 'vg', 'MarkerFaceColor','g', 'DisplayName','Start')
plot(x(StartStop{:,2}), ones(size(StartStop{:,2}))*y(1), 'vr', 'MarkerFaceColor','r', 'DisplayName','Stop')
hold off
grid
xlabel('Time')
ylabel('Force')
title('Results Plot')
legend('Location','best')
axis([min(x) max(x) -0.5  0.25])

.