What is the best smoothing procedure to calculate differentials for a large number of data points (x,y)?

There is a set of data (X,Y) independently obtained from an experiment. Plotting Y vs X looks like this. There are ~114000 data points.

Upon zooming in, you can see the scatter in the data. Because of this when the slope (instantaneous differential) is calcultaed using diff(Y)./diff(X), the scatter is magnified and the trend I'm looking for is completely lost.

For now, I used an arbitratry smoothing procedure using moving average to somewhat acquire the trend I'm looking for, which looks like this:

Could someone please suggest a more standard procedure that I could employ that would be statistically accurate? I have attached the dataset (X,Y).

Answer 1

[Bruno Luong]

The Savitzky-Golay filter (moving polynomial fit) is a good filter that preserves decendly the slope.

There is a similar thread discussed not long ago, in short you fit data with appropriate model/tools 'depending on the characteristic of you data); then take the derivative of the model.

If I use my own free knots spline tool on your data I get this

load('example.mat')
pp=BSFK(X,Y); %FEX file
pp1=ppder(pp);
xq=linspace(min(X),max(X),513);
ydq=ppval(pp1,x)
plot(xq,ydq)

[cjnavee]

Thank you, I will try that out!