Neural Network (NARX) performance interpretation - how small is small?
2 ビュー (過去 30 日間)
I have built a neural network to forecast currency pair prices (trend) a few steps ahead. How do I interpret the performance plot results? For instance, the error histogram, the NMSE (what is considered small for currency pairs? How do I assess if these metrics are optimal?
回答 (1 件)
Greg Heath 2017 年 8 月 19 日
If you naively model a function with a straight line, the MSE is just the variance of the function and varies with the scale of the function.
In previous MATLAB posts I have used the notations
MSE00 = vart1 = mean(var(target'),1)
and use it as a standard for measuring MSE.
Using this NAIVE model as a comparative standard, the normalized MSE is defined by
NMSE = MSE/mean(var(target',1))
which is related to the statistical RSQUARE via
RSQUARE = 1 - NMSE
For reasonable models both NMSE and RSQUARE lie within the closed interval [ 0,1].
For most regression and classification models, I use a goal
NMSEgoal = 0.01 % Regression & Classification
However, in order to obtain that goal for CLOSED LOOP time series, I use the OPEN LOOP goals
NMSEgoal = 0.005 or 0.001
I don't know of anyone who complains when Rsquare >= 0.99 or equivalently, NMSE <= 0.01.
Hope this helps.
Thank you for formally accepting my answer