Loss Function and Model Quality Metrics

What is a Loss Function?

The System Identification Toolbox™ software estimates model parameters by minimizing the error between the model output and the measured response. This error, called loss function or cost function, is a positive function of prediction errors e(t). In general, this function is a weighted sum of squares of the errors. For a model with ny-outputs, the loss function V(θ) has the following general form:

$V(\theta )= \frac{1}{N}{\displaystyle \sum }_{t=1}^N{e}^T\left(t,\theta \right) W\left(\theta \right) e\left(t,\theta \right)$

where:

• N is the number of data samples.

• e(t,θ) is ny-by-1 error vector at a given time t, parameterized by the parameter vector θ.

• W(θ) is the weighting matrix, specified as a positive semidefinite matrix. If W is a diagonal matrix, you can think of it as a way to control the relative importance of outputs during multi-output estimations. When W is a fixed or known weight, it does not depend on θ.

The software determines the parameter values by minimizing V(θ) with respect to θ.

For notational convenience, V(θ) is expressed in its matrix form:

$V\left(\theta \right)=\frac{1}{N}trace\left(E^T\left(\theta \right) E\left(\theta \right) W(\theta )\right)$

E(θ) is the error matrix of size N-by-ny. The i:th row of E(θ) represents the error value at time t = i.

The exact form of V(θ) depends on the following factors:

• Model structure. For example, whether the model that you want to estimate is an ARX or a state-space model.

• Estimator and estimation options. For example, whether you are using n4sid or ssest estimator and specifying options such as Focus and OutputWeight.

Options to Configure the Loss Function

You can configure the loss function for your application needs. The following estimation options, when available for the estimator, configure the loss function:

Estimation Option	Description	Notes
Focus	Focus option affects how e(t) in the loss function is computed: When Focus is 'prediction', e(t) represents 1-step ahead prediction error: $e_p\left(t\right)=y_{measured}\left(t\right)-y_{predicted}(t)$ When Focus is 'simulation', e(t) represents the simulation error: $e_s\left(t\right)=y_{measured}\left(t\right)-y_{simulated}(t)$ Note For models whose noise component is trivial, (H(q) = 1), ep(t), and es(t) are equivalent. The Focus option can also be interpreted as a weighting filter in the loss function. For more information, see Effect of Focus and WeightingFilter Options on the Loss Function.	Specify the Focus option in the estimation option sets. The estimation option sets for oe and tfest do not have a Focus option because the noise-component for the estimated models is trivial, and so ep(t) and es(t) are equivalent.
WeightingFilter	When you specify a weighting filter, prefiltered prediction or simulation error is minimized: $$e_f(t)=ℒ(e(t))$$ where $ℒ(.)$ is a linear filter. The WeightingFilter option can be interpreted as a custom weighting filter that is applied to the loss function. For more information, see Effect of Focus and WeightingFilter Options on the Loss Function.	Specify the WeightingFilter option in the estimation option sets. Not all options for WeightingFilter are available for all estimation commands.
EnforceStability	When EnforceStability is true, the minimization objective also contains a constraint that the estimated model must be stable.	Specify the EnforceStability option in the estimation option sets. The estimation option sets for procest and ssregest commands do not have an EnforceStability option. These estimation commands always yield a stable model. The estimation commands tfest and oe always yield a stable model when used with time-domain estimation data. Identifying unstable plants requires data collection under a closed loop with a stabilizing feedback controller. A reliable estimation of the plant dynamics requires a sufficiently rich noise component in the model structure to separate out the plant dynamics from feedback effects. As a result, models that use a trivial noise component (H(q) = 1), such as models estimated by tfest and oe commands, do not estimate good results for unstable plants.
OutputWeight	OutputWeight option configures the weighting matrix W(θ) in the loss function and lets you control the relative importance of output channels during multi-output estimations. When OutputWeight is 'noise', W(θ) equals the inverse of the estimated variance of error e(t): $W\left(\theta \right)={\left(\frac{1}{N}{E}^T\left(\theta \right) E\left(\theta \right)\right)}^{-1}$ Because W depends on θ, the weighting is determined as a part of the estimation. Minimization of the loss function with this weight simplifies the loss function to: $V\left(\theta \right)=det\left(\frac{1}{N}{E}^T\left(\theta \right) E\left(\theta \right)\right)$ Using the inverse of the noise variance is the optimal weighting in the maximum likelihood sense. When OutputWeight is an ny-by-ny positive semidefinite matrix, a constant weighting is used. This loss function then becomes a weighted sum of squared errors.	Specify the OutputWeight option in the estimation option sets. Not all options for OutputWeight are available for all estimation commands. OutputWeight is not available for polynomial model estimation because such models are always estimated one output at a time. OutputWeight cannot be 'noise' when SearchMethod is 'lsqnonlin'.
ErrorThreshold	ErrorThreshold option specifies the threshold for when to adjust the weight of large errors from quadratic to linear. Errors larger than ErrorThreshold times the estimated standard deviation have a linear weight in the loss function. $V(\theta )= \frac{1}{N}\left({\displaystyle \sum }_{t\in I}{e}^T\left(t,\theta \right) W\left(\theta \right) e\left(t,\theta \right)+{\displaystyle \sum }_{t\in J}{v}^T\left(t,\theta \right) W\left(\theta \right) v\left(t,\theta \right)\right)$ where: I represents those time instants for which $$\begin{array}{l} \left|\text{e}\left(\text{t}\right)\right|<\rho *\sigma \\ \end{array}$$, where ρ is the error threshold. J represents the complement of I, that is, the time instants for which $$\left|\text{e}\left(\text{t}\right)\right|>=\rho *\sigma$$. σ is the estimated standard deviation of the error. The error v(t,θ) is defined as: $v\left(t,\theta \right)=e\left(t,\theta \right)*\sigma \frac{\rho }{\sqrt{\left|e\left(t,\theta \right)\right|}}$	Specify the ErrorThreshold option in the estimation option sets. A typical value for the error threshold ρ = 1.6 minimizes the effect of data outliers on the estimation results.
Regularization	Regularization option modifies the loss function to add a penalty on the variance of the estimated parameters. The loss function is set up with the goal of minimizing the prediction errors. It does not include specific constraints on the variance (a measure of reliability) of estimated parameters. This can sometimes lead to models with large uncertainty in estimated model parameters, especially when the model has many parameters. Regularization introduces an additional term in the loss function that penalizes the model flexibility: $V\left(\theta \right)= \frac{1}{N}{\displaystyle \sum }_{t=1}^N{e}^T\left(t,\theta \right) W\left(\theta \right) e\left(t,\theta \right) + \frac{1}{N}\lambda {\left(\theta -{\theta }^{*}\right)}^{T}R \left(\theta -{\theta }^{*}\right)$ The second term is a weighted (R) and scaled (λ) variance of the estimated parameter set θ about its nominal value θ*.	Specify the Regularization option in the estimation option sets. For linear-in-parameter models (FIR models) and ARX models, you can compute optimal values of the regularization variables R and λ using the arxRegul command.

Effect of Focus and WeightingFilter Options on the Loss Function

The Focus option can be interpreted as a weighting filter in the loss function. The WeightingFilter option is an additional custom weighting filter that is applied to the loss function.

To understand the effect of Focus and WeightingFilter, consider a linear single-input single-output model:

$$y(t)=G(q,\theta )u(t)+H(q,\theta )e(t)$$

Where G(q,θ) is the measured transfer function, H(q,θ) is the noise model, and e(t) represents the additive disturbances modeled as white Gaussian noise. q is the time-shift operator.

In frequency domain, the linear model can be represented as:

$$Y(\omega )=G(\omega ,\theta )U(\omega )+H(\omega ,\theta )E(\omega )$$

where Y(ω), U(ω), and E(ω) are the Fourier transforms of the output, input, and output error, respectively. G(ω,θ) and H(ω,θ) represent the frequency response of the input-output and noise transfer functions, respectively.

The loss function to be minimized for the SISO model is given by:

$V(\theta )= \frac{1}{N}{\displaystyle \sum }_{t=1}^N{e}^T\left(t,\theta \right) e\left(t,\theta \right)$

Using Parseval’s Identity, the loss function in frequency-domain is:

$$V(\theta ,\omega )= \frac{1}{N}{\Vert E(\omega )\Vert }^2$$

Substituting for E(ω) gives:

$$V(\theta ,\omega )=\frac{1}{N}{\Vert \frac{Y(\omega )}{U(\omega )}-G(\theta ,\omega ))\Vert }^2\frac{{\Vert U(\omega )\Vert }^2}{{\Vert H(\theta ,\omega )\Vert }^2}$$

Thus, you can interpret minimizing the loss function V as fitting G(θ,ω) to the empirical transfer function $$Y(\omega )/U(\omega )$$, using $$\frac{{\Vert U(\omega )\Vert }^2}{{\Vert H(\theta ,\omega )\Vert }^2}$$ as a weighting filter. This corresponds to specifying Focus as 'prediction'. The estimation emphasizes frequencies where input has more power ($${\Vert U(\omega )\Vert }^2$$ is greater) and de-emphasizes frequencies where noise is significant ($${\Vert H(\theta ,\omega )\Vert }^2$$ is large).

When Focus is specified as 'simulation', the inverse weighting with $${\Vert H(\theta ,\omega )\Vert }^2$$ is not used. That is, only the input spectrum is used to weigh the relative importance of the estimation fit in a specific frequency range.

When you specify a linear filter $ℒ$ as WeightingFilter, it is used as an additional custom weighting in the loss function.

$$V(\theta )=\frac{1}{N^2}{\Vert \frac{Y(\omega )}{U(\omega )}-G(\theta ))\Vert }^2\frac{{\Vert U(\omega )\Vert }^2}{{\Vert H(\theta )\Vert }^2}{\Vert ℒ(\omega )\Vert }^2$$

Here $$ℒ(\omega )$$ is the frequency response of the filter. Use $$ℒ(\omega )$$ to enhance the fit of the model response to observed data in certain frequencies, such as to emphasize the fit close to system resonant frequencies.

The estimated value of input-output transfer function G is the same as what you get if you instead first prefilter the estimation data with $ℒ(.)$ using idfilt, and then estimate the model without specifying WeightingFilter. However, the effect of $ℒ(.)$ on the estimated noise model H depends on the choice of Focus:

• Focus is 'prediction' — The software minimizes the weighted prediction error $$e_f(t)=ℒ(e_p(t))$$, and the estimated model has the form:

  $$y(t)=G(q)u(t)+H_1(q)e(t)$$

  Where $$H_1(q)=H(q)/ℒ(q)$$. Thus, the estimation with prediction focus creates a biased estimate of H. This is the same estimated noise model you get if you instead first prefilter the estimation data with $ℒ(.)$ using idfilt, and then estimate the model.

  When H is parameterized independent of G, you can treat the filter $ℒ(.)$ as a way of affecting the estimation bias distribution. That is, you can shape the trade-off between fitting G to the system frequency response and fitting $$H/ℒ$$ to the disturbance spectrum when minimizing the loss function. For more details see, section 14.4 in System Identification: Theory for the User, Second Edition, by Lennart Ljung, Prentice Hall PTR, 1999.

• Focus is 'simulation' — The software first estimates G by minimizing the weighted simulation error $$e_f(t)=ℒ(e_s(t))$$, where $e_s\left(t\right)=y_{measured}\left(t\right)-G(q)u_{measured}\left(t\right)$. Once G is estimated, the software fixes it and computes H by minimizing pure prediction errors e(t) using unfiltered data. The estimated model has the form:

  $$y(t)=G(q)u(t)+He(t)$$

  If you prefilter the data first, and then estimate the model, you get the same estimate for G but get a biased noise model $$H/ℒ$$.

Thus, the WeightingFilter has the same effect as prefiltering the estimation data for estimation of G. For estimation of H, the effect of WeightingFilter depends upon the choice of Focus. A prediction focus estimates a biased version of the noise model $$H/ℒ$$, while a simulation focus estimates H. Prefiltering the estimation data, and then estimating the model always gives $$H/ℒ$$ as the noise model.

Model Quality Metrics

After you estimate a model, use model quality metrics to assess the quality of identified models, compare different models, and pick the best one. The Report.Fit property of an identified model stores various metrics such as FitPercent, LossFcn, FPE, MSE, AIC, nAIC, AICc, and BIC values.

• FitPercent, LossFcn, and MSE are measures of the actual quantity that is minimized during the estimation. For example, if Focus is 'simulation', these quantities are computed for the simulation error e_s (t). Similarly, if you specify the WeightingFilter option, then LossFcn, FPE, and MSE are computed using filtered residuals e_f (t).

• FPE, AIC, nAIC, AICc, and BIC measures are computed as properties of the output disturbance according to the relationship:

  $y\left(t\right)=G\left(q\right)u\left(t\right)+H\left(q\right)e(t)$

  G(q) and H(q) represent the measured and noise components of the estimated model.

  Regardless of how the loss function is configured, the error vector e(t) is computed as 1-step ahead prediction error using a given model and a given dataset. This implies that even when the model is obtained by minimizing the simulation error e_s (t), the FPE and various AIC values are still computed using the prediction error e_p (t). The actual value of e_p (t) is determined using the pe command with prediction horizon of 1 and using the initial conditions specified for the estimation.

  These metrics contain two terms — one for describing the model accuracy and another to describe its complexity. For example, in FPE, $det\left(\frac{1}{N}{E}^T E\right)$ describes the model accuracy and $\frac{1+\frac{np}{N}}{1-\frac{np}{N}}$ describes the model complexity.

  By comparing models using these criteria, you can pick a model that gives the best (smallest criterion value) trade-off between accuracy and complexity.

Quality Metric	Description
FitPercent	Normalized Root Mean Squared Error (NRMSE) expressed as a percentage, defined as: $FitPercent= 100\left(1-\frac{\Vert y_{measured}-y_{model}\Vert }{\Vert y_{measured}-\overline{y_{measured}}\Vert }\right)$ where: ymeasured is the measured output data. $\overline{y_{measured}}$ is its (channel-wise) mean. ymodel is the simulated or predicted response of the model, governed by the Focus. ||.|| indicates the 2-norm of a vector. For input or output data, FitPercent is an ny-by-nexp matrix, where ny is the number of outputs and nexp is the number of experiments. For FRD data, FitPercent is an ny-by-nu matrix, where nu is the number of inputs (for FRD data, nexp is always one). FitPercent varies between -Inf (bad fit) to 100 (perfect fit). If the value is equal to zero, then the model is no better at fitting the measured data than a straight line equal to the mean of the data.
LossFcn	Value of the loss function when the estimation completes. It contains effects of error thresholds, output weight, and regularization used for estimation.
MSE	Mean Squared Error measure, defined as: $MSE= \frac{1}{N}{\displaystyle \sum }_{t=1}^N{e}^T\left(t\right) e\left(t\right)$ where: e(t) is the signal whose norm is minimized for estimation. N is the number of data samples in the estimation dataset.
FPE	Akaike’s Final Prediction Error (FPE), defined as: $FPE=det\left(\frac{1}{N}{E}^T E\right)\left(\frac{1+\frac{n_p}{N}}{1-\frac{n_p}{N}}\right)$ where: np is the number of free parameters in the model. np includes the number of estimated initial states. N is the number of samples in the estimation dataset. E is the N-by-ny matrix of prediction errors, where ny is the number of output channels.
AIC	A raw measure of Akaike's Information Criterion, defined as: $AIC=N\ast log\left(det\left(\frac{1}{N}{E}^T E\right)\right)+2\ast n_p+N\left(n_y\ast \log \left(2\pi \right)+1\right)$
AICc	Small sample-size corrected Akaike's Information Criterion, defined as: $$AICc=AIC+2\ast n_p\ast \frac{(n_p+1)}{(N-n_p-1)}$$ This metric is often more reliable for picking a model of optimal complexity from a list of candidate models when the data size N is small.
nAIC	Normalized measure of Akaike's Information Criterion, defined as: $nAIC=log\left(det\left(\frac{1}{N}{E}^T E\right)\right)+\frac{2\ast n_p}{N}$
BIC	Bayesian Information Criterion, defined as: $BIC=N\ast log\left(det\left(\frac{1}{N}{E}^T E\right)\right)+N\ast \left(n_y\ast \log \left(2\pi \right)+1\right)+n_p\ast \text{log}(N)$