In the chi-square test, how to calculate (the correct number of parameters and consequently) the correct number of degrees of freedom, without using the chi2gof function?

Question

In the chi-square test, how to calculate (the correct number of parameters and consequently) the correct number of degrees of freedom, without using the chi2gof function?

I have indeed noticed that the number of degrees of freedom was slightly different in one matlab answer and in the chi2gof function....

Option 1: "df = nbins - 1"

Population = [996, 749, 370, 53, 9, 3, 1, 0];
Sample = [647, 486, 100, 22, 0, 0, 0, 0];
Population2 = [996, 749, 370, sum(Population(4:8))];
Sample2 = [647, 486, 100, sum(Sample(4:8))];
chi2stat = sum((Sample2-Population2).^2./Population2);
df = length(Population2)-1;
pcrit = .05; 
chi2crit = chi2inv(pcrit,df);
h2 = chi2stat > chi2crit; 
p2 = 1 - chi2cdf(chi2stat,df);
fprintf('h=%d, p=%.3f df=%d\n',h2,p2,df);

h=1, p=0.000 df=3

Option 2: "df = nbins - 1 - nparams"

"chi2gof compares the value of the test statistic to a chi-square distribution with degrees of freedom equal to nbins - 1 - nparams, where nbins is the number of bins used for the data pooling and nparams is the number of estimated parameters used to determine the expected counts."

bins = 0:5;
obsCounts = [6 16 10 12 4 2];
n = sum(obsCounts);
pd = fitdist(bins','Poisson','Frequency',obsCounts');
expCounts = n * pdf(pd,bins);
[h,p,st] = chi2gof(bins,'Ctrs',bins,...
                        'Frequency',obsCounts, ...
                        'Expected',expCounts,...
                        'NParams',1)

h = 0

p = 0.4654

st = struct with fields:
    chi2stat: 2.5550
          df: 3
       edges: [-0.5000 0.5000 1.5000 2.5000 3.5000 5.5000]
           O: [6 16 10 12 6]
           E: [7.0429 13.8041 13.5280 8.8383 6.0284]

Answer 1

[dpb]

Although you specified 'Ctrs', bins, chi2gof created only 5 bins because the obsCounts values for the last two bins in the 'Frequency' vector were too small individually. Hence the DOF for the chi-square test statistic turns out to be based on 5-1-1 --> 3 instead of 6-1-1 --> 4 that may have been what you were expecting?

[Sim]

Thanks a lot @dpb for your answer! Actually, I was asking something slightly different. Let me explain better. I have noticed that chi2gof uses

df = nbins - 1 - nparams

while in an aswer provided by @William Rose, he used something like

df = nbins - 1

(to be precise he used: "df = length(Population2)-1")

how can I calculate the correct number of degrees of freedom "df", without using chi2gof?

In other words, if the general formula is

df = nbins - 1 - nparams

How can I calculate the correct number of parameters, without using chi2gof?

Notes related to my question

When I open chi2gof, I see the "nparams" value is calculated for 4 different cases, that I enumerated as (1), (2), (3) and (4), here below:

%   The following options determine the null distribution for the test.  You
%   should not specify both 'cdf' and 'expected'.
%
%      Name       Value
%     'cdf'       A fully specified cumulative distribution function.  This
%                 can be a ProbabilityDistribution object, a function
%                 handle, or a function. name.  The function must take X
%                 values as its only argument. Alternately, you may provide
%                 a cell array whose first element is a function name or
%                 handle, and whose later elements are parameter values,
%                 one per cell. The function must take X values as its
%                 first argument, and other parameters as later arguments.
%     'expected'  A vector with one element per bin specifying the
%                 expected counts for each bin.
%     'nparams'   The number of estimated parameters; used to adjust
%                 the degrees of freedom to be NBINS-1-NPARAMS, where
%                 NBINS is the number of bins.
%
%   If your 'cdf' or 'expected' input depends on estimated parameters, you
%   should use the 'nparams' parameter to ensure that the degrees of freedom
%   for the test is correct.  Otherwise the default 'nparams' value is
%
%     (1) 'cdf' is a ProbabilityDistribution:  number of estimated parameters
%     (2) 'cdf' is a function:                 0
%     (3) 'cdf' is a cell array:               number of parameters in the array
%     (4) 'expected' is specified:             0

Going through the chi2gof code about the "nparams" we can see exactly where and how "nparams" were calculated (@MathWorks Support Team: I hope I can write here a small snippet of the code):

% Get expected counts
% ...something...
if ~isempty(expected)
    % ...something...
else
   % ________________________________________________________
   % (?) 'cdf' is not specified: number of parameters from the estimated
   % normal distribution = 2 (I guess, mean and standard deviation?)
   % ________________________________________________________
   % Get them from the cdf
   if isempty(cdfspec)
      % Default cdf is estimated normal
      % ...something...
      if isempty(nparams)
          nparams = 2;
      end
   % ___________________________________________________________________
   % (1) 'cdf' is a ProbabilityDistribution:  number of estimated parameters
   % ___________________________________________________________________
   elseif isa(cdfspec,'ProbDist') || isa(cdfspec,'prob.ProbabilityDistribution') 
       try 
           nparams = length(cdfspec.Params) - sum(cdfspec.ParamIsFixed);
       catch
          if isempty(nparams) 
             % If the distribution is not fittable and no specification of NPARAMS, 
             % set the number of estimated parameters to 0
             nparams = 0;
          end
       end
       % ...something...
   % ________________________________________________________
   % (3) 'cdf' is a cell array: number of parameters in the array
   % ________________________________________________________
   elseif iscell(cdfspec)
      % Get function and args from cell array
      % ...something...
      if isempty(nparams)
          nparams = numel(cdfargs);
      end
   else
      % Function only, no args
      % ...something...
   end
   % ...something...
   % Compute the expected values 
   % ...something...
end

% Figure out degrees of freedom
if isempty(nparams)
    nparams = 0;   % default if not specified or determined above
end
df = nbins - 1 - nparams;

In summary

In the chi2gof code, among the four cases mentioned at the beginnning of chi2gof,

%     (1) 'cdf' is a ProbabilityDistribution:  number of estimated parameters
%     (2) 'cdf' is a function:                 0
%     (3) 'cdf' is a cell array:               number of parameters in the array
%     (4) 'expected' is specified:             0

I was able to find (only) the following ones:

%     (1) 'cdf' is a ProbabilityDistribution:  number of estimated parameters
%     (3) 'cdf' is a cell array:               number of parameters in the array
%     (?) 'cdf' is not specified:              number of parameters from the estimated normal (nparams = 2)

Therefore, I am confused about how the number of parameters is calculated in chi2gof, and in particular about the calculation of the number of parameters, that is then used to calculate the degrees of freedom "df":

df = nbins - 1 - nparams

Just following my intuition, I think that:

(1) If I need to compare two frequency distributions, i.e. two histograms where I know the heights of the histogram bars, that represent the "counts" of a certain quantity, in each bin of the histogram, I can safely use these equations:

nparams = 0;
df = nbins - 1;

(2) If I need to find the best fitting distribution to my dataset (which I can organise as an histogram), the number of parameters is assessed based on the type of "cdf" I am using (i.e. ProbabilityDistribution, function or array) and that, still, is not clear to me (@MathWorks Support Team):

nparams = ...; % to be estimated based on the type of "cdf", right?
df = nbins - 1 - nparams;

I would greatly appreciate a clarification on the calculation of the number of the parameters, and, consequently, on the degrees of freedom.

[dpb]

You lose a degree of freedom for every parameter estimated from the dataset -- if you estimate none and use raw counts, then Nparams is 0.

While the description is somewhat convoluted, chi2gof attempts to automate that calculation; the different cases just depend upon whether it can be inferred from the input what that number is, not that the actual number is any different for the same case irrespective of how the data are passed. If a pdf object is passed, the number of parameters is determinable from the object; if you use a function, the function can't always be parsed to determine that number so it requires you to provide the correct number. Don't confound the description of how (or if) it does the determination with the number itself.

[Sim]

Thanks for your explanation @dpb! (however, I am still puzzled... :-)

[dpb]

Not sure what the remaing puzzle is so don't know how to try to add anything that haven't already said.

The correction to the number of DOF based solely on number of (collapsed) bins is simply how many parameters of the distribution used to calculate the expected counts per bin were estimated from the data itself -- IF ("the big if") the theoretical distribution parameter values are based on the input data itself.

If you test against counts from a theoretical distribution that is obtained from other considerations, then you've not estimated any further parameters from the count data itself and nParams=0.

[Sim]

Thanks @dpb! :-) :-)