Kolmogorov-Smirnov and related goodness-of-fit tests

I understand how to compare two distributions using the KS or related tests. But my situation is somewhat different. I have 15 distributions from one population (~100 samples per distribution) and 15 distributions from a different population. I can generate the mean +/- sdev distribution for each of the 2 populations. How do I use the KS, Anderson-Darling, or Cramer Von Mises tests with these data (i.e., N=15 sample distributions from each of 2 populations)? I want to test whether the two populations are significantly different from each other.

[Jeff Miller]

Why not just combine the 15 samples from each population and then compare two samples of about 1500 each? If each set of 15 samples really does come from the same population, then I see no reason to keep them separate.

[the cyclist]

Jeff's idea was the first thing I thought of as well. But ...

I think you need to be really clear on the language you are using in this question. What do you mean by a "distribution from a population"? That phrase has no precise meaning to me.

It seems like it means "sample from a population", but then we are left with "I have 15 samples from one population (~100 samples per sample)". So, that makes me uncomfortable.

You might want to give a bit more detail.

[Stephen]

thanks, Jeff & cyclist! yeah, I'm not so good with stat-speak. Here's the situation: I'm measuring the fluorescent intensity at 100 points across the diameter of microscoptic structures. I have 15 such structures from one group of animals, and 15 similar structures from another group of animals that have received a drug treatment. I can plot the distribution of the fluorescence vs distance (diameter) for 100 points in each animal, 15 such distributions in group A and 15 in group B. Being biological samples, there is a good deal of variance in the distributions even within a group. My question is, how can I use KS or related tests to test whether the average (+/- sdev) distribution for group A differs from group B? Do I merely calculate the two average distributions, ignore any variance, and use those 2 distributions for the KS test? This doesn't seem right, given the variance within each group.

[Jeff Miller]

The additional info is helpful, but the relevance of "100 points across the diameter" is still a bit mysterious.

If you want to test whether the averages of group A and B are different, it seems to me that the simplest approach is to reduce the 100 measurements for each structure into a single average for that structure, and then do a 'ttest2' to compare the group A averages against the group B averages (or possibly use a nonparametric 'ranksum' test if the 15 scores in each group look really non-normal). This obviously gives you no information about the influence, if any, of diameter, but I am not sure whether that has any relevance.

You also ask about whether the sdevs of the two populations are different, but I'm not sure what variance you are thinking of here. There is some variance across the 100 scores for each structure. There is some possibly very different variance of the 100-measurement averages across structures within a group. Do you want to compare either of these variances, or something else entirely?

Finally, to me the KS test doesn't seem appropriate for any of the questions you are asking. It would have relatively low power to detect differences in averages or sdevs, and it could easily report differences between two distributions even if the means and sdevs were identical. So, I recommend looking at focussed tests of means and variances.

[Stephen]

Jeff, t-test is inappropriate. The distributions are highly non-Gaussian. To save 10,000 words, below is a fig showing the situation. The symbols show mean +/- sem of 100 measurements across the diameter of a structure from control (red, n=15 different animals) and drug-treatment (blue, n=15 animals). The question is, how can I use KS or related tests to determine whether these 2 distributions of immunostaining across the X dimension are significantly different. They certainly appear so to the eye. But other drug treatments aren't so apparent to the eye. Need statistics.

[Jeff Miller]

I see that you are using the term "distribution" much differently than I originally thought. That is, for you a "distribution" is not a frequency distribution of randomly varying scores within a single condition, but rather a systematic change in scores across diameter position.

This graph does not display any information relevant to the question of whether the scores are Gaussian in the statistically relevant sense, which is whether the 15 scores that went into each one of the 2*100 plotted points all come from normal distributions (with potentially 200 different means).

For your question, I would consider using two-factor ANOVA. One factor is control vs drug treatment, with n=15 in each group. This is a "between-Ss" factor. The other factor is X dimension, with 100 levels. This is a "repeated measures" factor since each animal was measured at all levels. If you have a specific functional model for how Y changes across X, then you can remove the X factor and replace it with the parameters of that function (estimated separately for each animal).

[Stephen]

oK, Jeff. Thanks for the insights.