In: Statistics and Probability
SInce we are using terminologies, like SSwithin and SSbetween, I am assuming we are talking about a CRD model. Such a model is given by:
The F_critical is determined by the corresponding upper quantile of the F distribution with degrees of freedom v-1and n-v, where v is the total number of groups (or treatments) and n is the total number of observations.
SS_{Total} can be broken down as:
Here v is the number of groups and ni is the number of observations in the ith group.
The F stat is given by:
Thus, if SSwithin decreases without any changes in SSbetween, we see that the F statistic should increase in value and hence we would would have more inclination towards rejection of the null hypothesis while doing hypothesis test.
If I believe the null hypothesis (that the treatments have similar effects) to be true, then I believe that the F ratio above should have a smaller value. The reason being that, if teatments, don't really have different effects, then there should not be much variation in the group specific sample means and hence SSbetween should be much small as compared to SSwithin (which is measure of variance within the groups). Thus, I would expect F statistic to have smaller value.