Schonemann, P. H.
Some new results on hit rates and base rates in mental testing
Chinese Journal of Psychology, 1997, 39, 173-192.
Recent work on hit rates and base rates (Schonemann and Thompson, 1996) is extended: A flawed premise in the derivation of an earlier hit rate approximation, HR1, is corrected, leading to a slightly more complicated approximation, HR2. However, over the targeted parameter region, the differences between HR1 and HR2 are small.
After deriving exact hit rates for 2x2 contingency tables with binary criteria, they are compared with HR1 and HR2, and also with hit rates for continuous criteria inferred, via Bayes' Theorem, from Taylor and Russell's (1939) tables. Overall, the simpler approximation HR1 outperforms HR2.
Finally, a new approximation is derived for the minimum validity needed that a test improves over random admissions in terms of total percent of correct classifications. More than four decades ago, Meehl and Rosen (1955) warned that validity coefficients, in isolation, are insufficient for gauging the practical merit of a test, because
"... when the base rates of the criterion classification deviate greatly from a 50 percent split, use of a test sign having slight or moderate validity will result in an increase of erroneous clinical decisions" (p. 215. Emphasis in the original).
The present results corroborate these concerns.
Click here for Text
(a) This is is a continuation of
Schonemann P. H. and Thompson (1996) Hit-rate bias in mental testing. Cahiers de Psychologie / Current Contents of Cognition, 3-28.
(b) The undiminished relevance of the above quote from Meehl and Rosen (1955) to this day becomes apparent if one takes the trouble to check on the precidictive validities of the college entrance tests such as the SAT and ACT. For 1st year GPA they range in the upper 30s to low 40s (Donlon, 1984), while for more relevant long range criteria, such as 4th year GPA or college graduation, they decline into the 20s (Humphreys, 1968). As Tables 6 and 7 show, such tests do not improve total percent of correct classification over random admission except for virtually perfect 50/50 base rate splits.
Donlon T. F. (1984) The College Board Technical handbook for the Scholastic Aptitude Test and Achievement Tests. New York: CEEB.
Humphreys, L. G. (1968) The fleeting nature of the prediction of college academic success. Journal of Educational Psychology, 59, 375-380.
(c) So far as I am aware, the present paper and its predecessor, Hit-rate bias in mental testing, are first in presenting comprehensive tables of hit-rates and base-rates as a function of quota and validity. While the base-rate problems has been known for a long time, the discussions we have found include only episodic illustrations rather than exhaustive tabulations of the actual values. The same is true for hit-rates and false alarm rates.