Schonemann, P. H. and Thompson, W. W.
Hit-rate bias in mental testing
Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 1996, 15, 3-28.
New results are given concerning a form of test bias which, in the past with few exceptions (esp. Cole, 1973; Hartigan & Wigdor, 1989) seems to have been largely ignored. We call it "hit-rate bias" because it is defined as the discrepancy between the hit-rates (=probability that a qualified testee passes the test) in a low and a high scoring group. Typically, it favors the high-scoring group. In contrast to Cole (1973), our focus is on binary criteria, such as college graduation.
In the first, theoretical part, we present a (Hit-Rate Bounds) Theorem which underscores that raising predictor standards is not equivalent to raising criterion standards, as some believe. Instead, it typically increases hit-rate bias.
We then derive and tabulate a simple approximation for estimating hit-rates as a function of validity, base-rate, and admission quota.
In the empirical portion of the paper, we evaluate the extent of hit-rate-bias in practice by re-analyzing a number of data sets involving the SAT, the ACT, and the GATB. Finally, we discuss how the addition of test score to high school record affects hit-rate bias in predicting college graduation. We find it increases the bias.
(a) The paper contains some mistakes, most of which are cleared up in a follow-up paper:
Schonemann, P. H. (1997) Some new results on hit rates and base rates in mental testing . Chinese Journal of Psychology, 39, 173-192.
(1) Figure 2b on p.23 is in error;
(2) In deriving the simple approximation formulae for hit-rates and false alarm rates (Appendix 2, p. 27f), the standard deviation in fla. (19) should have been the square root of the total variance, not just of the within variance. In the above follow-up paper it is shown that, as long as the validities stay below .5, the corrected approximation differs only minutely from that given in the 1996 CPC paper and that, in fact, the "incorrect" approximation outperformed the "correct" approximation in comparisons with the exact values (obtained by iteration).
(b) The findings of the paper bear directly on the ongoing affirmative action debate. On reanalyzing data collected under the auspices of the NCAA, we found the suspicions of numerous critics of the college admission tests confirmed that both the SAT and the ACT discriminate against minorities in terms of hit-rates: Qualified minority candidates face steeper odds than qualified white candidates to pass the admission tests, where "qualified" is defined by college graduation.
This type of bias is not limited to college entrance exams. For example, Hartigan and Wigdor (Fairness in Emploment Testing: Validity generalization, minority issues, and the General Aptitude Test battery. Washington: National Academy Press, 1989) also found it for the GATB:
"At this point in history, it is certain that the use of the GATB without some sort of score adjustment would systematically screen out blacks, some of whom could have performed satisfactorily on the job. Fair test use would seem to require at the very least that the inadequacies of the technology should not fall more heavily on the social groups already burdened by the effects of past and present discrimination." (p. 260)