(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Camelia Simoiu, Sam Corbett-Davies, and Sharad Goel write:
Outcome tests are a popular method for detecting bias in lending, hiring, and policing decisions. These tests operate by comparing the success rate of decisions across groups. For example, if loans made to minority applicants are observed to be repaid more often than loans made to whites, it suggests that only exceptionally qualified minorities are granted loans, indicating discrimination. Outcome tests, however, are known to suffer from the problem of infra-marginality: even absent discrimination, the repayment rates for minority and white loan recipients might differ if the two groups have different risk distributions. Thus, at least in theory, outcome tests can fail to accurately detect discrimination. We develop a new statistical test of discrimination—the threshold test—that mitigates the problem of infra-marginality by jointly estimating decision thresholds and risk distributions. Applying our test to a dataset of 4.5 million police stops in North Carolina, we find that the problem of infra-marginality is more than a theoretical possibility, and can cause the outcome test to yield misleading results in practice.
It’s an interesting combination of economics and statistics. Also, they do posterior predictive checks and use Stan! I only wish that on Figure 8 they’d’ve labeled the lines directly. Or at least put the codes of the legend in the same order as the lines in the graph. Figure 9, too. Also, I think Figure 7 would’ve worked better as a 2 x 4 grid of graphs. All those dots with different colors are just too hard to visually process.
The post “The problem of infra-marginality in outcome tests for discrimination” appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
