Researchers from Penn State and Columbia University have developed an artificial intelligence (AI) tool capable of detecting discrimination.

The team developed the AI tool to detect discrimination with respect to protected attributes such as race and gender by both human decision makers and AI systems with an eye toward the concept of causality where one thing (cause) leads to another (effect).

Considering that AI systems are increasingly involved with making decisions concerning everything from policing, employment, higher education, consumer finance and business, researchers concentrated specifically on employment decisions and tested their AI tool by rephrasing the question 'Is there gender-based discrimination in salaries?' to 'Does gender have a causal effect on salary?', or in other words, 'Would a woman be paid more if she was a man?' Using counterfactual inference algorithms, the team developed an answer to what was seemingly a hypothetical question.

According to Aria Khademi, graduate student in information sciences and technology at Penn State, "one intuitive way of arriving at a best guess as to what a fair salary would be for a female employee is to find a male employee who is similar to the woman with respect to qualifications, productivity and experience. We can minimize gender-based discrimination in salary if we ensure that similar men and women receive similar salaries."

Yet, using information such as income data from the U.S. Census Bureau, the team analyzed adult income data sets about demographic, salary and employment-related information for almost 50,000 individuals and discovered that the odds of a woman earning an annual salary greater than $50,000 is just one third that of a man.

Unfair treatment of persons based on protected attributes such as race, gender or ethnicity, perpetuated by either human decision makers or AI systems tasked with autonomous decision making, isn’t always easy to detect. Specifically, AI systems are only as good as the data they are trained on. For instance, a company that has historically only ever hired men for a specific position will likely train its AI system using that historical data and subsequently only select male candidates for that position.

"There's nothing wrong with the machine learning algorithm itself," said Vasant Honavar, professor of Information Sciences and Technology at Penn State. "It's doing what it's supposed to do, which is to identify good job candidates based on certain desirable characteristics. But since it was trained on historical, biased data it has the potential to make unfair recommendations."

As such, the team believes that their AI tool is urgently needed.

"Our tool can help ensure that such systems do not become instruments of discrimination, barriers to equality, threats to social justice and sources of unfairness," said Honavar.