Commission raises discrimination concerns over AI-driven pricing based on consumers’ personal data
The Ontario Human Rights Commission warned that surveillance pricing, which uses artificial intelligence (AI) and personal data to set individualized prices, could violate Ontario’s Human Rights Code if it results in discrimination based on protected characteristics.
The commission raised concerns that retailers could use automated algorithms to charge consumers different prices for identical products based on their personal circumstances and behaviour, often without their knowledge.
Unlike dynamic pricing, which adjusts prices based on market demand, surveillance pricing uses personal information to set prices in real time. The practice could prevent consumers from comparing prices or knowing whether retailers charge others differently.
The Ontario Human Rights Commission said surveillance pricing could violate the code if retailers use information connected to protected grounds, including race, sex, disability, age or family status, to impose higher prices.
The commission emphasized that discriminatory impact, rather than a business’s intention, determines whether a practice violates the code. Businesses could therefore face human rights concerns even if they do not intend to discriminate when implementing surveillance pricing.
The commission identified several hypothetical scenarios involving potential discrimination. These included a woman paying higher ride-sharing fares than men late at night, a person with a mobility disability facing higher food delivery charges, and an Asian consumer paying more for decorative items before Lunar New Year.
The commission said it has increased its work examining how emerging technologies and business practices, including surveillance pricing and other AI applications, intersect with human rights protections.
In partnership with other leading human rights, privacy, and law reform organizations, the Commission has released a Human Rights AI Impact Assessment with practical guidance to identify and mitigate potential discrimination from AI systems, and Principles for the Responsible Use of Artificial Intelligence.
The resources provide practical guidance to help organizations identify and mitigate potential discrimination arising from AI systems while respecting privacy and human rights.
The commission urged Ontario businesses and organizations considering the use of personal data or AI technologies to consult these resources to ensure compliance with the Code.
The commission said its mandate includes proactively addressing emerging risks to prevent discrimination as organizations explore the potential benefits of artificial intelligence.
