Fighting Back on Algorithmic Opacity
A review of existing tools and policies that promote transparency in Algorithmic Decision-Making Systems
Series on Responsible AI
Algorithmic systems are plagued with opacity, and often become visible to the public only after their harms materialize.
Let’s take two prominent examples of opaque high-risk algorithmic systems.
In the Netherlands, a black-box algorithmic system called SyRI was used to detect welfare fraud. The system was found to have severe racial bias and was ineffective at detecting fraud. The deployment of this system resulted in severe harms caused to families wrongfully flagged as fraudulent. Low-income families were forced to pay back money they didn’t owe, resulting in evictions and indebtedness. Activists and journalists lobbied for years to retire SyRI, which was eventually found to be unlawful as it did not did not comply with the right to privacy under the European Convention of Human Rights.
In the US, a recidivism prediction algorithm called COMPAS was used by courts to assess the likelihood of a defendant becoming a recidivist, thus impacting jail sentences and bail amounts. A landmark investigative piece that leveraged external audits found evidence of bias in COMPAS as black defendants were far more likely than white defendants to be incorrectly flagged to be at higher risk of recidivism. It was also shown that COMPAS is no better at predicting an individual’s risk of recidivism than random volunteers recruited from the internet.
In both of these cases, activists, investigative journalists, and academics (a group I refer to as Public-Interest Groups) carried out a significant effort to achieve some level of transparency into the performance of these systems.
Algorithmic opacity can be experienced at multiple levels. At the most fundamental level, the system can be invisible, meaning impacted individuals are unaware of its existence. Next, there is process-related opacity, where the design of the system and the processes that operate it are unknown to the public. Finally, there is outcome-related opacity. The performance, effectiveness, and accuracy of these algorithmic systems are not typically publicly shared, and impacted individuals are not given clear pathways for remedial.
Why focus on transparency?
Transparency is an important pre-requisite to enable the responsible use of algorithmic systems. Transparency delivers critical information on the system’s design and performance that can help lead to accountability, agency and recourse.
Transparency can also be a vague term that can be difficult to measure.
How might one determine if an algorithmic system is transparent enough? Would dumping large files of data, source code, and documentation onto the public make a system transparent?
Maybe, but this type of transparency is not meaningful to the general public as the documentation may not be directly relevant or comprehensible. Data dumps can also lead to information overload as one might not know where to start to understand how they are impacted by the system.
We should argue instead for the promotion of meaningful transparency.
Meaningful transparency **** is driven by stakeholder information needs. It means delivering the information pertinent to each stakeholder group in the format that is best suited for their understanding.
For example, an oversight body or external auditor may need access to source code, data and in-depth information to validate the responsible use of the algorithmic system; whereas impacted individuals would be concerned with how the system might impact them and what channels are available for feedback and remedial.
From a societal viewpoint, there are three levels of meaningful algorithmic transparency:
- Level 0: Visibility Baseline. This might include basic information on the existence of the system, its scope and owner.
- Level 1: Process Visibility. This includes disclosures about the system’s design and the processes that govern it. This information is helpful to assess the system’s implementation of responsible use safeguards.
- Level 2: Outcome Visibility. This includes disclosures related to the outcomes that the system produces. This information should be assessed to understand the system’s compliance with Responsible Use principles: fairness, explainability, security, safety, robustness, and privacy.
Existing efforts to operationalize transparency
Operationalizing transparency means:
- Defining transparency requirements and when they are applicable;
- Creating and adopting tools that enable compliance with transparency requirements; and
- Validating compliance with transparency requirements.
Governments are critical to enabling algorithmic transparency as, in theory, they have to the means to regulate the use of algorithmic systems and enforce checks and balances.
In reality, algorithmic regulation is still in its infancy. Most governments that seek to regulate algorithmic have focused their efforts on developing standards and cataloguing the algorithmic systems used by the public sector. Others struggle to regulate algorithmic systems due to a lack of capacity and technical expertise.
The most comprehensive algorithmic regulation to be proposed to date is the European Commission’s Artificial Intelligence Act, shared in April 2021. This is a landmark proposal in many ways: it mandates the creation of a public database of high-risk systems and disclosure of conformity assessments. There is hope that the EU AI Act could catalyze increasing change in public and private sector approaches to algorithmic transparency and potentially have global repercussions, similar to how GDPR influenced global privacy regulation.
The EU AI Act has also been criticized by several prominent public interest groups, chiefly because of its vagueness concerning risk-level definitions and how they would be enforced. Other criticisms include not creating a meaningful space for civil society participation in the process, and failing to include provisions for redress and accountability.
Beyond the scope of regulation, a number of tools to operationalize transparency have been proposed and adopted by private and public institutions. These include self-administered impact assessments, external audits, and documentation mechanisms.
An exhaustive list of algorithmic transparency tools and policies can be found in the table below.
Persisting challenges
In addition to the regulatory challenges discussed, there are several additional problems to overcome:
- Inherent model opacity. As the use of black-box models has proliferated, how can we ensure explainability of outcomes?
- Trade secrets. Algorithmic solution providers often cite "IP concerns" when it comes to making transparency disclosures. One emerging concern is that by making the inner workings of a decision-making system public, it may create externalities where dishonest agents could game the system to their advantage.
- Cost to comply. Most organizations developing algorithmic systems would need to invest resources to document, assess, and comply with proposed requirements. This creates adoption friction.
Towards meaningful transparency
Based on this review, there is a clear need for a comprehensive approach towards creating meaningful algorithmic transparency.
First, we need a robust risk assessment framework for algorithmic systems.
Next, we need to define disclosures required for each risk-type throughout the system lifecycle. It can be helpful to consider creating a spectrum of disclosures with the most sensitive data shared only with auditors through privileged access channels.
Finally, we need the right incentive mechanism to ensure adherence and right of recourse for civil society.
Going deeper: Algorithmic Registers
My thesis largely focused on the potential role of algorithmic registers in enabling transparency. In its simplest form, an algorithmic register is a log of algorithmic decision making systems in use by an entity and includes relevant disclosures for concerned stakeholders.
Over the past couple of years, a number of registers have been deployed. These include registers released by local governmental bodies, such as the ones in Amsterdam, Helsinki and Nantes, as well as those created by public interest groups, such as the one created by Privacy Network in Italy.
Based on a review of this first generation of algorithmic registers, I found that algorithmic registers are a versatile tool that – if designed with care:
- Can enable meaningful transparency by meeting the information needs to different stakeholder groups;
- Can incentivize system owners to implement better internal ‘responsible use’ controls;
- Can enable citizen feedback loops and amplify the role of public interest groups while reducing the transparency burden;
- Can complement existing accountability mechanisms and get organizations prepared for a changing regulatory environment.
Part of a series on Responsible AI based on my graduate thesis "Beyond the Black Box" (MIT 2021). The ideas presented were developed based on the feedback and support of several practitioners with direct experience of regulating, deploying, and assessing AI systems. I am sharing and open-sourcing my findings to enable others to easily study and contribute to this space.
Share This Article
Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.
Write for TDS
