14.12.23
Technology is not value-neutral and it never has been. Designers' choices often carry ethical risks as they play a crucial role in
determining how a technology will be used, for what purposes, and which users will have privileged access to it.
These decisions are closely tied to underlying values and ethical considerations.
In this post, we shed light on how to identify ethical risks based on the technical and socio-economic features of an AI system.
The white papers “OECD Framework for the Classification of AI Systems”
by the Organisation for Economic Cooperation and Development (OECD) and
“Data Justice in Practice: A Guide for Policymakers” by the Global Partnership
on Artificial Intelligence (GPAI) will serve as central references.
Classifying an AI system is the first step of an Ethical Impact Assessment (EIA). This is necessary to identify the sensitive
areas in which AI risks may arise. Moreover, understanding the purpose and the technical features of the system also allows it to be
framed within the risk matrix provided by the AI Act. Depending on the level of risk, different regulatory requirements will apply.
To perform a classification, the OECD recommends analyzing five dimensions: People & Planet; Economic Context; Data & Input; AI Model; Tasks & Output.
In the following, we will briefly illustrate how defining the system’s features in each of these dimensions contributes to risk identification.
People & Planet
According to the OECD definition, this dimension “identifies individuals and groups that interact with or are affected by an applied AI system.”
It also evaluates the impact of an AI system on “human rights, the environment, well-being, society and the world of work.” Investigating this
dimension is important to understand the consequences of a specific AI application for human rights and democratic values such as safety and security,
physical and psychological integrity, freedom of expression and association, equality and non-discrimination, and so on. For example, if we consider
an AI-driven credit scoring system, concerns will include the potential of creating unequal access to credit and the subsequent effect on the
financial well-being of individuals. Such concerns should be addressed through regular audits ensuring decisions are unbiased and fairly allocated.
Economic context
This encompasses, among other things, the economic sector in which the system is deployed (e.g. insurance, automotive, healthcare), its business
function, impact, and scale. The OECD highlights that the policy implications of deploying AI systems vary significantly from one sector to another.
Depending on the sector, the application of the same task in different functional areas has a different impact on economic and social benefits,
jobs and skills, education, safety, and so on. Moreover, in critical sectors such as energy, transport, water, health, digital infrastructure,
and finance, critical functions are accompanied by heightened risk considerations. In our example, the impact of the credit scoring system on
the banking sector and on credit assessment roles and procedures must be evaluated. In response to potential challenges, possible measures include
implementing strategies for job displacement or retraining, as well as ensuring proper oversight and adherence to AI and credit assessment
regulations. In addition, proactive loss mitigation plans may be developed to manage potential financial consequences resulting from system errors.
Data and Input
Following the OECD classification, characteristics of this dimension include “the provenance of data and inputs, machine and/or human collection method,
data structure and format, and data properties.” Models require good data, yet collecting extensive and detailed data often conflicts with individual privacy.
Moreover, data collection methodologies can be associated with risks concerning working conditions of digital laborers, consumer protection, system’s biases,
transparency and explainability, resource consumption, safety, and robustness. Considering the credit scoring use case, concerns arise from potential biases present in
data due to socio-economic factors, and the overall robustness of the credit scoring model given the quality of data used. Measures to tackle these issues
include ensuring data collection adheres to regulatory standards and customer agreements, as well as implementing bias monitoring and elimination techniques.
AI Model
This is a representation of the environment of an AI system. According to the OECD, this encompasses processes, objects, ideas, people and/or interactions
taking place in its deployment and application context. For example, expert knowledge and performance measures used for training and optimization,
or the objectives set while monitoring generated output, are elements of an AI model. Also, in this case, the system’s explainability, robustness,
fairness, and resilience to cyberattacks can vary depending on AI model configurations and features. For instance, a model having no completely
centralized training dataset better addresses data protection issues than a model having a centralized one; having randomness elements in a model
can affect results’ reproducibility; using deep neural networks having low-interpretability can impact explainability; and so on. In our credit
scoring example, some of the main concerns include opacity issues deriving from AI model complexity, as well as security vulnerabilities leading
to data breaches. To address these issues, more interpretable models should be preferred as long as they don’t undermine system robustness and
data protection, and periodic cybersecurity assessments should be performed.
Tasks & Output
The last dimension proposed by the OECD refers to the specific tasks the system performs and considers its concrete outputs and resulting actions.
Examples of tasks include image recognition, predicting future values for data, detection of events such as fraud or human mistakes, content recommendation,
sentiment analysis, and so on. Depending on the task and the area of application, the OECD remarks that specific actions might be required to mitigate
ethical risks. For instance, keeping a human in the loop may be important for accountability in forecasting, while higher transparency and disclosure
of the fact that the user is interacting with an AI system might be required in interaction support tasks (e.g. chatbots). In our example, one of the
core concerns is ensuring the accuracy and reliability of the credit scores generated by the AI system. There could also be issues arising due to a
lack of explanation for scores, leading to potential miscommunication. Measures to address these concerns include regular performance audits, as well
as implementing a review procedure providing explanations about how the credit score was calculated and allowing individuals to challenge their credit scores.
To complete the analysis of possible risk scenarios, it is important to assess in detail the stakeholders impacted by the AI system and how.
Following the insights of the GPAI white paper “Data Justice in Practice”, we distinguish four phases of a stakeholder assessment.
Stakeholder Identification
The stakeholder identification process involves identifying all individuals and social groups that may impact or be impacted by the
AI application. As part of this process, it should be determined if any of these stakeholders possess sensitive or protected characteristics
that may increase their vulnerability to abuse, adverse impact, or discrimination. Furthermore, the procedure entails evaluating whether the
AI system presents significant concerns to specific groups of stakeholders due to vulnerabilities caused by the system’s use.
Scoping potential stakeholder impacts
This process entails an evaluation of how the AI application may affect a set of core ethical priorities, including the respect of human dignity,
protection of human freedom and autonomy, prevention of harm, fairness and non-discrimination, data protection, and respect of private and family
life. The goal is to foresee positive and negative outcomes, including any harm that could arise if the AI system malfunctions or produces unintended consequences.
Analyzing stakeholder salience
This involves evaluating which stakeholder groups are most likely to be positively or negatively impacted by the AI system, bearing in mind their specific needs.
This includes understanding the dynamics that could influence how benefits and risks are distributed among stakeholders. Particular attention is paid to those
stakeholders whose limited influence might impede their ability to share in the AI system's benefits or protect themselves from its potential risks.
The objective is to ensure a fair and equitable consideration of all parties involved or affected by the AI system.
Determining Stakeholder Engagement Methods
This last step considers possible ways of collecting meaningful stakeholder feedback on the AI application and including the perspective of
impacted parties in the implementation of the AI solution. It involves recognizing and accommodating the needs of diverse stakeholders, particularly
addressing participation barriers for vulnerable groups. Engagement methods range from highly interactive and co-creative approaches, such as
collaborative activities where stakeholders help shape discussions and decision-making, to more informative and consultative ones, such as online surveys
and interviews, newsletters, and forums. Choosing the right engagement methods is critical to ensure that these solutions are accessible, considerate
of different levels of understanding, and effective in facilitating meaningful participation.
Summing up, our key takeaway is that the ethical risks of AI systems are closely linked to their specific functions and the individuals and groups they impact.
By carefully classifying these systems and understanding the stakeholders involved, we pave the way for more informed risk assessments.
As we move to our next topic, we'll tackle the specific areas where we can audit these risks, making sure AI works responsibly and benefits everyone.
Stay tuned for an insightful look at how we can keep our AI systems in check and ethically sound.
