13.2.1 Understanding Offloading in Education

In order to better analyze how AI is offloading human learning and teaching, two different models can be used.Footnote ²² First, the Detect-Diagnose-Act Framework distinguishes three mechanisms underlying the functioning of AI in education (see Figure 13.1). In detect, the data that AI uses to understand student learning or teacher teaching are made explicit. The constructs AI analyses to understand the learning or teaching process are outlined in the diagnosis. Finally, act describes how the diagnostic information is translated into didactic pedagogical action. For example, an ALT for mathematical learning uses the learners’ answers to questions as input to diagnose a learner’s knowledge of a specific mathematical topic.Footnote ²³ This insight is used to adjust the difficulty level of problems provided to the learner and to determine how a learner should continue to practice this topic. Below, we provide an example of how this can look like in practice under “Case 1.”

Figure 13.1 Detect-Diagnose-Act FrameworkFootnote ²⁵

From the teaching perspective, the task of adjusting problems to students’ individual needs is offloaded to ALT. The technology and the teachers share the task of determining when a learner has reached sufficient mastery. Although these technologies support teachers,Footnote ²⁴ it is important to ensure that teachers stay in control. From the learner’s perspective, the need to monitor and control learning is reduced as the technology supports learning by adjusting the problem’s difficulty, which decreases the need for learners to self-regulate their learning and may affect the development of these skills.Footnote ²⁶

In this way, the Detect-Diagnose-Act Framework helps analyze offloading by AI, illustrating how particular AI solutions function in educational arrangements. At the same time, this model only describes the AI’s roles and largely ignores the roles of learners and teachers. Here the six levels of the automation model can be used to understand the division of control between AI, learners, and teachers in education. This model distinguishes six levels of automation in which the degree of control gradually transfers from the teacher to the AI system. The model starts with full teacher control and ends with full automation or AI control (see Figure 13.2). Hence the model goes from no offloading to AI to full offloading.

Figure 13.2 Six Levels of Automation ModelFootnote ²⁷

This model includes elements from the detect-diagnose-act framework. The input lines at the bottom represent detection and data collection in intelligent technologies. The data forms the basis for the AI system to diagnose meaningful constructs for learning and teaching, as described earlier. Hence, more data and different data streams are required for further automation. The diagnostic information is consequently transformed into different pedagogical didactical actions that can be taken in response. The main focus of this model is to make explicit which actors, that is, teachers, learners, or the AI system, perform those actions. This largely determines the position of an educational arrangement with AI on the model.

This model has distinct levels of automation at which AI can execute actions.

First, the AI system can provide information and describe student behavior without taking over any control (teacher assistance level). The information provided is known to impact teacher behavior.Footnote ²⁸ It can be communicated in different forms describing, guiding, and even recommending particular actions.Footnote ²⁹ Second, the AI system can enact simple actions during learning. These actions typically are at three levels: the step level providing feedback within a problem, the task level adjusting the task to the student’s needs, or the curriculum level optimizing the order of learning topics. In this partial automation level, AI only takes over tasks at one particular level, either enacting step, task, or curriculum adaptivity in interaction with the learner. In Case 1, an example of task adaption is outlined. In conditional automation, multiple tasks are taken over by AI, which can be a combination of different levels of adaptivity. With the transition of tasks to the AI system, the importance of the interface between the system and the teacher increases. For teachers to orchestrate the learning scenarios in the classroom, AI must inform the teacher adequately about the actions taken. Hence coordination between AI and humans becomes more critical. In high automation, control transfers primarily to AI and teachers step in only for specific tasks. Teacher actions are needed in case AI does not oversee the learning context. Here AI steers learning to a large extent. Finally, in full automation, the system autonomously supports learning and teaching without human control.

This model is functional for describing the augmentation perspective of AI in education, positioning the current role of AI in education, and discussing the future development of the role of AI in education. It can also help foster the discussion about the envisioned role of AI in education, in which it should be made explicit that the goal is not to reach full automation. Successful augmentation requires an ongoing interaction between humans and AI, and the interface between humans and AI is critical.Footnote ³⁰ The Detect-Diagnose-Act Framework and the Six Levels of Automation Model help to understand offloading by AI in specific educational arrangements and analyze the implications of AI in education more broadly. These insights can help teachers and educational professionals understand different applications of AI in the educational domain, allow scientists from different disciplines to compare use cases and discuss implications, and enable companies to position their products in the EdTech market.

The current generation of ALTs uses data on student performance to adapt problems to learners’ predicted knowledge levels and to provide additional information on their progress in teachers’ dashboards. These technologies enable more efficient teaching of foundational skills, and free time to focus on more complex problem-solving, self-regulation, and creativity. Adaptive learning technologies offer advantages, including advanced personalization of practice materials tailored to each student’s needs and the ability for teachers to devote more time to tasks beyond the reach of technology, such as providing elaborate feedback or helping students who need additional instruction. This case represents an example of partial automation, in which the teacher and the ALT work closely together. The functions of the ALT are to describe, diagnose, and advise the teacher through the dashboards based on ongoing student activities and, in specific cases, to select student problems. Teachers continue to control most organizational tasks in this learning scenario and remain responsible for monitoring the functioning of the technology, in which teacher dashboards play an important role. ALTs are one example of AI in education, below we provide an overview applications.

13.2.2 Applications of AI in Education

Generally, applications of AI in education can be divided into student-faced, teacher-faced, and administrative AI solutions, depending on the actor/stakeholder they support.Footnote ³² Below the most commonly used AI systems of each type are shortly outlined.

13.3.1 Overview of AI Ethics Frameworks

The mid-2010s saw a surge in AI ethics discussions, spurred by rapid advances in deep learning and growing controversies surrounding the technology’s implications. More specifically, the years between 2016 and 2019 have seen the proliferation of AI ethics guidelines issued by technology companies, NGOs, think tanks, international organizations, and research institutions.Footnote ⁴⁸ Jobin et al.Footnote ⁴⁹ analyzed 84 published sets of ethical principles for AI, which they concluded converged on five areas: transparency, justice and fairness, non-maleficence, responsibility, and privacy. Similarly, a comparative analysis by Fjeld et al.Footnote ⁵⁰ identified an emerging normative core comprised of 8 key themes: privacy, accountability, safety and security, transparency and explainability, fairness and nondiscrimination, human control of technology, professional responsibility, and the promotion of human values. While this convergence may be seen as a sign of maturation and a key step for the development of binding rules and laws, a review by Blair Attard-Frost et al.Footnote ⁵¹ revealed a disproportionate emphasis on principles intended for the governance of algorithms and technologies and little attention to the ethics of business practices and the political economies within which AI technologies are embedded. These latter aspects are of key importance in the context of education, given that the adoption of AI in schools can accelerate the commodification of education and further embed large private tech companies into the provision of public goods.Footnote ⁵²

In recent years the AI ethics discussion gradually moved from the enumeration of key values toward efforts to translate abstract principles into real-world practices. However, this is wrought with several difficulties, and the field is currently exploring various approaches.Footnote ⁵³ For example, at the time of writing, the OECD’s Policy Observatory cataloguesFootnote ⁵⁴ over 500 procedural, educational, and technical tools intended to support trustworthy and responsible AI development. However, there is currently little evidence about this uptake and impact. A 2021 review of AI impact assessments and audits concluded that most approaches suffered from a lack of stakeholder participation, failed to utilize the full range of possible techniques and that internal self-assessment methods exhibited scarce external verification or transparency mechanisms.Footnote ⁵⁵

Finally, in addition to developments in AI ethics, there has been increasing regulatory attention in several jurisdictions, including the EU,Footnote ⁵⁶ the UK,Footnote ⁵⁷ the United States,Footnote ⁵⁸ and China, along with calls for international harmonization. The European Union adopted the world’s first comprehensive regulation, the AI Act, in July 2024, which enshrines several previously voluntary ethical principles into law. As a result, schools will need to implement a comprehensive AI governance strategy to adequately deal with transparency, data protection and risk assessment requirements. The law also classifies certain uses of AI in education as high risk, including systems that determine access to educational institutions, determine the appropriate education level for students, evaluate learning outcomes, or monitor students for prohibited behaviour during tests. These use-cases are subject to additional regulatory requirements.Footnote ⁵⁹

Still, AI represents a uniquely difficult technology for lawmakers to regulate.Footnote ⁶⁰ Given the pace, potential scale, and complexity of AI’s societal impacts, ethical frameworks, guidelines, and tools for responsible technology development will likely continue to evolve alongside regulatory efforts.

13.3.2 Ethics of AI in Education

When AI is applied in the domain of education, it may substitute, augment, modify, or redefine existing educational practices.Footnote ⁶¹ Consequently, the ethics of AI in education should not just be based on an ethics of AI, but also based on an ethics of education.Footnote ⁶² Aiken and Epstein set the stage for ethics in AI education back in 2000. They advocated for focusing on human needs rather than letting technology dictate decisions.Footnote ⁶³ They considered a multidimensional view of humans, looking at ethical, social, intellectual, and other aspects. This laid the groundwork for today’s human-centered AI ethos.Footnote ⁶⁴ Aiken and Epstein’s guidelines emphasized positive, adaptive AI that supported diverse learning approaches and cultures, respected teachers and underscored the human role in education. However, principles we commonly see in AI ethics guidelines today, such as transparency, explainability, and avoiding bias, are notably absent, as these emerged later due to the rise of data-driven deep learning systems. Despite the changes in AI technologies, Aiken and Epstein’s principles still remind us to prioritize human-centered educational values in AI development and align well with the augmentation perspective on AI in education.

13.4.1 The Value Compass for the Digital Transformation of Education

Within the Netherlands, there has been an increasing discussion of the impact of digital technologies on education. This has been symbolized by a call for action in a national newspaper by the rector magnifici of all Dutch public universities,Footnote ⁸¹ warning about the influence of large tech corporations on the public educational system and calling for the higher education sector to take responsibility for its public values. This can be seen as the start of a national discussion on prioritizing educational values in public education, which was reflected in a subsequent advisory report on public values in education by the Dutch Association of Universities (UNL).Footnote ⁸² Public values are the values that ground and give meaning to our interactions, societal institutions and political structures.Footnote ⁸³ Public values are not fixed, but are the result of continuous societal and political processes.Footnote ⁸⁴ And the public interests they represent are of such importance that they need to be safeguarded within the public sector.Footnote ⁸⁵ The underlying thought is that the digitalization of public services needs to be guided by “fundamental public values such as privacy, autonomy, equity and equality.”Footnote ⁸⁶

As technologies become more pervasive in educational institutions, we see, on the one hand, how these technologies start to shape educational practices and, on the other hand, how the dependency on existing software providers can become problematic. AI applications in education only compound those challenges, raising questions about how the roles of students and teachers change and which new responsibilities emerge for educational institutions.Footnote ⁸⁷ To facilitate navigating the influence of digital technologies in education, Kennisnet,Footnote ⁸⁸ SURF,Footnote ⁸⁹ and the Rathenau InstituteFootnote ⁹⁰ developed a Value Compass as a reference framework for public values in education (see Figure 13.3).Footnote ⁹¹ This common language aims to elicit a discussion that transcends functionalities, costs and benefits toward formulating shared ambitions for a future of digital education.

Figure 13.3 The value compassFootnote ⁹²

The value compass emphasizes three core values as central to educational practices: autonomy, justice, and humanity. These values are loosely defined as constellations of other values, such as privacy (autonomy), inclusivity (justice), and well-being (humanity). Here autonomy is seen as the ability “to live under your own laws” for diverse educational stakeholders, the students and teachers, and the institutions themselves.Footnote ⁹³ Justice is defined as collected values that mainly describe the importance of treating others in terms of themselves and treating them equally in an inclusive manner.Footnote ⁹⁴ The human aspect of education is central to the value of humanity, consisting of the meaningful contact, respect, safety, and well-being necessary to value the unique aspects of each student.Footnote ⁹⁵ The framework was developed through deliberative engagement with sector stakeholders, including a published beta version for public consultation, as can be read in the Rathenau reportFootnote ⁹⁶ which describes in more detail how this compass of values was developed. Working from a previous list of seven key themes and questions around the digital transformation of the public sector,Footnote ⁹⁷ the value compass conceptualizes themes such as power relations and control over technology into values such as autonomy. The three main values of autonomy, justice, and humanity can be seen as equally important, with underlying instrumental values that allow the operationalization of these values.

The Value Compass is not a normative framework for digital transformation but a basis for considering educational values in decision-making.Footnote ⁹⁸ The value compass is used for deliberative workshops, for example, a workshop of SURF with the national student bodies ISO and LSVB on the ethics of using online proctoring in examinations.Footnote ⁹⁹ Here the values in the value compass helped guide the discussion through all relevant perspectives. A more normative approach can be achieved through value hierarchies, where one conceptualizes values into norms and design requirements to guide choices in digital transformation.Footnote ¹⁰⁰ For example, Kennisnet conceptualized the value of inclusivity for a digital learning system into the norm “accessible for all students,” which led to the design requirement of “meets web accessibility requirements.” These requirements can then be taken into account within the development or procurement processes. Through a lens of public values, educational institutions can proactively structure the digital transformation by “weighing values,” using them as a guide in shaping considerations and priorities.Footnote ¹⁰¹ By looking at the design, procurement, and use of new technologies through the lens of these values, the education sector can become an active participant in the digitalization of education.Footnote ¹⁰²

13.4.2 The NOLAI as an Approach to Drive Responsible Use of AI in Education

To finalize this chapter, we introduce the example of the National Education Lab AI (NOLAI)Footnote ¹⁰³ in the Netherlands, which pursues an embedded ethics approach to iteratively develop, prototype, implement, evaluate and scale responsible AI technologies for primary and secondary education.

NOLAI is an innovative research initiative at the Faculty of Social Sciences of Radboud University in the Netherlands in collaboration with several strategic partnersFootnote ¹⁰⁴ and the Dutch Growth Fund.Footnote ¹⁰⁵ NOLAI’s main goal is to develop intelligent technologies that improve the quality of primary and secondary education. The institute aims to achieve this goal by developing innovative prototypes that use AI and promoting the responsible use of AI in education. This is done in two programs: the co-creation program and the scientific program. The co-creation program develops innovative prototypes and applications of AI in co-creation with schools, scientists, and businesses. The scientific program develops knowledge in five focus areas: teacher professionalization, technology for AI in education, sustainable data, pedagogy & didactics and embedded ethics.

NOLAI’s main activities are developing state-of-the-art applications of AI in education and investigating their use NOLAI’s activities start with dialogues with schools to explore their needs for using AI to improve education and literature reviews that, map current knowledge. After, NOLAI brings scientists and educational practitioners together to develop AI prototypes, explore current applications of AI technologies and ambitions for the future with businesses. An example of a co-creation project by NOLAI is the visualization of student data collected across different learning management, ALTs, and summative assessment systems. This project is a collaboration between three schools, an ALT company, an assessment company, and pedagogical and AI scientists. The collaborative and interdisciplinary approach ensures the connection between educational practice, science, and business development.

NOLAI stimulates the responsible development of AI in education. NOLAI has a dedicated Data and Privacy Officer who helps the co-creation projects comply with relevant privacy and data protection regulations. Also, all projects need approval from institutional ethical committees that monitor the ethical conduct of the research being conducted. In addition, as there are many open questions about the ethical issues that will emerge throughout the development and implementation of AI in education, NOLAI strives to further the discussions around the responsible use of AI in education in the Netherlands with its embedded ethics approach.

Embedded ethics approaches have most famously been developed in computer science education, where it is referred to as an approach for teaching ethics in computer science curricula and aims to incorporate ethics into the entire engineering process in an integrated, interdisciplinary, and collaborative way.Footnote ¹⁰⁶ As such, embedded ethics is aptly seen as an ongoing process of anticipating, identifying, and addressing ethical features of technological innovations by helping developers to integrate ethical awareness and critical reasoning in their technical projects, thereby benefitting individuals and society at large.Footnote ¹⁰⁷

The embedded ethics approach developed within NOLAI complements such existing approaches with for example, insights from the “ethics parallel research” approach that has been developed to provide ethical guidance parallel to the development process of emerging biomedical innovations.Footnote ¹⁰⁸ This last approach has many similarities with the embedded ethics approaches as it can also be characterized by focusing on bottom-up, inductive ethical dilemmas and stimulating ethical reflexivity and awareness. An important difference is that the ethics parallel research approach argues for the inclusion of a wider variety of stakeholders in the deliberation process (beyond engineers and ethicists) and is less focused on the design of technology but also its broader sociopolitical implications.

This means that for the embedded ethics approach within NOLAI, ethicists will closely collaborate with various stakeholders in co-creation projects, including technologists, company representatives, scientists, and educational professionals. As the co-creation projects develop and mature, the ethicists aim to provide ethical support and develop sustainable processes to advance responsible innovation, navigating the “messy” reality of the co-creation projects and the ethical questions, complex dilemmas, and practices that emerge. This means they will advise stakeholders and support them with anticipating, identifying, and addressing moral dilemmas iteratively and continuously.

In addition to ethical literature and theory, ethicists within NOLAI will conduct empirical research within the co-creation projects to inform and advance their ethical support. Through various qualitative research studies, including participant observations, focus groups, and surveys, the ethicists will study the effects and implications of introducing AI systems in education. For example, they will study students’ and teachers’ moral beliefs, intuitions, and reasoning using AI systems within NOLAI. These findings can help align the AI systems with students’ and teachers’ needs and wishes. In another study, ethicists will use qualitative research methods to explore value conflicts that emerge when AI systems are introduced in classrooms and how these conflicting values are balanced in practice. The findings help formulate “best practices” regarding implementing AI systems in education.

As a last step, the ethicists within NOLAI will use the insights gained from their participation in the co-creation projects and the results of their empirical studies to inform more abstract ethical debates about AI in education. The diversity and large amounts of co-creation within NOLAI provide an exceptional opportunity to help answer complex ethics questions outlined in this chapter.

14.2.1 AI in Media Content Gathering and Production

Beckett’s survey of journalism and AI presents an impressive list of possible AI uses in day-to-day journalistic practice.Footnote ⁷ At the beginning of the news-creating process, AI can help gather material, sift through social media, recognize genders and ages in images, or automatically add tags for newspaper articles with topics or keywords.Footnote ⁸

AI is also used in story discovery to identify trends or spot stories that could otherwise be hard to grasp by the human eye and to discover new angles, voices, and content. To illustrate, already in 2014, Reuters News Tracer project used natural language processing techniques to decide which topics are newsworthy.Footnote ⁹ It detected the bombing of hospitals in Aleppo and the terror attacks in Nice and Brussels before they were reported by other media.Footnote ¹⁰ Another tool, the Topics Compass, developed under EU-funded Horizon 2020 ReTV project, allows an editorial team to track media discourse about a given topic coming from news agencies, blogs, and social media platforms and to visualize its popularity.Footnote ¹¹

AI has also been proven useful in investigative journalism to assist journalists in tasks that could not be done by humans alone or would have taken a considerable amount of time. To illustrate, in a cross-border Panama Papers investigation, the International Consortium of Investigative Journalists used an open-source data mining tool to sift through 11.5 million whistleblowers’ documents.Footnote ¹²

Once journalists have gathered information on potential stories, they can use AI for the production of news items: text, images, and videos. Media companies such as the Associated Press, Forbes, and The New York Times have started to automate news content.Footnote ¹³ Terms like robot journalism, automated journalism, and algorithmic journalism have been used interchangeably to describe this phenomenon.Footnote ¹⁴ In addition, generative AI tools such as ChatGPT,Footnote ¹⁵ Midjourney,Footnote ¹⁶ or DALL-EFootnote ¹⁷ are being used to illustrate news stories, simplify text for different audiences, summarize documents, or writing potential headlines.Footnote ¹⁸

14.2.2 AI in Media Content DistributionFootnote ¹⁹

Media organizations can also use AI for providing personalized recommendations. Simply put, “recommendation systems are tools designed to sift through the vast quantities of data available online and use algorithms to guide users toward a narrower selection of material, according to a set of criteria chosen by their developers.”Footnote ²⁰

In recent years, online news media (e.g., online newspapers’ websites and apps) started engaging in news recommendation practices.Footnote ²¹ Recommendation systems curate users’ news feed by automatically (de)prioritizing items to be displayed in user interfaces, thus deciding which ones are visible (to whom) and in what order.Footnote ²²

The 2022 Ada Lovelace reportFootnote ²³ provides an informative in-depth snapshot of the BBC’s development and use of recommendation systems, which gives insights into the role of recommendations in public service media (PSM).Footnote ²⁴ As pointed out by the authors, developing recommendation systems for PSM requires an interrogation of the organizations’ role in democratic societies in the digital age, that is, how to translate the public service valuesFootnote ²⁵ into the objectives for the use of recommendation systems that serve the public interest. The report concludes that the PSM had internalized a set of normative values around recommendation systems: Rather than maximizing engagement, they want to broaden their reach to a more diverse set of audiences.Footnote ²⁶ This is a considerable difference between the public and private sectors. Many user-generated content platforms rank information based on how likely a user is to interact with a post (comment on it, like it, reshare it) or to spend more time using the service.Footnote ²⁷

Research shows that social media platforms are using a mix of commercial criteria, but also vague public interest considerations in their content prioritization measures.Footnote ²⁸ Importantly, prioritization of some content demotes other.Footnote ²⁹ As a way of example, Facebook explicitly says it will not recommend content that is associated with low-quality publishing, including news that it is unclear about its provenance.Footnote ³⁰ In fact, online platforms use a whole arsenal of techniques to (de)amplify the visibility or reach of some content.Footnote ³¹ To illustrate, in an aftermath of Russian aggression on Ukraine, platforms announced they would restrict access to RT and Sputnik media outlets.Footnote ³² Others have also been adding labels and started reducing the visibility of content from Russian state-affiliated media websites even before the EU-imposed sanctions.Footnote ³³

Overall, by selecting and (de)prioritizing news content and deciding on its visibility, online platforms take on some of the functions so far reserved to traditional media.Footnote ³⁴ Ranking functions and optimization metrics in recommendation systems have become powerful determinants of access to media and news content.Footnote ³⁵ This has consequences for both the fundamental right to freedom of expression and media freedom (see Section 14.3).

14.2.3 AI in Fact-Checking

Another important AI potential in media is fact-checking. The main elements of automated fact-checking are: (1) identification of false or questionable claims circulating online; (2) verification of such claims, and (3) (real-time) correction (e.g., flagging).Footnote ³⁶

To illustrate, platforms such as DALIL help fact-checkers spot questionable claims which then require subsequent verification.Footnote ³⁷ Then, to verify the identified content, the AI(-enhanced) tools can perform a reverse image search, detect bot accounts and deep fakes, assess source credibility, check nonfactual statements (claims) made on social media or analyze the relationships between accounts.Footnote ³⁸ WeVerify plug-in is a highly successful tool which offers a variety of verification and analysis features in one platform to fact-check and analyze images, video, and text.Footnote ³⁹ Some advanced processing and analytics methods can also be used to analyze different types of content and apply a trustworthiness scoring to online articles.Footnote ⁴⁰

The verified mis- or disinformation can then be flagged to the end user by adding warnings and providing more context to content rated by fact-checkers. Some platforms have also been labeling content containing synthetic and manipulated media.Footnote ⁴¹

Countering disinformation with the use of AI is a growing research area. The future solutions based on natural language processing, machine learning, or knowledge representation are expected to deal with different content types (audio, video, images, and text) across different languages.Footnote ⁴² Collaborative tools that enable users to work together to find, organize, and verify user-generated content are also on the rise.Footnote ⁴³

14.2.4 AI in Content Moderation

AI in content moderation is a broad topic. Algorithmic (commercial) content moderation can be defined as “systems that classify user-generated content based on either matching or prediction, leading to a decision and governance outcome (e.g., removal, geoblocking, and account takedown).”Footnote ⁴⁴ This section focuses on the instances where AI is used either by media organizations to moderate the discussion on their own sites (i.e., in the comments section) or by social media platforms to moderate posts of media organizations and journalists.

14.3.1 Lack of Data Availability

Lack of data availability is a cross-cutting theme, with serious consequences for the media sector. Datasets are often inaccessible or expensive to gather and data journalists rely on private actors, such as data brokers which have already collected such data.Footnote ⁶³ This concentrated control over the data influences how editorial decision-making is automated (see Section 14.3.6).

Data availability is also of paramount importance for news verification and fact-checking activities. Access to social media data is vital to analyze and mitigate the harms resulting from disinformation, political microtargeting, or the effect of social media on elections or children’s well-being.Footnote ⁶⁴ This is because it enables journalists and researchers to hold platforms accountable for the working of their AI systems. Equally, access to, for example, social media data is important for media organizations that are developing their own AI solutions – particularly in countries where it can be difficult to gain access to large quantities of data in the local language.Footnote ⁶⁵

Access to platforms’ data for researchers is currently mainly governed by contractual agreements, platforms’ own terms of service, and public application programming interfaces (APIs). Application programming interfaces access can be restricted or eliminated at any time and for any reason.Footnote ⁶⁶ The UN Special Rapporteur on the promotion and protection of the right to freedom of opinion and expression stressed a lack of transparency and access to data as “the major failings of companies across almost all the concerns in relation to disinformation and misinformation.”Footnote ⁶⁷

A key challenge for research access frameworks is to comply with the General Data Protection Regulation (GDPR).Footnote ⁶⁸ Despite a specific derogation for scientific research purposes (art. 89), the GDPR lacks clarity regarding how platforms might share data with researchers (e.g., on what legal grounds).Footnote ⁶⁹ To mitigate this uncertainty, various policy and regulatory initiatives aim to clarify how platforms may provide access to data to researchers in a GDPR-compliant manner.Footnote ⁷⁰ In addition, there have been calls for a legally binding mechanism that provides independent researchers with access to different types of platform data.Footnote ⁷¹

The Digital Services Act (DSA) requires providers of very large online platforms (VLOPs) and very large online search engines (VLOSEs) to grant vetted researchers access to data, subject to certain conditions.Footnote ⁷² Data can be provided “for the sole purpose” of conducting research that contributes to the detection, identification and understanding of systemic risks and to the assessment of the adequacy, efficiency and impacts of the risk mitigation measures (art. 40(4)). Vetted researchers must meet certain criteria and procedural requirements in the application process. Importantly, they must be affiliated to a research organization or a not-for-profit body, organization or association (art. 40(12)). Arguably, this excludes unaffiliated media practitioners, such as freelance journalists or bloggers. Many details about researchers’ access to data through the DSA will be decided in delegated acts that have yet to be adopted (art. 14(13)).

Moreover, under the Digital Markets Act,Footnote ⁷³ the so-called gatekeepers will have to provide advertisers and publishers with access to the advertising data and allow business users to access the data generated in the context of the use of the core platform service (art. 6(1) and art. 6(8)).

Furthermore, the European strategy for dataFootnote ⁷⁴ aims at creating a single market for data by establishing common European data spaces to make more data available for use in the economy and society. The Data Governance ActFootnote ⁷⁵ and the Data Act proposalFootnote ⁷⁶ seek to strengthen mechanisms to increase data availability and harness the potential of industrial data, respectively. Lastly, the European Commission announced the creation of a dedicated media data space.Footnote ⁷⁷ The media data space initiative, financed through the Horizon Europe and Digital Europe Programmes,Footnote ⁷⁸ aims to support both PSM and commercial media operators to pool their content and customer data to develop innovative solutions.

14.3.2 Data Quality and Bias in Training Datasets

Another, closely related, consideration is data quality. There is a growing literature on the quality and representation issues with training, testing, and validation data, especially those in publicly available datasets and databases.Footnote ⁷⁹ Moreover, generative AI raises controversies regarding the GDPR compliance of the training dataFootnote ⁸⁰ and brings a broader question of extraction fairness, defined as “legal and moral concerns regarding the large-scale exploitation of training data without the knowledge, authorization, acknowledgement or compensation of their creators.”Footnote ⁸¹

The quality of training data and data annotation is crucial, for example, for hate speech and abusive language detection in comments. A 2022 report by the EU Agency for Fundamental Rights shows how tools that automatically detect or predict potential online hatred can produce biased results.Footnote ⁸² The predictions frequently overreact to various identity terms (i.e., words indicating group identities like ethnic origin or religion), flagging text that is not actually offensive.Footnote ⁸³ Research shows that social media content moderation algorithms have difficulty differentiating hate speech from discussion about race and often silence marginalized groups such as racial and ethnic minorities.Footnote ⁸⁴ At the same time, underrepresentation of certain groups in a training dataset may result in them experiencing more abusive language than other groups.

There are blurred lines between what constitutes hateful, harmful, and offensive speech, and these notions are context dependent and culturally specific. Many instances of hate speech cannot be identified and distinguished from innocent messages by looking at single words or combinations of them.Footnote ⁸⁵ Such contextual differentiation, between, for example, satirical and offensive uses of a word proves challenging for an AI system. This is an important technical limitation that may lead to over- and under-removal of content. Both can interfere with a range of fundamental rights such as the right to freedom of expressionFootnote ⁸⁶ (see Section 14.3.4), the right to data protection, as well as the right to nondiscrimination.

The consequence of using unreliable data could be the spread of misinformationFootnote ⁸⁷ as illustrated by inaccurate responses to news queries from search engines using generative AI. Research into Bing’s generative AI accuracy for news queries shows that there are detail errors and attribution errors, and the system also sometimes asserts the opposite of the truth.Footnote ⁸⁸ This, together with the lack of media literacy, may cause an automation bias, that is, the uncritical trust in information provided by the automated system despite the information being actually incorrect.

14.3.3 Transparency

Transparency can mean many different things. Broadly speaking, it should enable people to understand how an AI system is developed, is trained, how it operates, and how it is deployed so that they can make more informed choices.Footnote ⁸⁹ This section focuses on three aspects of transparency of AI in media.Footnote ⁹⁰

The first aspect relates to internal transparency, which describes the need for journalists and other non-technical groups inside media organizations to have sufficient knowledge around the AI systems they use.Footnote ⁹¹ The importance of closing the intelligibility gap around AI within a media organization is necessary for an understanding of how AI systems work to use them responsibly.Footnote ⁹² The AI Act requires providers and deployers of AI systems including media organizations to ensure, to their best extent, a sufficient level of AI literacy of their staff and other persons dealing with the operations and use of AI systems on their behalf (art. 4).Footnote ⁹³

The second aspect concerns external transparency, which refers to transparency practices directed toward the audience to make them aware of the use of AI. The AI Act requires providers of AI systems, such as OpenAI, to make it clear to users that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use (art. 50(1)).Footnote ⁹⁴ As a rule, they must also mark generative AI outputs (synthetic audio, image, video or text content) as AI-generated or manipulated (art. 50(2)). For now it remains unclear what forms of transparency will be sufficient and whether they will be meaningful to the audience. Transparency requirements also apply to those who use AI systems that generate or manipulate images, audio or video content constituting a deep fake (art. 50(4) para 1). However, if the content is part of an evidently artistic, creative or satirical work, the disclosure should not hamper the display or enjoyment of the work. Moreover, deployers of an AI-generated or manipulated text which is published with the purpose of informing the public on matters of public interest shall disclose that the text has been artificially generated or manipulated. There is an important exception for the media sector. If the AI-generated text has undergone a process of human review or editorial control within an organisation that holds editorial responsibility for the content (such as a publisher), disclosure is no longer necessary. This provision raises questions as to what will count as a human review or editorial control and who can be said to hold editorial responsibility.Footnote ⁹⁵ Moreover, research shows that audiences want media organisations to be transparent and provide labels when using AI.Footnote ⁹⁶

In addition to the AI Act, the DSA presents multiple layers of transparency obligations for the benefit of users that differ depending on the type of service concerned.Footnote ⁹⁷ In particular, it requires transparency on whether AI is used in content moderation. All intermediary services must publish in their terms and conditions, in a “clear and unambiguous language,” a description of the tools used for content moderation, including AI systems that either automate or support content moderation practices (art. 14). In practice, this means that users must know why, when, and how online content is being moderated, including with the use of AI, and when human review is in place.

The DSA also regulates recommender system transparency. As mentioned earlier, recommender systems can have a significant impact on the ability of recipients to retrieve and interact with information online. Consequently, providers of online platforms are expected to set out in their terms and conditions in plain and intelligible language the main parameters used in their recommender systems and the options for users to modify or influence them (art. 27). The main parameters shall explain why certain information is suggested, and include, at least, the criteria that are most significant in determining the information suggested, and the reasons for the relative importance of those parameters. There are additional requirements imposed on the providers of VLOPs and VLOSEs to provide at least one option for their recommendation systems which is not based on profiling.

There are also further obligations for VLOPs and VLOSEs to perform an assessment of any systemic risks stemming from the design, functioning, or use of their services, including algorithmic systems (art. 34(1)). This risk assessment shall include the assessment of any actual or foreseeable negative effects on the exercise of fundamental rights, including the right to freedom of expression and the freedom and pluralism of the media (art. 34(1)(b)). When conducting risk assessments, VLOPs and VLOSEs shall consider, in particular, whether the design of their recommender systems and their content moderation systems influence any of the systemic risks. If so, they must put in place mitigation measures, such as testing and adapting their algorithms (art. 35).

Lastly, intermediary services (excluding micro and small enterprises) must publish, at least once a year, transparency reports on their content moderation activities, including a qualitative description, a specification of the precise purposes, indicators of the accuracy and the possible rate of error of the automated means (art. 15). Extra transparency reporting obligations apply to VLOPs (art. 42).

The third aspect concerns third-party transparency, which refers to the importance of having insights into how AI systems provided by third-party providers have been trained on and how they work.Footnote ⁹⁸

In both the DSA and the AI Act, there are no explicit provisions that make such information widely available.Footnote ⁹⁹

14.3.4 Risks for the Right to Freedom of Expression

Article 10 of the European Convention of Human Rights (ECHR), as well as Article 11 of the Charter of Fundamental Rights of the European Union (CFR),Footnote ¹⁰⁰ guarantees the right to freedom of expression to everyone. The European Court of Human Rights (ECtHR) has interpreted the scope of Article 10 ECHR through an extensive body of case law. The right to freedom of expression includes the right to impart information, as well as the right to receive it. It protects the rights of individuals, companies, and organizations, with a special role reserved for media organizations and journalists. It is their task to inform the public about matters of public interest and current events and to play the role of the public watchdog.Footnote ¹⁰¹ The right applies offline and on the Internet.Footnote ¹⁰²

One of the main risks for freedom of expression associated with algorithmic content moderation is over-blocking, meaning the unjustified removal or blocking of content or the suspension or termination of user accounts. In 2012, the Court of Justice of the EU held that a filtering system for copyright violations could undermine freedom of information since it might not distinguish adequately between lawful and unlawful content, which could lead to the blocking of lawful communications.Footnote ¹⁰³ This concern is equally valid outside the copyright context. The technical limitations of AI systems, together with regulatory pressure from States who increasingly request intermediaries to take down certain categories of content, often based on vague definitions, incentivize platforms to follow a “if in doubt, take it down” approach.Footnote ¹⁰⁴ There is, indeed, growing empirical evidence of platforms’ over-removal of content.Footnote ¹⁰⁵ To illustrate, social media platforms have deleted hundreds of posts condemning the eviction of Palestinians from the Sheikh Jarrah neighborhood of JerusalemFootnote ¹⁰⁶ or restricted access to information about abortion.Footnote ¹⁰⁷ Both examples are a consequence of the algorithmic content moderation systems either not being able to recognize context or not knowing underlying facts and legal nuances. Such automated removals, even if unintentional and subsequently revoked, potentially limit both the right to impart information (of users who post content online) and the right to receive information (of third parties who do not get to see the deleted content).

On the other hand, the under-blocking of certain online content may also have a negative impact on the right to freedom of expression. Not acting against illegal content and some forms of legal but harmful content (i.e., hate speech) may lead people (especially marginalized communities) to express themselves less freely or withdraw from participating in the online discourse.

In addition, in the context of fact-checking, AI cannot yet analyze entire, complex disinformation narratives and detect all uses of synthetic media manipulation.Footnote ¹⁰⁸ Thus, an overreliance on AI systems to verify the trustworthiness of the news may prove detrimental to the right to freedom of expression.

To mitigate these risks, the DSA provides certain procedural safeguards. It does not force intermediary services to moderate content, but requires that any restrictions imposed on users’ content based on terms and conditions are applied and enforced “in a diligent, objective and proportionate manner,” with “due regard to the rights and legitimate interests of all parties involved” (art. 14(4)). Not only do they have to take due regard to fundamental rights in cases of content removal, but also when restricting the availability, visibility, and accessibility of information. What due regard means in this context will be defined in courts. Moreover, the DSA requires intermediary services to balance their freedom to conduct a business with other rights such as users’ freedom of expression. Online platforms also have to provide a statement of reasons as to why the content has been removed or the account has been blocked and implement an internal complaint-handling system that enables users to lodge complaints (art. 21). Another procedural option is the out-of-court dispute settlement or a judicial remedy.Footnote ¹⁰⁹

A novelty foreseen by the DSA is an obligation for VLOPs and VLOSEs to mitigate systemic risks such as actual or foreseeable negative effects for the exercise of fundamental rights, in particular freedom of expression and information, including the freedom and pluralism of the media, enshrined in Article 11 of the CFR, and foreseeable negative effects on civic discourse (art. 34).

News personalization from the freedom of expression perspective looks paradoxical at first glance. As Eskens points out, “news personalisation may enhance the right to receive information, but it may also hinder or downplay the right to receive information and the autonomy with which news users exercise their right to receive information.”Footnote ¹¹⁰ Given that content prioritization practices have a potential for promoting trustworthy and reliable news, it can be argued that platforms should be required to ensure online access to content of general public interest. The Council of Europe, for instance, suggested that States should act to make public interest content more prominent, including by introducing new obligations for platforms and intermediaries, and also impose minimum standards such as transparency.Footnote ¹¹¹ Legal scholars have proposed exposure diversity as a design principle for recommender systemsFootnote ¹¹² or the development of “diversity-enhancing public service algorithms.”Footnote ¹¹³ But who should decide what content is trustworthy or authoritative, and based on what criteria? Are algorithmic systems of private platforms equipped enough to quantify normative values such as newsworthiness? What safeguards would prevent States from forcing platforms to prioritize State-approved-only information or government propaganda? Besides, many of the problems with content diversity are at least to some extent user-driven – users themselves, under their right to freedom of expression, determine what kind of content they upload and share.Footnote ¹¹⁴ Legally imposed public interest content recommendations could limit users’ autonomy in their news selection by paternalistically censoring the range of information that is available to them. While there are no such obligations in the DSA, some legislative proposals at the national level are currently reviewing such options.Footnote ¹¹⁵

14.3.5 Threats to Media Freedom and Pluralism Online

Freedom and pluralism of the media are pillars of liberal democracies. They are also covered by Art. 10 ECHR and Art. 11 CFR. The ECtHR found that new electronic media, such as an online news outlet, are also entitled to the protection of the right to media freedom.Footnote ¹¹⁶ Moreover, the so-called positive obligations doctrine imposes an obligation on States to protect editorial independence from private parties, such as social media.Footnote ¹¹⁷

Social media platforms have on multiple occasions erased content coming from media organizations, including public broadcasters, and journalists. This is often illustrated by the controversy that arose around Facebook’s decision to delete a post by a Norwegian journalist, which featured the well-known Vietnam War photo of a nude young girl fleeing a napalm attack.Footnote ¹¹⁸ Similarly, users sharing an article from The Guardian showing Aboriginal men in chains were banned from Facebook on the grounds of posting nudity.Footnote ¹¹⁹ Other examples include videos of activists and local news outlets that documented the war crimes of the regime of Bashar al-Assad in SyriaFootnote ¹²⁰ or a Swedish journalist’s material reporting sexual violence against minors.Footnote ¹²¹ This is due to technical limitations of the algorithmic content moderation tools and their inability to distinguish educational, awareness raising or journalistic material from other content.

In order to prevent removals of content coming from media organizations, a so-called media exemptionFootnote ¹²² was proposed during the discussions of the DSA proposal, aiming to ensure that the media would be informed and have the possibility to challenge any content moderation measure before its implementation. The amendments were not included in the final text of the DSA. There is no special protection or any obligation of prior notice to media organizations in the DSA. Media organizations and journalists can invoke the same procedural rights that apply to all users of online platforms. One can also imagine that mass-scale algorithmic takedowns of media content, suspension or termination of journalists’ accounts by VLOPs could amount to a systemic risk in a form of a negative effect on the exercise of the freedom and pluralism of the media.Footnote ¹²³ However, what qualifies as systemic, and when a threshold of systemic risk to freedom and pluralism of the media is reached, remains undefined.

Recognizing media service providers’ role in the distribution of information and in the exercise of the right to receive and impart information online, the European Media Freedom Act (EMFA) grants media service providers special procedural rights vis-à-vis VLOPs. Where a VLOP considers that content provided by recognized media service providersFootnote ¹²⁴ is incompatible with its terms and conditions, it should “duly consider media freedom and media pluralism” in accordance with the DSA and provide, as early as possible, the necessary explanations to media service providers in a statement of reasons as referred to in the DSA and the P2B Regulation (recital 50, art. 18). In what has been coined as a non-interference principleFootnote ¹²⁵, VLOPs should provide the media service provider concerned, prior to the suspension or restriction of visibility taking effect, with an opportunity to reply to the statement of reasons within 24 hours of receiving it.Footnote ¹²⁶ Where, following or in the absence of a reply, a VLOP takes the decision to suspend or restrict visibility, it shall inform the media service provider concerned without undue delay. Moreover, media service providers’ complaints under the DSA and the P2B Regulation shall be processed and decided upon with priority and without undue delay. Importantly, EMFA’s Article 18 does not apply where VLOPs suspend or restrict the visibility of content in compliance with their obligations to protect minors, to take measures against illegal content or in order to assess and mitigate systemic risks.Footnote ¹²⁷

Next to media freedom, media pluralism and diversity of media content are equally essential for the functioning of a democratic society and are the corollaries of the fundamental right to freedom of expression and information.Footnote ¹²⁸ Media pluralism is recognized as one of the core values of the European Union.Footnote ¹²⁹

In recent years, concerns over the decline of media diversity and pluralism have increased.Footnote ¹³⁰ Online platforms “have acquired increasing control over the flow, availability, findability and accessibility of information and other content online.”Footnote ¹³¹ Considering platforms’ advertising-driven business model based on a profit maximization, they have more incentives to increase the visibility of content that would keep users more engaged. It can be argued that not only does this fail to promote diversity, but it strongly reduces it.Footnote ¹³² The reduction of plurality and diversity of news content resulting from platforms’ content curation policies may limit users’ access to information. It also negatively affects society as a whole, since the availability and accessibility of diverse information is a prerequisite for citizens to form and express their opinions and participate in the democratic discourse in an informed way.Footnote ¹³³

14.3.6 Threats to Media Independence

The growing dependence on automation in news production and distribution has a profound impact on editorial independence as well as on the organizational and business choices of media organizations. One way in which automation could potentially challenge editorial independence is media reliance on non-media actors such as engineers, data providers, and technology companies that develop or fund the development of the datasets or algorithms used to automate editorial decision-making.Footnote ¹³⁴

(News) media organizations depend more and more on platforms to distribute their content. The phenomena of platformed publishing refers to the situation where news organizations have no or little control over the distribution mechanisms decided by the platforms.Footnote ¹³⁵ Moreover, media organizations optimize news content to make it algorithm ready, for example, by producing popular content which is attractive for the platforms’ recommender systems.Footnote ¹³⁶ The entire news cycle, from production, distribution, to consumption of news “is (re)organized around platforms, their rules and logic and thus influenced and mediated by them.”Footnote ¹³⁷ Individuals and newsrooms, therefore, depend structurally on platforms, which affects the functioning and power allocation within the media ecosystem.Footnote ¹³⁸

Moreover, platforms provide essential technical infrastructure (e.g., cloud computing and storage), access to AI models, or stand-alone software.Footnote ¹³⁹ This increases the potential for so-called infrastructure captureFootnote ¹⁴⁰ and risks shifting even more control to platform companies, at the expense of the media organizations autonomy and independence.

The relationship between AI, media, and platforms, raises broader questions about the underlying political, economic, and technological power structures and platforms’ opinion power.Footnote ¹⁴¹ To answer these challenges, legal scholars have called for rethinking media concentration rulesFootnote ¹⁴² and media law in general.Footnote ¹⁴³ However, the considerations about opinion power of platforms, values, and media independence are somehow missing from the current EU regulatory initiatives. The EMFA rightly points out that providers of video-sharing platforms and VLOPs “play a key role in the organisation of content, including by automated means or by means of algorithms,” and some “have started to exercise editorial control over a section or sections of their services” (recital 11). While it does mention “the formation of public opinion” as relevant parameter in the assessment of media market concentrations (art. 21), it does not provide a solution to address the concerns about the dependency between platforms’ AI capacities and media organizations.Footnote ¹⁴⁴

15.3.1 Custodianship and Patient Autonomy

When considering patient autonomy as a concept reflecting individuality,Footnote ²⁰ the question arises how the GDPR allows the data subject to decide autonomously about the reuse of personal data for the development or functioning of AI. While, as explained earlier, data protection is not enacted as an absolute right, patients can decide autonomously about the processing of their data unless the law provides otherwise. In general terms, Article 8 of the European Convention on Human Rights and Article 52 of the Charter of Fundamental Rights of the European Union provide that limitations to the fundamental rights to the respect for private life and the protection of personal data shall be allowed only when necessary in a democratic society and meeting the objectives of general interest or the protection of rights and freedoms of others. A cumulative reading of Articles 6 and 9 of the GDPR can establish a more concrete interpretation of this general principle. Together, Articles 6 and 9 of the GDPR provide the limitative list of situations in which the (secondary) use of personal health-related data is allowed without the patient’s consent.Footnote ²¹ In these situations, the data subject’s wish is considered to not necessarily prevail over the interests of other parties or society. Examples include the collection of health-related data for the treatment of a patient. Depending on specifications in Member State law, the collection can be based on Article 6, 1. (b) “performance of a contract to which the data subject is party” or 6, 1. (c) “legal obligation to which the data controller is subject” on the one hand, and Article 9, 2. (h) “necessary for the provision of health” on the other hand.Footnote ²² A national cancer screening program is another example. In this case, the data collection is typically enacted in Member State law, causing Article 6, 1. (e) “performance of a task in the public interest” to apply in combination with Article 9, 2. (h) “necessary for purposes of preventive medicine.”

Another situation in which the data subject’s individual wishes do not prevail over society’s interest concerns scientific research. By default, data can be reused for scientific research. The data subject’s consent (opt-in) is not required, and when in the public interest, the data subject does not even enjoy a right to opt out.Footnote ²³ First of all, Article 5, 1. (b) of the GDPR provides a specification of the purpose limitation principle indicating that “further processing for […] scientific […] research purposes […] shall, in accordance with article 89 (1), not be considered to be incompatible with the initial purpose.” Additionally, Article 9, 2. (j) provides that contrary to the general prohibition to process health-related data, the processing is allowed when necessary for the purpose of scientific research. The application of Article 6.4 of the GDPR to the secondary use of personal data has raised some discussions, but not in a research context. Read together with Recital 50, Article 6.4. of the GDPR indicates that a new legal basis is not required when the secondary processing can be compatible with the primary processing. A combined reading of Article 6.4. and Article 5, 1. (b) has convinced manyFootnote ²⁴ that a new legal basis is indeed not required when the purpose of the secondary processing is scientific research.Footnote ²⁵

It should, however, be noted that notwithstanding the intention of the GDPR to achieve a higher level of harmonization, one specific provision should not be overlooked when discussing patient autonomy in relation to health-related data. Article 9, 4. of the GDPR foresees that Member States may introduce further restrictions on the processing of health-related, genetic, and biometric data.Footnote ²⁶ Building on this provision, some Member States have introduced the obligation to obtain informed consent from the individual as an additional measure to empower patients.Footnote ²⁷

15.3.2 Informed Consent for Data Processing

When the purpose for which data are shared cannot be covered by a legal basis available in Article 6 and an additional safeguard as laid down in Article 9 of the GDPR, the (valid) informed consent of the patient should be sought prior to the secondary processing. In that case, the requested informed consent should reflect patient autonomy. The conditions for valid informed consent, as laid down in Articles 4 (11) and 7 of the GDPR, indicate that the concept of informed consent was developed as an instrument for individuals to express their wishes and be empowered. These articles stress that consent must be freely given, specific, informed, and reflect an unambiguous indication of the data subject’s wishes. The controller shall be able to demonstrate that the data subject has consented and shall respect the fact that consent can be withdrawn at any time, with no motivation required.

These requirements may sound obvious, but they are challenging to fulfill in practice. In particular, the fact that for informed consent to be freely given a valid alternative for not providing consent for the processing of personal data should be available is often an issue.Footnote ²⁸ Typically, data processing is a consequence of a service, product, or project,… especially in the context of AI. I cannot agree to participate in a data-driven research project to develop AI for medical imaging without allowing my MRI scan to be processed. I cannot use an AI-supported meal app that provides personalized dietary suggestions while not allowing data about my eating habits to be shared. I cannot use an AI-driven screening app for skin cancer without allowing a picture of my skin to be uploaded. In this case, it should be questioned whether data can be reused or shared for secondary purposes based on informed consent.

15.4.1 Anonymization and Pseudonymization

In the GDPR, a preference for the use of anonymized data over pseudonymized and non-pseudonymized data is expressed, for example, in the data minimization principle, as a security measure and in relation to scientific research.Footnote ³⁷ The use of anonymized data is considered to present a sufficiently low risk for the data subject’s fundamental rights to allow the processing without any further measures, and is hence excluded from the GDPR’s requirements.Footnote ³⁸ Pseudonymized data, however, fall under the GDPR because the data can still be attributed to an individual data subject.Footnote ³⁹

In healthcare and other data-intensive sectors, for data not to fall under the definition of personal data, as provided in Article 4(1) of the GDPR, is increasingly difficult due to enhanced data availability and data linkability.Footnote ⁴⁰ Data availability relates to the number of data kept about individuals. Data are not only kept in EHRs but spread over many other datasets held by public and private organizations. Data linkability relates to the ease with which data from different datasets can be combined. Machine learning and other types of AI have a distinct impact in this sense as they facilitate this process.

Requirements on open science,Footnote ⁴¹ explainability,Footnote ⁴² and citizen empowermentFootnote ⁴³ stimulate data holders to increase the level of data availability and linkability. To create innovations this is a great assumption, but there is another side to the coin. A higher level of data availability and linkability requires data holders, such as healthcare organizations, to increasingly qualify data as pseudonymous rather than anonymous.

Influential studies continue to show limitations in anonymization techniques in relation to patient data. Schwarz et al., for example, reidentified patients based on de-identified MRI head scans, which were released for research purposes. Schwarz’s research team showed that in 83% of the cases, face-recognition software matched an MRI with a publicly available picture. In 95% of the cases, the image of the actual patient was amongst the five selected public profiles.Footnote ⁴⁴ Studies such as Schwarz’s led to the development of “defacing techniques,” a privacy-enhancing measure to hinder the reidentification of head scans.Footnote ⁴⁵ However, is the hindrance caused by the defacing technique sufficient for the scan to qualify as nonpersonal data?

To answer that question, it is important to stress that the scope of the GDPR is not delineated based on the presence of certain specific identifiers in a particular dataset. Contrary to, for example, the US Health Insurance Portability and Accountability Act (HIPAA),Footnote ⁴⁶ which provides that individually identifiable information can be de-identified by removing the listed identifiers (exhaustive account) from the dataset, the GDPR requires a more complex assessment. The possibility for the controller or another person to single out a data subject building on the information in the dataset and any additional information that can be obtained using all the means reasonably likely must be evaluated. When considering the MRI image, this means that account must be taken of the MRI image with defacing techniques applied, pictures available on the internet, and the original MRI available in the EHR even when this image is not available to the data controller.Footnote ⁴⁷

15.4.2 Federated Learning, an Example of a Privacy-Enhancing Technical Measure

Federated analysis allows for building knowledge from data kept in different local sources (such as various EHRs in hospitals, public health databases in countries, or potentially even individual health “pods” kept by citizensFootnote ⁴⁸) while avoiding the transfer of individual-level data.Footnote ⁴⁹ Hence, federated analysis is presented as a solution to avoid the centralization of (personal) health-related data for secondary use.

Imagine building an AI model for cancer detection through MRI images: In a nonfederated scenario, the MRI images are requested through multiple participating hospitals, pseudonymized, and subsequently collected in a central, project-specific database. The algorithm is trained on the central database. In a federated scenario, however, the MRI images are not pooled in a central database. Instead, they remain with the local hospital. The algorithmic model, carrying out analytical tasks, visits the local databases (“nodes”) and executes tasks on the locally stored MRI images.Footnote ⁵⁰ Subsequently, aggregated results (the conclusions) are shared with a central node for merging and meta-analysis. On the condition of a small cell risk analysis,Footnote ⁵¹ these results can often be considered nonpersonal data because individual patients can no longer be singled out.

Avoiding centralization is particularly interesting because it can reduce the risk of illicit data usage. The control on the secondary use remains with the data holder: A data custodian (such as a hospital), the individual (such as a patient), or perhaps, as suggested in Article 17 et seq of the Data Governance Act, a recognized data altruism organization.Footnote ⁵² Unlike organizational measures, such as contractual arrangements on the purpose of the processing, federated learning thus allows the data holder to manage the processing independently.

The implementation of federated learning should, however, not trigger the assumption that the processing operations are not covered by the material scope of the GDPR. Federated learning does not avoid the processing of personal data for a secondary purpose. It merely avoids the transfer of personal data. In other words: the processing takes place locally, but data are reused for a purpose different from the purpose for which they were initially collected. Consequently, GDPR requirements must be complied with, including the need for a legal basis.

Following Article 4 (7) of the GDPR, the party defining the purpose and (at least the essential) means of the secondary use should be considered the data controller. Generally, the requestor, not the requestee, defines the purpose and means of the secondary processing. Therefore the requestor is considered the data controller.Footnote ⁵³ The location of the data processing (locally or centrally) is irrelevant. Who has access to the data is equally irrelevant.Footnote ⁵⁴ Consequently, although a data transfer agreement may be avoided when sharing merely anonymous data with the central node, a data processing agreement (or joined controller agreement) must be in place before reusing the data.Footnote ⁵⁵

15.4.3 Transparency, an Example of a Privacy-Enhancing Organizational Measure

The importance of transparency cannot be overestimated. As indicated by the EDPB in the adopted Article 29 Working Party Guidelines on transparency under the GDPR: “transparency is a long established feature […] engendering trust in the processes which affect the citizen by enabling them to understand, and if necessary, challenge those processed.”Footnote ⁵⁶ The transparency principle entails an overarching obligation to ensure fairness and accountability. Therefore, data controllers must provide clear information that allows data subjects to have correct expectations.

The transparency obligation is a general obligation isolated from any information obligations that may follow from informed consent as a legal basis. Whichever legal basis is most suitable and whether it concerns primary or secondary use, the data controller is responsible for providing transparent information actively (following Articles 13 and 14 GDPR) and passively (following a data subject access request under Article 15 GDPR). This includes the obligation to inform about (intentions to) reuse.Footnote ⁵⁷

Today data controllers often focus on the availability of general information on websites, in brochures, and in privacy notices, to comply with their transparency obligation. Unfortunately, these general information channels often prove insufficient to enable data subjects to really understand for which purposes and by whom data about them is used. They feel insufficiently empowered to hold the data controller accountable or to exercise control over their personal data. If other patients’ rights such as the right not to know, can be respected, wouldn’t it make sense to create personalized overviews of secondary data processing operations in an era where personalization is a buzzword? These overviews could be provided through consumer interfaces such as client accounts, personalized profiles, or billing platforms. In healthcare, it is no longer uncommon for healthcare providers to provide patients with a direct view of their medical records through an app or website. A patient-tailored overview of secondary use could be included in this patient viewer.

As a side note, it must be mentioned that the EDPB announced further clarifications on the scope of the exceptions to the obligation to actively inform data subjects individually.Footnote ⁵⁸ Article 14, 5. (b) of the GDPR acknowledges that when data were not obtained directly from the data subject, it may occur that “the provision of information proves impossible or would involve a disproportionate effort.”Footnote ⁵⁹ In earlier interpretations, the limitations of this exception were stressed explaining It that the data controller must demonstrate either impossibility or a disproportionate effort. In demonstrating why Article 14, 5. (b) should apply, data controllers must mention the factors that prevent them from providing the information and illustrate the impact and effects for the data subject when not provided with the information in the case of disproportionate effort.Footnote ⁶⁰

16.2.1 Traditional Methods to Predict Credit Default Risk

When going about an evaluation, the lender faces uncertainty about an applicant’s credit default risk. In the parlance of economics, he must rely on observable variables to reconstruct hidden fundamental information.Footnote ¹⁸ Sociologists add the role of trust in social relations to explain the denial or success of a loan application.Footnote ¹⁹ The potential borrower will provide some information himself, for instance on existing obligations, income, or assets. To reduce uncertainty, the lender will often require additional input. Historically, a variable as qualitative and vague as “character” was “considered the foundation of consumer creditworthiness.”Footnote ²⁰ Starting in the 1930s, lenders profited from advances in statistics which allowed to correlate attributes of individual loan applicants with high or low credit default risk.Footnote ²¹ Depending on the country, the relevant characteristics “can include a wide variety of socioeconomic, demographic, and other attributes or only those related to credit histories.”Footnote ²² Being a white, middle-aged man with a stable job and living in wholly owned property usually predicted a lower credit-default risk than being a young man living in a shared flat, an unmarried woman, or a person of color without a stable job.

The exercise requires two main ingredients: mathematical tools and access to data. The former serves to form statistical buckets of similarly situated persons and to correlate individual attributes of loan applicants with attributes that were in the past found to be relevant predictors of credit default risk. The latter was initially provided by lenders compiling their own data, for instance on credit history with the bank, loan amounts, or in- and outgoing payments. Later, credit registries spawned that are, until today, a key source of data.Footnote ²³ The type of data a credit registry collects is highly standardized, but varies across countries. Examples are utilities data, late payments, number of credit cards used, collection procedures, or insolvency proceedings.Footnote ²⁴ Using the available input data, the probability of credit default risk is often expressed in a standardized credit score.Footnote ²⁵

16.2.2 AI-Based Methods to Predict Credit Default Risk

Over the last decade, both ingredients,Footnote ²⁶ mathematical tools and access to data, have changed radically. Digitization across many areas of life leaves digital footprints of consumers. Collecting and processing these “big data” allows us to refine statistical output that can now work with large amounts of information, far beyond what traditional credit registries hold. Artificial intelligence in the form of machine learning helps to correlate variables, interprets results, sometimes learns from these, and, in that way, finds ever more complex patterns across input data.Footnote ²⁷

“All data is credit data”Footnote ²⁸ is an often-quoted remark, hinting at the potential to use unanticipated attributes and characteristics toward a prediction of credit default risk. This starts with an applicant’s online payment history or performance on a lending platform but does not stop there.Footnote ²⁹ Age or sex, job or college education, ZIP code, income, or ethnic background can all be relevant predictors. Depending on a jurisdiction’s privacy laws, more variables can be scrutinized. Such “alternative data” include, for instance, preferred shopping places, social media friends, political party affiliation, number of typos in text messages, brand of smartphone, speed in clicking through a captcha exercise, daily work-out time, or performance in a psychometric assessment. All these are potential sources for correlations, the AI system will detect between individual data points, and the goal it is optimizing. An example of an optimization goal (which is also called “definition of success”Footnote ³⁰) is credit default. An AI system working with this definition of success is useful for a lender who wishes to price his loan according to the probability of default. The system will find various correlations between the big data input variables and the optimization goal. Some correlations might be unsurprising, such as a high steady income and low default. Others will be more unexpected. The German company Kreditech illustrates this. It learned that an important variable its AI system found was a specific type of font on an applicant’s electronic devices.Footnote ³¹ A research study provides another illustration by concluding that:

16.2.3 Inclusion Through AI-Based Methods

A score that an AI model develops based on alternative data can provide access to finance for persons who have found this difficult in the past, due to their unusual profile. A recent immigrant will often not be able to provide a utility payment or a credit history in his new country of residence. However, this “thin-file applicant” might have relevant alternative data of the type described earlier to support his creditworthiness assessment.

Several empirical studies have found that AI-based credit scoring broadens financial access for some thin-file applicants.Footnote ³³ One important source of data are mobile phones. “The type of apps installed or information from call log patterns,” so the authors of one study find, “outperforms a traditional model that relies on credit score.”Footnote ³⁴ Another study, working with an e-commerce company, produced good predictions using only ten digital footprint variables.Footnote ³⁵ The authors find, for example, that the difference in default rates between customers using an Apple and customers using an Android device is equivalent to the difference in default rates between a median credit score and the 80th percentile of the credit score.Footnote ³⁶ Yet another illustration is provided by the US online lending company Upstart.Footnote ³⁷ Upstart claims to outperform traditional scoring as to all borrowers, specifically as to those with traditionally low credit scores. A study on this company shows that:

A US regulatory agency, the Consumer Financial Protection Bureau, investigated Upstart’s business model and confirmed that, in the aggregate, applicants with low credit scores were approved twice as frequently by Upstart if compared with a hypothetical lender.Footnote ³⁹

16.3.1 Algorithmic Discrimination

The US lender Upstart illustrates not only how AI can further inclusion. Additionally, it provides an example of AI models producing unequal output across groups of loan applicants. Among those who profit, persons facing historical or current discrimination are underrepresented. A well-documented example concerns a report from a US NGO which ran a mystery shopping exercise with Upstart.Footnote ⁴⁰ It involved three loan applicants that were identical to college degree, job, and yearly income, but had attended different colleges: New York University, Howard University, which is a historically Black college, and New Mexico State University, a Hispanic-serving institution. Holding all other inputs constant, the authors of the study found that a hypothetical applicant who attended Howard or New Mexico State University would pay significantly higher origination fees and interest rates over the life of their loans than an applicant who attended NYU.

One explanation for these findings points to the AI system predicting credit default in a world of (past and current) discrimination where Black and Hispanic applicants face thresholds that otherwise similarly situated borrowers do not. Along those lines, the unequal output reflects real differences in credit default risk.

Alternatively (or additionally), the AI system’s result might be skewed by a variety of algorithmic biases.Footnote ⁴¹ In this case, its credit score or creditworthiness assessment paints a picture that does not correctly reflect actual differences. Historical bias provides one example. A machine-learning system is trained based on past data, involving borrowers, their characteristics, behavior, and payment history.Footnote ⁴² Based on such data, the AI system learns which individual attributes (or bundles of attributes) are good predictors of credit default. If certain groups, for example unmarried women, have in the past faced cultural or legal obstacles in certain countries, the AI system learns that being an unmarried woman is a negative signal. It will weigh this input variable accordingly. In this way, an attribute that would have lowered a credit score in the past will lower it today, even if the underlying discriminatory situation has been overcome (as is partly the case for unmarried women). Majority bias is another example.Footnote ⁴³ The AI system builds its model by attributing weight to input variables. What it finds for most candidates that were successful in the past will be accorded considerable weight, for instance, stability as to place of residence. Candidates who score badly in that respect because their job requires them to move often will face a risk premium. It is important to understand that the machine-learning system finds correlations only and does not attempt to establish causation. An individual might have very good reasons for moving houses often, he might even hold a high-paying job that requires him to do so. The AI system will still count this as a negative attribute if the majority of past candidates did not move often.

Another empirical study suggests that there can be yet further factors at play. Researchers explored data on the US Government Sponsored Enterprises Fannie Mae and Freddie Mac.Footnote ⁴⁴ These offer mortgage-backed lenders a guarantee against credit risk, charging a fee that depends only on the borrower’s credit score and the loan-to-value ratio. Against that background, one would assume that for candidates with identical scores and LTV ratios, interest rates must be identical. This was not what the study found. Hispanic and Black borrowers faced markups of 7.9 basis points for purchase mortgages and 3.6 basis points for refinance mortgages. The “morally neutral” AI system might have understood this as a promising strategy to meet the goal of maximizing profit for the lender. Instead, the lender might have himself formulated the AI’s definition of success as identifying applicants who were open to predatory pricing, for instance because they urgently needed a loan or were financially less literate than other borrowers.Footnote ⁴⁵

16.3.2 Opaque Surveillance

Let us revisit the lender who faced uncertainty as to potential borrowers. Today, he uses credit scores and creditworthiness assessments that rely on a limited list of input variables. Usually, their relevance for credit default risk is obvious, and advanced statistics allows for good predictions. For the applicant, this entails the upside that he will typically know which attributes are important to be considered a good credit risk.Footnote ⁴⁶ Against that background, one would expect that an unobservable and hardly quantifiable variable such as “character”Footnote ⁴⁷ loses significance. With AI-based scoring, this might change, if the seemingly objective machine lends new credibility to such concepts. The researchers who used mobile phone data for credit scoring interpreted their findings as a strategy to access “aspects of individuals behavior” that “has implications for the likelihood of default.”Footnote ⁴⁸ The authors of the e-commerce studyFootnote ⁴⁹ explicitly suggest that the variables they investigated provide a “proxy for income, character and reputation.”Footnote ⁵⁰

A borrower whose credit application is assessed by an algorithm might feel compelled to give access to his personal data unless he is prepared to accept a lower credit rating. At the same time, he does not necessarily know which elements of the data he hands over will be relevant. Credit applicants today often know what is important for obtaining credit and have legal rights to be informed about a denial.Footnote ⁵¹ Under an AI black box model, not even the lender is necessarily aware of what drives the credit score his AI system produces.Footnote ⁵² If he does, his incentives to inform the applicant are often small. This is especially likely if the lender feels he found a variable that is a powerful predictor but at the same can be manipulated by the applicant. Consider, for instance, a finance or a dating app on his phone, one helping, the other hurting his credit score, while both apps are easily installed/uninstalled.

AI scoring of this type places consumers in a difficult spot. They are likely to worry about a “world of conformity”Footnote ⁵³ where they “fear to express their individual personality online” and “constantly consider their digital footprints.”Footnote ⁵⁴ They might feel that they are exposed to arbitrary decisions that they do not understand and that are sometimes unexplainable even to the person using the algorithm.Footnote ⁵⁵ Economists have predicted that consumers might try to randomly change their online behavior in the hope for a better score.Footnote ⁵⁶ Manipulation along those lines will work better for some variables (such as regularly charging a mobile device) than for others (such as changing mobile phone brand or refraining from impulse shopping).Footnote ⁵⁷ One strategy is to mimic the profile of an attractive borrower. This suggests side effects to the overall usefulness of AI scoring. If it is costless to mimic an attractive borrower, an uninformative pooling equilibrium evolves: All senders choose the same signals.Footnote ⁵⁸ Firm behavior might adapt as well.Footnote ⁵⁹ A firm whose products signal low creditworthiness could try to conceal its products’ digital footprint. Commercial services may develop, offering such services or making consumers’ digital footprint look better. Along similar lines, the US CFPB fears that the chances to change credit standing through behavior may become a random exercise.Footnote ⁶⁰ While the applicant today receives meaningful information about (many of) the variables which are relevant for his score, with opaque AI modelling, this is not guaranteed any more.

16.4.1 Anti-Discrimination Laws Faced with AI

Is unequal output across groups of applicants, for instance as to sex or race, necessarily a cause for concern? Arguably, the answer depends on the context and the goal pursued by the lender.

Under the assumption that the AI model presents an unbiased picture of reality, an economist would find nothing wrong with unequal output if it tracked features that are relevant to the lender’s business strategy. Any creditor must distinguish between applicants for a loan,Footnote ⁶¹ and score and rank them, usually according to credit default risk. This produces statistical discrimination that reflects the state of the world. Prohibiting unequal output entirely would force the lender to underwrite credit default risk he does not wish to take on.Footnote ⁶²

By contrast, if the assumption of the AI system reflecting an unbiased picture of the world does not hold, the unequal output can be a signal for potential inefficiencies. An AI model’s flawed output, for example due to historical or majority bias,Footnote ⁶³ leads to opportunity cost if it produces too many false negatives: Candidates that would be a good credit risk but are flagged as a bad credit risk. In this case, the lender should have granted a loan, but he refused, based on the flawed AI system. If, by contrast, the model triggers too many false positives, it can skew the lender’s portfolio risk. The lender has underwritten more contracts than his business model would have suggested to.

A lawyer has a more complicated answer to the question mentioned earlier.Footnote ⁶⁴ Depending on the situation, unequal output can violate anti-discrimination laws. A clear case is presented by a lender who denies a loan because of a specific attribute – sex, race, religion, ethnicity, or similar protected characteristics. If this is the case (and the plaintiff can prove it), the lender is liable for damages.Footnote ⁶⁵ The test prong because of is met, if the protected characteristic was one reason toward the decision.

Direct discriminationFootnote ⁶⁶ in this form is not a risk that is unique to AI-based decision-making. Quite to the contrary, many point out that a well-trained, objective AI system will overcome human biases and discriminatory intentions.Footnote ⁶⁷ However, if a lender pursues discriminatory motives or intentionally seeks members of protected communities because they are more vulnerable to predatory pricing, AI systems compound the risks plaintiffs face. This has to do with the AI system’s potential to help the lender “mask” his true preferences.Footnote ⁶⁸ Masking behavior can be successful because anti-discrimination law needs a hook, as it were, in the lender’s decision-making process. The plaintiff must establish that he was discriminated against because of a protected characteristic, such as race or sex. A discriminatory lender can try to circumvent this legal rule by training its AI system to find variables that correlate narrowly with a protected characteristic. Eventually, circumventing the law will not be a valid strategy for the lender. However, the applicant might find it very hard to prove that this was the lender’s motive, especially in jurisdictions that do not offer pre-trial discovery.Footnote ⁶⁹

The disproportionate output of an AI system does not always go back to business strategies that imply direct discrimination.Footnote ⁷⁰ One of the most characteristic features of algorithm-based credit risk assessments is to find unanticipated correlations between big data variables and the optimization goal.Footnote ⁷¹ What happens if it turns out that a neutral attribute, for instance the installation of a finance app on a smartphone, triggers disproportionate results across sex? Unless a lender intentionally circumvents the rule to not discriminate against women, this is not a case of direct discrimination.Footnote ⁷² Instead, we potentially face indirect discrimination.Footnote ⁷³ Under this doctrine, a facially neutral attribute that consistently leads to less favorable output for protected communities becomes “suspicious,” as it were. Plaintiffs must establish a correlation between the suspicious attribute and the unequal output. Defendants will be asked to provide justificatory reasons for using the attribute, in spite of the troubling correlation.Footnote ⁷⁴ In a credit underwriting context, the business rationale of the lender is a paradigm justificatory reason. The plaintiff might counter that there are equally powerful predictors with less or no discriminatory potential. However, the plaintiff will often fail to even establish the very first test prong, namely to identify the suspicious attribute. The more sophisticated the AI system and the larger the big data pool, the more likely it is that the AI system will deliver the same result without access to the suspicious variable. The reason for this is redundant encoding: The information encoded in the suspicious variable can be found in many other variables.Footnote ⁷⁵

16.4.2 AI Biases and Quality Control in the EU Artificial Intelligence Act

The EU AI ActFootnote ⁷⁶ starts from the assumption that AI credit underwriting can lead to discriminatory output:

However, the remedy the AI Act proposes is not to adjust anti-discrimination law. Instead, it proposes a strategy of product regulation. Artificial intelligence systems employed in the loan context “should be classified as high-risk AI systems, since they determine those persons’ access to financial resources or essential services such as housing, electricity, and telecommunication services.”Footnote ⁷⁸

Employing a high-risk AI system entails mandatory compliance checks as to monitoring, testing, and documentation with an eye on both software and data.Footnote ⁷⁹ Article 9 of the Act has the software part in mind. It requires identifying risks to fundamental rights and developing appropriate risk management measures. Ideally, the design or development of the high-risk AI system eliminates or reduces these risks. If they cannot be eliminated, they must be mitigated and controlled.Footnote ⁸⁰ Article 10 addresses training data which might lead to the biases mentioned earlier.Footnote ⁸¹ According to Article 14 of the proposal, high-risk AI systems must be “designed and developed in such a way (…) that they can be effectively overseen by natural persons.”Footnote ⁸² Supervisory agencies are in charge of enforcing compliance with the AI Act. For credit institutions, the competent banking regulator is entrusted with this task. Nonbank entities, for instance credit scoring agencies, will be supervised by a different body in charge of AI.Footnote ⁸³ Both supervisors must deal with the challenge of quantifying a fundamental rights violation (which includes balancing borrower’s rights against competing rights) to produce a workable benchmark for risk management.

16.4.3 Privacy and Retail-Borrower Protection Laws Faced with AI

Private enforcement via litigation is not included in the AI Act.Footnote ⁸⁴ Against this background, the following section looks at privacy and retail-borrower protection laws that provide such tools. Privacy law aims at keeping personal data private and subjecting its use by third parties to certain requirements. Retail-borrower protection laws cover many aspects of a transaction between borrower and lender. These include rights that are especially useful in the context of algorithmic credit evaluations, for example a right to be informed about a denial of credit.

17.3.1 The Specific Case of Algorithmic Management: Automated Decision-Making and Monitoring Systems

One of the most prominent uses of AI systems in the context of employment concerns the management of workers. It can be broadly understood as the use of data-driven tools that collect and analyze an extensive amount of data on workers in order to allocate tasks to them, to evaluate their performance, or to discipline them.Footnote ¹⁴ Taking a more granular view, algorithmic management can be defined as

There is a genuine risk that the deployment of such tools occurs to the detriment of workers’ social protection, allowing discrimination and the worsening of their working conditions.

Algorithmic management is a concept that has first been used in the mid-2000s, mainly in the context of delivery platforms in the USA. It started as a relatively “unseen” practice, but its wider adoption is happening incrementally and has now spread to many sectors of the economy.Footnote ¹⁶ Even though most algorithmic management tools are commercialized outside the European Union (EU), they are often also implemented in companies and organizations established within the EU or elsewhere in the world.

Algorithmic management can take different forms, with many examples found in recruitment processes to screen or analyze the CVs of job candidates, or to “assess” facial expressions during video interviews. It is however also used to assess the job performance of workers, by tracking and analyzing their physical work.Footnote ¹⁷ For instance, wearable devices are now used in warehouses, call centers, and other workplaces to produce metrics on productivity and create rankings of workers’ performance.Footnote ¹⁸ Other wearables used for safety purposes integrate sensors to measure physical distancing, to detect a potential harmful environment on a specific site, or to measure physiological parameters of individual workers and observe their health conditions and well-being.Footnote ¹⁹ Algorithmic systems also exist that direct workers by specifying the tasks they have to carry out, including the order and timeframe in which this needs to be done.Footnote ²⁰ Other tools can track and evaluate teams at individual levels, collecting metrics to measure their level of interactionFootnote ²¹ or to nudge workers’ behavior. This algorithmic management practice already permeates sectors such as transport and logistics, with the objective of optimizing delivery services, but other sectors are following suit.

Finally, algorithmic management is at the heart of digital labor platforms, where it is used to process customer feedback or to analyze and evaluate workers. This is especially the case for transport-oriented platforms, which use algorithmic systems to manage drivers.Footnote ²² These workers receive their assignments based on, among others, their location and profile, and they are then provided with directions to pick up customers or deliver orders.

17.3.2 Challenges of Algorithmic Management

These examples demonstrate that AI systems are already present in workplaces and raise numerous questions for labor law.Footnote ²³ Adopting an interdisciplinary approach, this section offers a non-exhaustive mapping of the most relevant issues that require attention, touching upon elements that relate not only to labor law provisions, but also privacy and data protection rules, fundamental rights, and computer science practices.

• Large processing of data. The first issue to point out, is that algorithmic management relies on the complex processing of large amounts of personal data, which allows for the “quantification of workers.”Footnote ²⁴ Two important questions that need to be addressed in this context are: Why are personal (and potentially also sensitive) data collected? And for what purposes are such data used? After all, someone must explicitly take the decision of collecting and using data, and such decisions needs to be accounted for. As noted earlier, when algorithmic management systems are used to assess workers’ performance, this can enable the analysis or monitoring of processes that would go beyond the supervision of a human manager. In other words, algorithmic management not only leads to monitoring workers to extents unthinkable in the past, but also to the collection and processing of data to analyze and make decisions about workers’ jobs and their working conditions in a far more intrusive manner. Although algorithms themselves are used to implement certain rules,Footnote ²⁵ the actual decision-making is not yet fully in the hands these systems. Indeed, behind AI systems, there are human beings who set rules and parameters for data collection and for decision-making. Accordingly, before such systems are implemented in workplaces, important concerns about responsibility and accountability for these systems and their impact on workers need to be addressed.

• Transparency of AI systems. A second issue relates to the transparency of the systems’ architecture, operations, and opacity around the purpose of the systems. To determine whether algorithmic management systems process personal data lawfully, it is important that their operations can be verified. This is also important to assess whether they are affected by certain biases – which can lead to the discrimination of vulnerable groups of workers – or whether they showcase other failures due to poor data quality, inaccuracy or errors. Some cases have already evidenced that bias in learning systems resulted in discriminatory outcomes. For example, Amazon introduced a recruiting Machine Learning based algorithm to sort out and select talented applicants on the basis of selected features of their CV. However, the algorithm considered male candidates as more talented, because it had been trained on the basis of data that reflected the male dominance across the tech industry.Footnote ²⁶ Therefore, besides risks of privacy violations, the lack of transparency about how data is collected and processed may, among others, put workers in a position where they will not be treated equally and, hence, be deprived of job opportunities on the basis of discriminatory criteria.

• Privacy and data protection rights. A third issue relates to the effective protection of workers’ right to privacy and data protection when working alongside AI systems. Article 22 of the General Data Protection Regulation (GDPR)Footnote ²⁷ gives data subjects “the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.” However, the scope of application of this article seems to be interpreted rather restrictively.

Three legal proceedings were brought before the District Court of Amsterdam by the App Drivers and Couriers Union, a platform workers union, claiming violations of the EU GDPR: a first case, Uber drivers v. Uber, on transparency requests and access to personal dataFootnote ²⁸; a second case, Uber drivers v. Uber, on deactivation of drivers accounts and termination of their contractFootnote ²⁹; and a third case, Ola drivers v. Ola Cabs, on transparency requests by drivers and access to their data.Footnote ³⁰

The App Drivers and Couriers Union relied inter alia on Articles 15 of the GDPR to gain access to all their personal data processed by the platforms, including the use of drivers’ monitoring systems such as Uber’s Real Time ID and Ola’s Guardian system. On the basis of Article 22 of the GDPR, the drivers also asked to receive information on the platforms’ algorithmic systems that make decisions about them, including the deactivation of their accounts, known as “robo-firing.”

However, in judgments pronounced on March 11, 2021, the Amsterdam District Court rejected most of the drivers’ claims, as it found that Article 22 did not apply to these algorithmic systems. The platforms had demonstrated that staff members intervened during the decision-making process and that the decisions did not significantly affect the drivers, thus rendering the protection of Article 22 inapplicable.

Moreover, the App Drivers and Couriers Union appealed to receive access to their personal data and to receive explanation about how automated decisions were made. The Court of Appeal in Amsterdam on April 4, 2023Footnote ³¹ decided positively in their favour. The court ruled that Uber had to provide access to the personal data related to profile, tags, reports per journey, individual ratings, upfront pricing-system, information regarding recipients of personal data, a category from the Guidance note, and information about automated individual decision-making. It also ruled that profiling and management assessments are personal data and they must be disclosed. Regarding the (automated) decisions, the court found that they were taken fully automatically, and that there was insufficient evidence of human intervention which affected drivers significantly, partly because they affect their income without the access to the App. The court rejected Uber’s argument of drivers taking collective action to seek access to their data, amounted to an abuse of data protection rights. It confirmed the right of third parties, including trade unions, to establish a gig workers data trust.Footnote ³² On the third judgment related to Ola Cars, the court ruled that data access requests fell within the scope of Article 15 of the GDPR. It ordered Ola Cars to disclose information to workers on automated decision-making relating to work allocation and fares.

Despite the more successful appeal, these cases demonstrate that the provisions and principles of the GDPR may be difficult to exercise vis-à-vis employers, in concrete to access request to data, transparency of the operation and logic of algorithmic management systems or other AI tools. In the context of employment, it is important to note that the employer, as a controller, must fulfill their obligations. These obligations include ensuring the protection of the data subject and being accountable for all personal data processed related to workers. It sheds light into how is crucial to provide additional information regarding the intended or future processing, on how automated decision-making may impact workers and on the limits of profiling. It is hence fair to question whether the workers’ fundamental right to privacy is sufficiently safeguarded when their behavior and performance in the workplace is reduced or transformed to mere numbers and digits on the parameters or criteria applied in AI tools or algorithmic systems. The European Court of Human Rights (ECtHR) has given a broad interpretation to Article 8 of the European Convention on Human Rights, whereby the right to respect for private and family life also includes the right “to develop one’s physical and social identity”Footnote ³³ and “to establish and develop relationships with other human beings.”Footnote ³⁴ Therefore, it would be paramount to provide further clarification on how these rights can actually be protected when AI applications are used at work.

Technostress. A fourth issue is the use of algorithmic management to influence (nudge) and control workers’ behavior,Footnote ³⁵ which may have consequences to occupational safety, health or well-being. The primary promise of AI in this domain is its ability to ability to enhance the accurate prediction of potential accidents. AI applications are increasingly integrated into equipment, industrial machines, drones, robots or self-driving vehicles. They can also be embedded in systems associated with personal protective equipment, such as bracelets, wearables, exoskeletons, sensors, and other hardware.

However, it is important to recognize that these systems rely on personal and sensitive data and on an increasingly digitized working environment, thereby posing risk significantly impacting workers’ occupational health, safety and well-being. Additionally, complications may arise from other factors, such as when operators or managers have incomplete understanding of the data and its analysis. In broad terms, the key concerns arising within this context can be categorized into two dimensions: physical and psychosocial.

When it comes to workers’ physical safety, there can occur various types of risks, for example, a risk that the AI-driven machine, robot or partly automated vehicle can wrongly process or analyze the data it collects, thereby leading to an erroneous output or acting unexpectedly, resulting in an injury or accident. Similar risks exist when there are errors or inaccuracies in the dataset, when the system’s parameters are not properly tuned or optimized, or when the system’s accuracy – and hence the accuracy of its predictions – is faulty.Footnote ³⁶ Risk assessment is crucial to identify the possible sources, mitigate or eliminate possible harms.

In relation to the psychosocial and well-being aspects, workers may experience high stress levels when they are aware that their behavior, location, performance, and even emotions, are being monitored and analyzed. In this context, the issue of technostress becomes relevant. The term was coined in 1984 and scholars describe this phenomenon as “any negative impact on attitudes, thoughts, behaviors, or body physiology that is caused either directly or indirectly by technology.”Footnote ³⁷ It combines five common technostressors: techno-overload, techno-invasion, techno-complexity, techno-insecurity, and techno-uncertainty. Technostress has become increasingly significant in the workplace, particularly when workers have to rely on information and communication technologies (ICTs) and AI-driven applications to carry out their tasks.Footnote ³⁸

Technostress may impact workers’ psychophysical health and work-life, the consequences can be seen on the short- and long-term on somatic, cognitive–emotional, and behavioural levels,Footnote ³⁹ for example, causing psychophysical distress and depleting both emotional and cognitive resources in the individual directly or indirectly.Footnote ⁴⁰ These consequences can extend beyond the individual level, affecting the organizational and societal environments, because of the “always-on” culture and nature of work-related to ICT arrangements, that “create an unbridgeable gap between how an individual is expected to behave during family time and job requests as mediated through ICTs.”Footnote ⁴¹

Finally, algorithmic management can also be used to reward and discipline workers, to elicit cooperation or to enforce compliance.Footnote ⁴² The automated deactivation of riders’ accounts, whether temporarily or permanently, has for instance become a common practice. Some digital labor platforms reward high rated drivers by enabling them access to rides that are financially more attractive, and by giving them priority in queues at popular places (such as airports).Footnote ⁴³ In this way, digital platforms “enforce” compliance by providing workers access to higher remuneration when they show certain behavior. There are also examples regarding algorithmic disciplining processes that lead to the discipline of workers. For instance, there is some evidence that in the hospitality sector algorithmic evaluations based on online reviews may result in dismissing staff members, when they do not meet the expected targets.Footnote ⁴⁴ These uses contribute to precarious working conditions.

Considering the earlier mentioned concerns, one can raise the question how they should best be managed and how the risks should be prevented and mitigated? As companies have a legal responsibility to improve workers health by preventing excessive work-related technostress, it is recommended to apply specific prevention policies and other appropriate measures to remove risk factors and mitigate the potential negative effects associated with this practice.Footnote ⁴⁵ Policies and strategies can be centered on the user, the technological environment, organizational environment, and social environment.Footnote ⁴⁶

More concretely, this involves establishing an ICT environment that meets job-specific requirements to reduce the frequency, duration, and/or intensity of technostressors; tailoring the use of ICTs to the needs of workers; providing transparency over how work-related data collected by technology is processed and used; providing transparency regarding which technologies are used for which purpose; and conducting impact assessments on the possible risks and consequences of AI systems prior to their implementation.Footnote ⁴⁷ Further, a robust legislative framework is essential to address the identified issues of concern and to respond in a preventive and adequate manner. This may include the prohibition of some of these practices to the extent they are not compatible with workers’ fundamental rights. Overall, given the goals of labor law, it may well be required that a new balance between workers’ and employers’ interests must be sought in light of the increasing application of AI in workplaces. Yet before turning to new legislative initiatives on AI that are relevant for the sphere of work, we discuss some of the key rights and concepts of labor law.

17.4.1 Information, Consultation, and Participation

Both international and European law recognize the rights to information, consultation, and participation as three essential rights of labor law. At the international level, reference can be made to Convention 158 of the ILO, which states that workers’ representatives must be consulted “on measures to be taken to avert or to minimize the terminations and measures to mitigate the adverse effects of any terminations on the workers concerned such as finding alternative employment.”Footnote ⁴⁸

ILO Recommendation 166 also refers to consultations on major changes in the undertaking:

Whereas the provisions of the earlier mentioned Convention must be implemented in the national legislation upon (voluntary) ratification by ILO Member States, the provisions of the Recommendation are not obligatory but rather contain reference standards for Member States on which they are encouraged to base their labor policies and legislation.Footnote ⁵⁰

In the EU, these labor law rights are enshrined in the Charter of Fundamental Rights of the European Union (CFR). Article 27 of the CFR states that “workers or their representatives must, at the appropriate levels, be guaranteed information and consultation in good time in the cases and under the conditions provided for by Community law and national laws and practices.”

Moreover, there are more than fifteen EU directives dealing with the right to information and consultation. The most important directives in this regard are: (1) the Directive on European Works CouncilsFootnote ⁵¹; (2) the Directive on Employee Involvement in the European Company,Footnote ⁵² (3) the European Framework Directive on Information and Consultation, which sets the minimum requirement for workers’ right to information, consultation, and participationFootnote ⁵³ and the (4) Directive relating to collective redundanciesFootnote ⁵⁴.

Whenever technological changes are introduced in European multinational companies, the Directive on European Works Councils applies. Point 1(a) of Annex I of the Directive states that “the information and consultation of the European Works Council shall relate in particular to […] substantial changes concerning organisation, introduction of new working methods or production processes to negotiate about the impact of the introduction of new processes […].” European Works Councils (EWC) are the information and consultation bodies that bring together both management and workers representatives from the European countries in which a given multinational company has operations. In the EWC, the representatives of central management inform and consult the workers’ delegation, and they can negotiate a variety of topics and company decisions that have an impact at a transnational level.Footnote ⁵⁵ Collectively, this body of legislation aims at providing workers and employers with strong social protection, in order to improve living and working conditions and to protect social cohesion.Footnote ⁵⁶ It should also be added that various of these rights are protected through national legislation too.

17.4.2 Social Dialogue

Another essential component of the labor law acquis concerns Social Dialogue. This refers to the specific role of social partners, meaning the recognized organizations representing the two sides of the industry, the employers and employees. The social partners usually comprise employers’ organizations and trade unions respectively.

Social Dialogue is a unique instrument of governance and cooperation. At international level, the ILO is the only tripartite United Nations agency that brings together governments, employers and workers of 187 member States. The ILO defines Social Dialogue as “all types of negotiation, consultation or simply exchange of information between, or among, representatives of governments, employers and workers, on issues of common interest relating to economic and social policy.” Social Dialogue can take numerous forms and the ILO recognizes the followingFootnote ⁵⁷:

• “Negotiation, consultation and information exchange between and among governments, employers’ and workers’ organizations;

• Collective bargaining between employers/employers’ organizations and workers’ organizations;

• Dispute prevention and resolution; and

• Other approaches such as workplace cooperation, international framework agreements and social dialogue in the context of regional economic communities.”

In Europe, social dialogue encompasses bi-partite dialogue between employer organizations and trade unions. Importantly, the Treaty on the Functioning of the European Union (TFEU) recognizes and promotes the role of the social partners in the EU, who can contribute to policy-making and design and implement national reforms in the social and employment areas, both at national and European level. Their involvement in policymaking has been acknowledged in Guideline 7 of Council Decision 2018/1215 for the employment policies of the Member States, as well as in Principle 8 of the European Pillar of Social Rights.Footnote ⁵⁸ Some recent examples of social dialogue themes are education, skills and training, the circular economy, climate change, telework, and the right to disconnect.Footnote ⁵⁹

The European Social Dialogue can also be tripartite and involve public authorities. It then refers to discussions, consultations, negotiations and joint actions involving European Social Partners, who are organizations working at EU level and taking part in consultations and negotiating agreements.Footnote ⁶⁰ Tripartite social dialogue contributes to the construction of EU economic and social policies, and has a role in strengthening democracy, social justice and a productive and competitive economy. The association of employers, workers organizations and governments, in the design and implementation of economic and social policies, allows for a balanced consensus in such policies and the taking into account of the interests of all the parties involved.Footnote ⁶¹

17.4.3 The Autonomous Framework Agreement on Digitalization

Social partners conduct bi-partite negotiations at inter-sectoral, sectoral or company level, which can result in autonomous agreements. Articles 154 and 155 of the TFEU provide a legal basis to negotiate Framework Agreements, which are contractually binding on social partners and their members.

In the field of digitalization, one of the core agreements that European social partners have negotiated and signed in 2000 concerns the “Autonomous Framework Agreement on Digitalization.”Footnote ⁶² This agreement is the result of challenging negotiations between the European Trade Union Confederation (ETUC), BusinessEurope, the European Centre of Employers and Enterprises providing Public Services and Services of general interest (CEEP) and the Association of Crafts and SMEs in Europe (SMEunited). It represents a shared commitment of the European cross-sectoral social partners to optimize the benefits and deal with the challenges of digitalization in the world of work.

The rationale of the Agreement is that digital technologies impact four interrelated dimensions: work content (skills), work organization (employment terms and conditions, work-life balance), working conditions (work environment, health and safety, physical and mental demands, well-being, climate, comfort, work equipment) and work relations (relations or interpersonal relations that can impact the performance and the well-being of the workers). To manage the interrelationship of these dimensions, the Agreement specifies that in addition, four issues need to be considered:

1. a) digital skills and securing employment: The challenge is to determine which digital skills and process changes are necessary, thereby allowing adequate training measures to be organized, and to foster digital transformation strategies in support of employment;

2. b) modalities of connecting and disconnecting from technology applications;

3. c) artificial intelligence, including the guarantee of the human-in-control principle; and

4. d) respect of human dignity and worker surveillance.Footnote ⁶³

European Social Partners recognize that AI systems have a valuable potential to increase the productivity of the enterprise, the well-being of the workforce and a better allocation of tasks between humans, between different parts of the enterprise, and between machines and humans. However, they also indicate that it is “important to make sure that AI systems and solutions do not jeopardize but augment human involvement and capacities at work.”Footnote ⁶⁴ They also stress that AI systems should be designed and operated in order to comply with legislation (including the GDPR), guarantee privacy rights and ensure the dignity of the individual worker.

In the Agreement, the European Social Partners also referred to the concept of “Trustworthy AI” which they aspire to implement, and defined it – based on the Ethics Guidelines of the Commission’s High-Level Expert Group on AIFootnote ⁶⁵ – as a concept that should meet three criteria:

These criteria should be met throughout the AI system’s entire life cycle and must be respected whenever AI systems are deployed in the world of work. The Agreement also acknowledges that there can be tensions between different principles, such as respect for human autonomy, prevention of harm, fairness and explicability of decision-making, and states that these tensions should be addressed. However, it does not provide mechanisms to do so.

By virtue of the Framework Agreement, social partners have taken the responsibility to implement the measures described therein at the national, sectoral, and enterprise level in all EU Member States. While, during the first year of the Agreement, their commitment focused on translating and disseminating its content, the second year is dedicated to the actual implementation of its measures.

17.5.1 The AI Act

In April 2021, the European Commission put forward a regulatory package on AI,Footnote ⁶⁷ with at its core the Regulation laying down harmonized rules on AI, hereafter referred to as the AI Act, published in the Official Journal of the EU on July 12, 2024.Footnote ⁶⁸ According to the European Commission, a legal framework on AI was needed “to foster the development, use and uptake of AI in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, including democracy, the rule of law and environmental protection.” The claim to be putting fundamental rights at the heart of its approach, has been the object of criticism.Footnote ⁶⁹

It should be noted that the AI Act is modeled after product market regulation, typically used for technical safety standards. Therefore it is not specifically designed to address social issues and does not provide workers with specific rights. To address social issues, another legal basis could have been added to the text. However, whichever legal basis, Recital 9 recognizes that it “should be without prejudice to existing Union law, in particular on data protection, consumer protection, fundamental rights, employment, and protection of workers,” and should not affect Union law on social policy and national labor law. Similarly, Article 2(11) emphasizes that the AI Act does not preclude the Union or Member States from maintaining or introducing laws, regulations, administrative provisions or collective agreements more favourable to workers. However, the disregard for labor-related issues may in fact diminish the legal protection currently afforded by labor law, posing a significant risk to be addressed.Footnote ⁷⁰

The EU Commission has recognized that the use of discriminatory AI systems in employment might violate many fundamental rights and lead to broader “societal consequences, reinforcing existing or creating new forms of structural discrimination and exclusion.”Footnote ⁷¹ Therefore, Annex III of the AI Act lists high-risks AI systems related to the employment sphere. Point 3 refers to systems relating to education and vocational training, Point 4 refers to systems that relate to employment, workers management, and access to self-employment. Are mentioned in particular: “AI systems intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates” and “AI systems intended to be used to make decisions affecting terms of work-related relationships, the promotion or termination of work-related contractual relationships, to allocate tasks based on individual behaviour or personal traits or characteristics or to monitor and evaluate the performance and behaviour of persons in such relationships.” As discussed in Chapter 12 of this book, high-risk AI systems are subjected to certain mandatory requirements.

In addition to regulating high-risks systems, the AI Act prohibits certain practices. In the context of work, Article 5(f) prohibits the use of AI systems “to infer emotions of a natural person in the areas of workplace and education institutions, except where the use of the AI system is intended to be put in place or into the market for medical or safety reasons.” Recital 44 states that such systems can “lead to discriminatory outcomes” and “be intrusive to the rights and freedoms of the concerned persons,” particularly considering the imbalance of power in the context of work or education combined with the intrusive nature of these systems. However, the prohibition does not apply to AI systems placed on the market strictly for medical or safety reasons, such as systems intended for therapeutical use. This exception allows the use of AI systems under the guise of medical or safety reasons, while the actual motive may be different.

Connected to this, it should be noted that the notion of “emotion recognition system” “does not include physical states, such as pain or fatigue, including, for example, systems used in detecting the state of fatigue of professional pilots or drivers for the purpose of preventing accidents” (Recital 18). Fatigue or pain are not considered as emotions and thus not subject to the prohibition established by Article 5(f).

Most of the burden to comply with the AI Act falls on providers. However, from a workplace perspective, very few employers are providers; the majority are deployers. It is therefore key for workers and their representatives to keep a close eye on how deployers comply, notably with their ex ante obligation to inform (or to inform and consult) workers or their representatives, under Union or national law, before putting into service or using a high-risk AI system at the workplace (Article 26(7)). If the conditions for those obligations in other legal instruments are not fulfilled, it still remains necessary to ensure that workers and their representatives are informed on the planned deployment of high-risk AI systems at the workplace (Recital 92).

Pursuant to Article 26 (and Annex VIII), deployers also have to comply with other specific obligations when deploying high-risk systems. These include transparency and information obligations to enable appropriate human oversight of high-risk systems (Articles 13 and 14), as well as disclosure obligations for the use of certain systems (Article 50). To secure that the provision of information (and consultation) occurs effectively, the AI Act encourages providers and deployers to ensure a sufficient level of AI literacy of their staff and other persons dealing with the operation and use of AI systems on their behalf (Article 4).

Finally, the AI Act addresses the role of trade unions in the context of stakeholder involvement and training. It emphasizes their involvement in the development and deployment of AI systems and voluntary codes of conduct (Recital 165). As social partners, they are not involved in the AI Office (Article 64), but they do have a role in the Advisory Forum established to advise and provide technical expertise to the European Artificial Intelligence Board and the Commission (Recital 150 and Article 67(2)).

17.5.2 The AI Liability Directive

The rules on AI could not have been complete without a liability regime. The EC’s AI Liability DirectiveFootnote ⁷² proposes a strict liability regime for non-contractual fault-based civil claims for damages arising from random events or incidents caused by an AI system, between entities not bound by a contract. The proposed directive is complementary to the AI Act and hence, focuses on high-risk AI systems considered products, whereby an injured person has to prove that an AI system caused damage. The key liable economic operator is the provider or user. The burden of proof for the injured person can be eased if certain conditions are met, in order to obtain compensation under national law. However, it is uncertain whether and how the AI liability directive applies to labor law. Three situations could arise. First, in cases where an AI system is involved in firing workers, the AI Liability Directive does not apply most likely due to the existence of an employment contract. Second, in cases where AI systems are used for recruitment services, the proposed directive does not apply because the AI system “only provided information or advice which was taken into account by the relevant human actor,” as Recital 15 states. Finally, in cases where a worker has suffered harm involving an AI system, the directive will probably not apply due to the contractual nature of these damages. However, other (safety) obligations in labor law may apply and the employer could be held liable on the basis of these obligations.

Finally, the AI liability directive is welcome as a first attempt to provide a harmonized regime to liability challenges that arise in an AI-context. It includes novel provisions, one of them being the disclosure of information, which could be potentially useful for deployers of AI systems. However, it remains unclear whether and how individuals can utilize the information disclosed effectively.Footnote ⁷³ The proposed directive also provides a rebuttable presumption of a causal link between the fault of the user of a high-risk AI system and the output of this AI system, when this user does not comply its obligations to use or monitor the AI system in accordance with the accompanying instructions.Footnote ⁷⁴ Despite its potential to substantially facilitate the proof of damages for workers, this presumption will not apply for many applications of work-related AI systems due to the contractual nature of the claims for these damages.

17.5.3 The New Machinery Regulation

Another piece of the “AI Package” concerns the new Regulation on Machinery 2023/1230, replacing Machinery Directive 2006/42/EC.Footnote ⁷⁵ This directive was based on the principles of the so-called “New Approach to Technical Harmonization and Standards.”Footnote ⁷⁶ It set a range of minimum health and safety requirements that machinery must fulfill to be placed on the market and the conformity assessment through which it can be demonstrated that the machine indeed fulfils them. Machines that meet these requirements are said to have a “presumption of conformity.” Given the role that machinery plays in many work environments, the directive was key to ensure workers’ safety, mainly in the industrial and manufacturing sectors.

An evaluation study conducted by the European Commission in 2018 identified a number of issues with the directive, and found that it required more efficiency. The study concluded that a new version of the legislation was needed, ideally in the form of a regulation, to facilitate the homogenous application of and alignment with horizontal rules on essential health and safety requirements to guarantee that all pieces of industrial machinery (interchangeable equipment, safety components or lifting accessories) are safe to use at work.

Beyond an update of the directive’s provisions, the new Machinery Regulation also proposes Essential Health and Safety Requirements (EHSRs) listed in Annex III that address the latest developments in digital technology, including the integration of AI systems and Internet of Things (IoT) into machinery equipment and the collaboration between human and robots. The EU Commission’s underlying rationale is that although the risks of AI systems are regulated by the AI Act, the entire machinery needs to be safe, considering the interactions between the machinery components, including AI systems. The EU Commission states that “machines are becoming more powerful, autonomous and some look almost like humans, which requires adapting the EHSRs related to the contact between the human and the machinery.”Footnote ⁷⁷ The regulation provides specifically that:

1. (a) For evolving machines: in the risk assessments, manufacturers will need to include those risks appearing after the machinery is placed on the market due to its evolving and autonomous behaviour;

2. (b) On ergonomics: under the intended conditions of use, the discomfort, fatigue and physical and psychological stress faced by the operator shall be reduced to the minimum possible, taking into account ergonomic principles;

3. (c) Regarding risks related to moving parts and psychological stress: the prevention of risks “shall be adapted to human-machine coexistence, in a shared space without direct collaboration, and human-machine interaction.”

The new Regulation also addresses “the risks related to ‘moving parts’ (accidents in human-robot collaboration), cyber-safety aspects in the connected machinery, and software updates after the placing on the market of the machinery product which might change functionality.”Footnote ⁷⁸ The Regulation was published in the Official Journal of the EU on June 29, 2023, but it will only become applicable from January 2027 onwards.

17.5.4 The Directive on Improving Working Conditions in Platform Work

A third important EU legislative initiative concerns the directive on Improving Working Conditions in Platform Work.Footnote ⁷⁹ This directive is the first piece of EU labor law that regulates automated monitoring and decision-making systems. In the European Commission’s view, platform work is developing rapidly, raising new challenges relating to working conditions, algorithmic management, access to social protection and benefits, and collective representation and bargaining. These concerns were more visible during the pandemic, given the increased reliance on platform work in this period.

The work that led to the directive was a two-phase consultation of the European Social Partners, through which the European Commission identified four specific challenges: (1) the employment status of platform workers, (2) the algorithm-based business model of the platforms, (3) the cross-border nature of platform work, and (4) the existence of regulatory gaps at EU level. Nicolas Schmit, Commissioner for Jobs and Social Rights, made an important statement referring to the role of social partners in this field: “We cannot lose sight of the basic principles of our European social model (…) and social partners’ views on this will be key in finding a balanced initiative for platform work in the EU.”Footnote ⁸⁰ Therefore, the Commission decided to propose a new directive that could help address these concerns.

The directive has two main goals: (1) to improve the working conditions of platform workers by facilitating the correct determination of their employment status through a rebuttable legal presumption, (2) to improve the protection of the personal data of platform workers by improving transparency and accountability in the use of automated monitoring and decision-making systems. The directive includes an innovative chapter on algorithmic management, with hybrid labor and data protection provisions. Article 7, in particular, limits the processing of personal data, notably biometrics data, or data related to the emotional or psychological state of workers. Article 7 not only applies to automated monitoring systems and automated decision-making systems, but also to automated systems supporting or taking decisions that affect persons performing platform work in any manner.

As the processing of workers’ data by automated monitoring and decision-making systems is likely to result in a high risk to workers’ rights, platforms must carry out a Data Protection Impact Assessment, following GDPR requirements (Article 8). They also must respect transparency and information obligations, in relation to the systems they use to take or support decisions that affect workers and their working conditions (Article 9). They must ensure human oversight (Article 10), with the involvement of workers’ representatives, of the impact of individual decisions taken or supported by their systems. Finally, workers have the right to obtain an explanation from the platform for any decision taken or supported by their systems, as well as the right to review it (Article 11). In relation to health and safety, platforms must evaluate the risks of automated monitoring or decision-making systems, in particular possible work-related accidents, as well as psychosocial and ergonomic risks. They may not use automated monitoring or decision-making systems in any manner that puts undue pressure on workers or puts at risk their safety and their physical and mental health (Article 12).

17.5.5 International Initiatives

The EU is not the only jurisdiction that is taking legislative action on AI. Several international organizations worked simultaneously in this sphere, and their outcomes will impact the labor law context. In 2024, the OECD revised the “Principles for responsible stewardship of trustworthy AI.” Initially adopted in 2019,Footnote ⁸¹ the Principle related to “transparency and explainability” mentions the need for AI actors “to commit to transparency and responsible disclosure regarding AI systems,” and “to make stakeholders aware of their interactions with AI systems, including in the workplace.”Footnote ⁸² At the same time, the Council of Europe, through its ad hoc Committee on Artificial Intelligence (CAHAI), and its successor the Committee on AI (CAI), adopted an international legally binding convention. The “Framework Convention on Artificial Intelligence, Human Rights, Democracy and the rule of law” aims to set “minimum standards for AI development” based on the Council of Europe’s standards of human rights, democracy and the rule of law.Footnote ⁸³ Similarly to the AI Act, the Council of Europe’s convention deals with AI systems according to a risk-based assessment, and imposes new obligations based on the level of risk posed by the systems. It does not make reference to labour issues. At the United Nations, the ILO has the intention to develop a Policy Observatory on AI and Work in the Digital Economy, and to analyse the implications of shifts in AI regulation for decent work.Footnote ⁸⁴ These are but a few of the initiatives that international organizations are establishing in this context.

Table 17.1 maps the key legal sources that are relevant for the use of AI systems in the workplace.

18.2.1 The Emergence of Predictive Policing in Europe

The origins of predictive policing can be found in the police strategy “Intelligence-led policing”, which emerged in the 1990s in Europe.Footnote ⁸ Intelligence-led policing can be seen as “a business model and managerial philosophy where data analysis and crime intelligence are pivotal to an objective, decision-making framework that facilitates crime and problem reduction, disruption and prevention through both strategic management and effective enforcement strategies that target prolific and serious offenders.”Footnote ⁹ One of the developments within intelligence-led policing was prospective hotspot policing, which focused on developing prospective maps. Using knowledge of crime events, recorded crime data can be analyzed to generate an ever-changing prospective risk surface.Footnote ¹⁰ This then led to the development of one of the first predictive policing applications in the United Kingdom, known as ProMap.Footnote ¹¹ Early in the twenty-first century, the rise of the use of predictive machine learning led to what is now known as predictive policing.

Predictive policing refers to “any policing strategy or tactic that develops and uses information and advanced analysis to inform forward-thinking crime prevention.”Footnote ¹² It is a strategy that can be situated in a broader preemptive policing model. Preemptive policing is specifically geared to gather knowledge about what will happen in the future with the goal to intervene before it is too late.Footnote ¹³ The idea behind predictive policing is that crime is predictable and that societal phenomena are, in one way or another, statistically and algorithmically calculable.Footnote ¹⁴

Although already being implemented since the beginning of the twenty-first century in the United States (US), Law Enforcement Agencies in Europe are increasingly experimenting with and applying predictive policing applications. Two types can be identified: predictive mapping and predictive identification.Footnote ¹⁵ According to Ratcliffe, predictive mapping refers to “the use of historical data to create a spatiotemporal forecast of areas of criminality or crime hot spots that will be the basis for police resource allocation decisions with the expectation that having officers at the proposed place and time will deter or detect criminal activity.”Footnote ¹⁶ Some law enforcement agencies use or have used software developed by (American) technology companies such as PredPol in the UK and Palantir in Denmark, while in other countries, law enforcers have been developing their own software. Examples are the Criminality Awareness System (CAS) in the Netherlands, PRECOBS in Germany, and a predictive policing algorithm developed in Belgium by criminology researchers in cooperation with the police.Footnote ¹⁷ Predictive mapping applications have in most cases focused on predicting the likelihood that a certain area is more prone to burglaries and adjusting patrol management according to the predictions.

Predictive identification has the goal to predict who is a potential offender, the identity of offenders, criminal behavior, and who will be a victim of crime.Footnote ¹⁸ These types of technologies build upon a long history of using risk assessments in criminal justice settings.Footnote ¹⁹ The difference is that the risk profiles are now often generated from patterns in the data instead of coming from scientific research.Footnote ²⁰ This type of predictive policing has been mainly applied in Europe in the context of predicting the likelihood of future crime (recidivism). However, other examples can be found in the use of video surveillance that deploys behavior and gait recognition. There are also developments in lie and emotion detection,Footnote ²¹ the prediction of radicalization on social media,Footnote ²² passenger profiling, and the detection of money laundering.Footnote ²³ A recent example can be found in the Netherlands where Amsterdam police uses what is known as the Top400. The Top400 targets 400 young “high potentials” in Amsterdam between twelve and twenty-four years old “that have not committed serious offences but whose behavior is considered a nuisance to the city.”Footnote ²⁴ In the context of the Top400, the ProKid+ algorithm has been used to detect children up to sixteen years old that could become “a risk” and might cause future crime related problems. When on the list, youngsters receive intensive counseling and they and their families are under constant police surveillance.Footnote ²⁵

18.2.2 Effectiveness of Predictive Policing

Evaluations of the effectiveness of predictive policing in preventing crime have, so far, been inconclusive due to a lack of evidence.Footnote ²⁶ In addition, not all evaluations have been conducted in a reliable way, and with general falling crime rates, it is hard to show that fall in crime is the result of the technology. Moreover, it is difficult to evaluate the technology’s effectiveness in preventing crime as algorithms identify correlations, not causality.

For instance, the Dutch Police Academy concluded in their evaluation of the CAS system that it does seem to prevent crime but that it does have a positive effect on management.Footnote ²⁷ The evaluation study conducted by the Max Planck Institute in Freiburg of a PRECOBS pilot-project in Baden-Württemberg concluded that it remains difficult to judge whether the PRECOBS software is able to contribute toward a reduction in home burglaries and a turnaround in case development. The criminality-reducing effects were only moderate and crime rates could not be clearly minimized by predictive policing on its own.Footnote ²⁸ In Italy, reliability of 70 percent was found for the predictive algorithm of KEYCRIME which predicted which specific areas in Milan would become a crime hotspot.Footnote ²⁹ In their overview of recent challenges and developments, Hardyns and Rummens did not find significant effects of predictive policing and argue that more research is needed to assess the effectiveness of current methods.Footnote ³⁰

Apart from inconclusive evaluations, several police forces stopped using the software altogether. For instance, the use of PredPol by Kent Police was discontinued in 2019 and the German police forces of Karlsruhe and Stuttgart decided to stop using PRECOBS software because there was insufficient crime data to make reliable predictions.Footnote ³¹ Furthermore, amid public outcry about the use of PredPol, the Los Angeles Police Department in the US stopped using the software, yet at the same time it launched a new initiative: “data-informed community-focused policing (DICFP).”Footnote ³² The goal of this initiative is to establish a deeper relationship between community members and police, and to address some of the concerns the public had with previous policing programs. However, critics have raised questions about the initiative’s similarities with the use of PredPol.Footnote ³³ Similar to PredPol, the data that is fed into the system is biased and often generated through feedback loops. Feedback loops refers to a phenomenon identified in research that police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.Footnote ³⁴

Regarding predictive identification, almost no official evaluations have been conducted. Increasingly, investigative journalists and human rights organizations are showing that there is significant bias in these systems.Footnote ³⁵ Moreover, issues that have been raised with the effectiveness of actuarial risk assessment methods before it was digitalized, such as the (un)reliability of the risk factor research that underscores the applied theories of crime, are not solved by implementing algorithmic decision-making.Footnote ³⁶ As to the use of predictive analytics in this area, the effectiveness of these systems likewise remains unclear. An assessment of a predictive model used by Los Angeles’ children’s services, which was promoted as highly effective in practice, “produced a false alarm 96 percent of the time.”Footnote ³⁷

In general, the effectiveness concerns that were already identified for (prospective) hot-spot policing on the one hand and traditional risk assessments on the other, prior to the implementation of AI systems, did not disappear. With regards to predictive mapping spatial displacement, which is when crime moves to a different area after implementing a control measure such as CCTV or increased police presence is but one example.Footnote ³⁸ It should also be noted that the long-term impacts of predictive policing on individuals and society are unclear and longitudinal research assessing this is not conducted. Finally, as demonstrated by the earlier overview, it is unclear if the adoption of predictive mapping will reduce overall crime, and whether it will be able to do so for different types of crime.Footnote ³⁹

18.3.1 Legal Issues

The European Union regulation on AI, published by the European Commission in 2024, provides numerous safeguards depending on how much risk a certain AI application poses to fundamental rights.Footnote ⁴⁰ As Chapter 12 of this book more extensively explains, the AI Act classifies AI systems into several categories, including low or limited risk (not subject to further rules), medium/opacity risk (with new transparency obligations), high risk (with a broad set of conformity assessment requirements), and unacceptable risk (which are prohibited).

In its amendments published in June 2023, the European Parliament clearly opted for stricter safeguards by removing exceptions for law enforcement’s use of real-time remote biometric identification systems, and prohibiting some applications that the Commission had previously classified as high risk, such as predictive policing, and more specifically predictive identification applications used in criminal justice.Footnote ⁴¹ However, ultimately, the final text provides extensive exceptions for law enforcement when it comes to real-time remote biometric identification systemsFootnote ⁴² and does not prohibit place-based predictive policing. It does prohibit predictive identification in so far the risk assessments are solely based on the “profiling of a natural person or on assessing their personality traits and characteristics.”Footnote ⁴³ It remains to be seen to what extent the interpretation, implementation, and enforcement of the regulation will provide sufficient democratic safeguards to protect the fundamental rights of citizens.

In addition to the AI regulation, the use of AI for law enforcement purposes is also regulated by the transposition into national laws of member states of the Law Enforcement Directive (LED).Footnote ⁴⁴ The application of this directive concerns the processing of personal data by competent authorities for the prevention, investigation, detection, and prosecution of criminal offenses or the execution of criminal penalties.Footnote ⁴⁵ It does not apply in the context of national security, to EU institutions, agencies, or bodies such as Europol, and it only applies to processing of personal data wholly or partly by automated means. The directive came about primarily out of the need felt by law enforcement agencies, including in response to terrorist attacks in the US and Europe in the first decades of the twenty-first century, to exchange data between member states. The directive, therefore, aims to strike a balance between law enforcement needs and the protection of fundamental rights.Footnote ⁴⁶

The focus of the directive is on “personal data.” This is defined as “any information relating to an identified or identifiable natural person (‘data subject’).”Footnote ⁴⁷ An identifiable natural person is one who can be “identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural, or social identity of that natural person.”Footnote ⁴⁸ Already in 2007 the European advisory body, the Data Protection Working Party Article 29 (WP29),Footnote ⁴⁹ proposed a very broad interpretation of personal data: “any information” includes not only objective and subjective information, but even false information. It does not just concern private or sensitive information.Footnote ⁵⁰ Information can be associated with an individual in three ways: (1) content (when it is about a particular person); (2) purpose (when data is used to evaluate, treat, or influence an individual’s status or behavior in a certain way) and (3) result (when it is likely to have an impact on the rights and interests of a particular person taking into account all the circumstances of a particular case).Footnote ⁵¹

It is often questioned, especially by law enforcement agencies themselves, if predictive mapping applications process “personal data.” Lynskey argues that based on the advice of WG 29 and case law, it is possible to conclude that data processing in predictive mapping involves the processing of personal data.Footnote ⁵² The data processed are potentially linked to the data subject because of the purpose (to treat people in a certain way) or the effect (impact on those identified in the hotspots). Regarding predictive identification, it is clearer that personal data are processed, both when it comes to the input data (as the content concerns the data subject) and the output data (as the purpose and effect of the data are used to influence the prospects of an identified individual). In practice, however, interpretations diverge. For instance, in the case of the CAS system in the Netherlands, the Dutch law enforcement authority nevertheless concluded that it is not processing personal data and, therefore, that the data protection regulation does not apply to the system’s use.Footnote ⁵³ This example shows that lack of clear guidance and specific regulation when it comes to the use of AI by law enforcement raises questions about the effectiveness of the current legislative safeguards for these applications.

18.3.2 Ethical and Social Issues

Predictive policing raises several ethical and social issues. These issues are dependent on what type of technology is implemented and the way the technologies are governed.Footnote ⁵⁴ They can not only impact the effectiveness and efficacy of the technology, but they can also cause harm.Footnote ⁵⁵ Below, I respectively discuss concerns pertaining to efficacy, governance, organization, and individual and social harms.

19.4.1 Impacting Fundamental Rights

One of the most common challenges arising from use of algorithmic regulation is its impact on fundamental rights, and most notably the right to privacy and data protection.Footnote ³⁷ The performance of algorithmic systems hinges on the availability, collection and processing of high volumes of (personal) data, especially when used to enable scaled decision-making about individuals. Such data collection can have vastly intrusive effects on people’s lives and can potentially be used in ways that undermine their agency.Footnote ³⁸

In the worst case, the irresponsible implementation of algorithmic regulation not only breaches privacy legislation (such as in the Dutch case discussed earlier), but can also lead to the mass-surveillance of citizens, in the name of efficiency. Unjustified intrusions into people’s private lives can also affect the value of democracy, especially when data is gathered about potential individuals or groups that are not favored by the government. Moreover, privacy is often instrumental to secure other fundamental rights, such as the right to free speech and the right to human dignity, which are hence also at stake.Footnote ³⁹ Yet given the financial investments that public administrations undertake when they decide to develop, procure or implement algorithmic regulation, and given the path dependencies this brings along, administrations are only incentivized to gather ever more data. A balance must hence be found between the government’s desire to exercise its tasks with more efficiency, and the protection of people’s private lives.

Another important concern relates to the way in which algorithmic systems can affect the right to non-discrimination.Footnote ⁴⁰ Human beings have a range of biases and prejudices, which can at times be unjust or discriminatory, for instance when they echo societal stereotypes, historical inequalities, or other (often unconscious) problematic influences. Since algorithmic systems are designed and developed by human beings, their output can reproduce these unjust human biases, thus mirroring and potentially even exacerbating discriminatory practices.Footnote ⁴¹ The earlier illustrations of algorithmic regulation have shown that this risk is not hypothetical. In the Dutch example, inequalities found their way into the system’s parameters and dataset during the development phase, thus adversely affecting certain groups of the population. In France and in the UK, the algorithmic system was deployed in an unequal manner and hence exposed individuals to its impact in an uneven way.

At a much more banal level, it is also possible that flaws or errors seep into the development and deployment process of algorithmic regulation – whether through erroneous human input or invalid correlations and inferences drawn by the system. This risk, and the problematic consequences ensuing therefrom, was illustrated by the benefits-allocation system deployed in the US, yet many other examples exist.Footnote ⁴² More generally, the fact that public administrations are responsible for the allocation of basic socio-economic rights also renders these rights vulnerable when their implementation is wholly or partly outsourced to a flawed or biased system.

Evidently, public administrations can also adversely affect people’s rights without the use of algorithmic systems. Yet their reliance on powerful tools that enable data processing at a much wider scale and higher speed, coupled with the opacity of the internal processes of these systems and the recurrent lack of transparency about their deployment not only aggravate these risks, but also makes it more difficult to discover them. Adding a layer of digitalization to public services can certainly provide benefits of scale, but it is also precisely this scale-element that renders it so risky when implemented without considering these concerns.

19.4.2 Eroding the Rule of Law

Algorithmic regulation also raises a number of challenges to the rule of law. In essence, the rule of law comes down to the idea that nobody stands above the law, and that citizens and government officials alike are subjected to legal rules.Footnote ⁴³ It embodies the notion that, rather than being governed by the arbitrary whims of “men” (who, as Aristotle already pointed out, are susceptible to arbitrary passions that undermine rational thinking),Footnote ⁴⁴ people should be governed by “laws,” which are based on reason and offer predictability. The rule of law is a broad term that has been defined in countless ways,Footnote ⁴⁵ yet the Council of Europe’s Venice CommissionFootnote ⁴⁶ provided a helpful conceptualization for the European legal order by breaking it down into several principles (which were subsequently taken over by the European Union).Footnote ⁴⁷ Under this conceptualization, the rule of law encompasses six principles: (1) the principle of legality; (2) the principle legal certainty; (3) the prohibition of arbitrariness of executive power; (4) equality before the law; (5) effective judicial protection, with access to justice and a review of government action by independent and impartial courts, also as regards human rights; and (6) the separation of powers.Footnote ⁴⁸ Under this conceptualization, the rule of law is hence understood not only as requiring procedural safeguards, but also substantive ones.Footnote ⁴⁹ Indeed, it is not enough to merely apply the law in an efficient and procedurally agreed upon manner. Otherwise, the law could simply be used as a (powerful) instrument to enforce illiberal and authoritarian policies – a practice that has been denoted as rule by law instead.Footnote ⁵⁰ Rather, the law and its application should also protect and comply with substantive values, and particularly respect for human rights and democracy. Accordingly, whenever public administrations want to exercise their powers, they are constrained by these rule of law-principles, which ensure that the law plays a protective role in society.Footnote ⁵¹

At first glance, algorithmic regulation seems rather innocuous from a rule of law-perspective, and could even be seen as potentially advancing its principles to a greater extent. By eliminating civil servants’ discretion from the picture, along with their potentially inconsistent or arbitrary decision-making, algorithmic regulation could arguably catalyze Aristotle’s aspiration of being ruled by law instead of men. However, on closer inspection, this unqualified relationship between the rule of law and the rule of men is too simplistic: just like laws are created, applied and interpreted by human beings, so are algorithmic systems inherently dependent on the human beings that design, develop and deploy them.Footnote ⁵² There are hence several distinctive challenges to the ideal of the rule of law when public administrations rely on algorithmic systems,Footnote ⁵³ of which an important one relates to the very act of implementing the law in an automated manner.

Automating the law’s application through algorithmic regulation requires a “translation process” from text-based laws and policies to digital code. To a greater or lesser extent, text-based provisions are inevitably open to different (and contestable) interpretations, sometimes inadvertently, sometimes purposely. It is often precisely this very openness of the law that enables it to play its protective role, by facilitating its tailored interpretation to the specific situation at hand. Indeed, laws typically consists of (overly) general rules set forth by the legislator, which must subsequently be interpreted and applied to concrete cases. However, once automated, the law becomes more rigid, as a particular interpretation must be codified or optimized for. There is thus a risk that this translation process changes the nature of the law in a problematic manner. The translation may, for instance, occur in a way that is too legalistic, that incorporates biases and inequalities, that deviates from the intent of the legislator and unwarrantedly bolsters the power of the executive, that undermines the predictability and congruence of the law’s application, that erodes essential rights and liberties or limits their scope, or that leaves individuals unable to contest the chosen interpretation and subject it to judicial review.

This risk not only undermines the rule of law’s principles, but also erodes the constraints they place on government power, to the detriment of other liberal democratic values. In other words, algorithmic regulation could turn into a powerful tool to enforce rule by law. While this risk is not limited to the algorithmic context, the automated application of the law could be a highly efficient tool to erode the very protection the law is meant to afford, on a broader scale than ever before – a threat I conceptualized as algorithmic rule by law.Footnote ⁵⁴ Bearing in mind the inherent malleability of software, and the additional level of opacity it introduces, it is thus essential to remain vigilant and put the necessary safeguards in place to ensure public administrations do not sacrifice efficiency over human rights, democracy and the rule of law.

19.4.3 Delegating Responsibility

Despite the push of the NPM movement to reconceptualize public administrations as “service providers” toward “customers,” citizens are more than just customers. A far more inherent power imbalance is at play between governments and individuals as, unlike in private settings, the latter cannot easily “shop” at another service provider when for example, their social welfare benefits are wrongfully denied. As Sofia Ranchordas and Luisa Scarcella pointed out, this power asymmetry can also exacerbate citizens vulnerabilities, and even instigate dehumanizing effects.Footnote ⁵⁵ This risk can be spurred by the datafication process that accompanies the use of algorithmic regulation,Footnote ⁵⁶ as it inevitably reduces individuals to numbers in a quite literal sense, thus diminishing their individuality and potentially even their human dignity. Yet it is far easier to overlook one’s responsibility for the well-being of a number than for the well-being of an individual human being.

One of the distinctive elements of algorithmic regulation concerns the elimination of the need for direct personal interactions between individuals and civil servants, as such interactions can instead be mediated through algorithmic systems. At the same time, this time-saving feature can also make it more difficult for citizens to interact with another human being that understands their needs and concerns, and that can rectify any erroneous information that public administrations may have (which is a common difficulty with networked databases).Footnote ⁵⁷ It can also render it more challenging for individuals to receive a clear explanation of the decisions affecting them, or to voice their concerns when problematic administrative acts are taken about them. In other words: it might be far more difficult for them to be heard, and to be acknowledged in their human individuality.Footnote ⁵⁸

In this regard, it is also important to consider the role of discretion, “a power which leaves an administrative authority some degree of latitude as regards the decision to be taken, enabling it to choose from among several legally admissible decisions the one which it finds to be the most appropriate.”Footnote ⁵⁹ Civil servants use this discretion when they decide how to apply general rules to specific cases in the most appropriate way, in line with the rule of law. However, when public administrations rely on algorithmic regulation, this typically reduces discretion at the level of individual officials. Instead of officials exercising their judgment in specific cases, it is the system that will “apply” the law to a given case and take or suggest a decision.Footnote ⁶⁰ Even when the system merely offers a suggestion, officials will often be strongly incentivized to follow it for reasons of efficiency and the system’s perceived authority. Indeed, if they would want to deviate from the system’s suggestion, they typically need to provide a justification for this deviation, which not only requires time but also space for critical judgment to go against the system’s centralized and allegedly more commanding suggestion.Footnote ⁶¹

Civil servants might also feel that, by relying on the system’s outcomes, they can delegate or at least share responsibility for the decision they take.Footnote ⁶² Especially when they do not see or talk with the individual concerned, the distance often renders it easier and more convenient to delegate a decision to an algorithmic system, though that could also lead to an (at least psychological) delegation of responsibility for that decision, and thus for the potential adverse consequences in case it is wrong or unjust. This can, in turn, increase the likelihood of negligence, nurture a lack of concern for citizens’ interests and how they are impacted, and more generally diminish the procedural legitimacy of decisions taken by public administrations. Prior to the implementation of algorithmic regulation, it is hence important to anticipate and mitigate this challenge.

19.4.4 Delegating Public Power

At the same time, it must be pointed out that administrative discretion does not disappear when algorithmic regulation is deployed. While it is significantly reduced at the level of individual civil servants, it is instead transferred to the level of the designers and developers of the algorithmic systems, who through their seemingly technical choices in fact shape the system’s highly normative outcomes.Footnote ⁶³ A related problem is the fact that these designers and developers are typically not the civil servants who have the necessary experience and training to adopt administrative acts, who have expertise on how a specific law should be applied, and who must abide by the public sector’s deontological standards. Rather, these algorithmic systems are often developed by data scientists and engineers working for private companies. This raises questions about the influence of the private sector on public decision-making, given the normative relevance of the systems they develop for this purpose. The less a public administration can count on in-house infrastructure and knowledge about how algorithmic systems work and what their capabilities and limitations are, the more this can lead to a problematic dependency on actors that are driven by nonpublic values.Footnote ⁶⁴ The use of algorithmic regulation should however never lead to the unwarranted delegation of public powers to private actors.

The COVID-19 pandemic, for instance, made many public administrations aware of the fact that they were utterly dependent on private actors to set up and use digital infrastructures for their day-to-day operations (many of which had to move entirely to the digital realm) and for their management of the pandemic itself (for instance through contact tracing apps).Footnote ⁶⁵ This also contributed to concerns around “digital sovereignty,” or a nation’s ability to autonomously decide on its relationship with (providers of) digital technology.Footnote ⁶⁶ As discussed elsewhere, digital sovereignty implies the exercise of control over two entwined elements, namely: (1) the normative values that underly the technology, and (2) the physical and socio-technical digital infrastructure that enables the technology.Footnote ⁶⁷ Sovereignty over both of these elements is essential to ensure that the core values which public administrations ought to protect – including human rights, democracy and the rule of law – are safeguarded also when they deploy algorithmic regulation.

19.5.1 Constitutional Law and Administrative Law

The most primary of these rules are enshrined in national constitutions and set out the competences that governments have when exercising their powers, as well as the limitations of such powers. Fundamental rights and freedoms – such as the right to equality and nondiscrimination, freedom of speech and association, the right to privacy, and the right to a fair trial – are typically part of constitution-level norms, either directly or through (international) human rights treaties. This enables (constitutional) courts to review public administrations’ actions in light of these limitations and safeguards. To carry out judicial review, it should not matter whether the administration’s actions were taken solely by human civil servants or with the help of algorithmic systems.

A more detailed set of rules can be found in the realm of administrative law. While this area of law emerged as a scientific study in Europe around the nineteenth century, as a body of law its was already established prior to that time in several countries.Footnote ⁶⁸ In essence, administrative law sets out the contours of the space of action of public administrations, drawing on constitutional norms and principles. It can thus be seen as limiting but also legitimizing and empowering administrations and the discretion they exercise when implementing the rules established by the legislative branch of power.Footnote ⁶⁹ While each country has its own administrative law rules, some commonalities can be identified, as these rules are closely associated with the rule of law principles mentioned earlier.Footnote ⁷⁰ They have sometimes also been conceptualized under the concept of “good governance” or “good administration.”Footnote ⁷¹ In what follows, let me discuss some of the principles that the Council of Europe’s Committee of Ministers laid down in its “Code of Good Administration” (“the Code”).Footnote ⁷²

The first concerns the principle of lawfulness, which broadly corresponds to the concept of “legality.”Footnote ⁷³ It states that public authorities must act in accordance with the law (including domestic, supranational and international law), and cannot take any arbitrary measures, also when exercising discretion. This means they must have a legal basis to act, in accordance with the rules that define their powers and procedures, and they can only use the powers conferred upon them for the purpose delimited in those rules. The interpretation of how this principle must be complied with in practice differs from state to state, but given the intrusiveness of algorithmic systems, many countries in Europe provide that outsourcing certain tasks to such systems requires a specific legal basis. Accordingly, the legislative branch will typically need to set out the conditions under which public administrations can rely on algorithmic regulation. Additionally, public administrations have the obligations to ensure that – when they deploy algorithmic regulation – this occurs in full compliance with existing legislation, including the protection of human rights.

The principle of equality is likewise mentioned in the Code and provides that public administrations must treat persons who are in the same situation in the same way.Footnote ⁷⁴ Any difference in treatment must be objectively justified – and merely claiming that an algorithmic system makes unintended discriminatory distinctions would not be a sufficient justification. Linked thereto is the principle of impartiality, which the Code conceptualizes as ensuring that public administrations act objectively, having regard only to “relevant matters” when they adopt administrative acts, and that they should not act in a “biased manner.”Footnote ⁷⁵ Individual public officials, too, must carry out their duties in an impartial manner, irrespective of their personal beliefs and interests. Applied to the context of algorithmic regulation, these principles hence require public administrations to ensure preemptively that the tools they deploy do not provide biased outcomes, and do not take into account elements that are not relevant for the administrative act in question. The latter point is especially interesting, since data-driven systems typically function by correlating different information points that may not necessarily have a causal link with the matter at hand. Importantly, respect for these principles must also be ensured when administrations procure algorithmic applications; they cannot escape this obligation by outsourcing the system’s development, but remain responsible to respect these principles also when they make use of (privately developed) systems.Footnote ⁷⁶ Public administrations must hence exercise a certain standard of care before they take specific actions, which arguably also extends to the action of implementing algorithmic regulation.

The Code’s principle of proportionality is also of relevance: measures affecting the rights or interests of individuals should only be taken where necessary “and to the extent required to achieve the aim pursued.”Footnote ⁷⁷ This principle is particularly important whenever civil servants exercise discretion, as it also states they must maintain “a proper balance between any adverse effects which their decision has on the rights or interests of private persons and the purpose they pursue,” without these measures being excessive.Footnote ⁷⁸ Given the problematic examples of algorithmic regulation discussed earlier, the proportionality principle plays an important role in the algorithmic context, especially when discretion is shifted away from individual civil servants who are able to balance different rights and interests, toward (designers of) algorithmic systems that rely primarily on pre-codified rules or optimization functions.

The Code also includes the principle of participation, which emerged more recently.Footnote ⁷⁹ This principle is closely connected to the notion of (deliberative) democracy and embodies the idea that public administrations should offer individuals the opportunity to participate in the preparation and implementation of administrative decisions which affect their rights or interests. As I argued elsewhere, one could claim that participation should not only extend to administrations’ decision-making processes based on algorithmic regulation, but also to the very choice taken by public administrations to implement algorithmic regulation in the first place.Footnote ⁸⁰

Finally, I should point out the principle of transparency, which states that public administrations must ensure that individuals are informed, by appropriate means, of their actions and decisions.Footnote ⁸¹ This may also include the publication of official documents and should in any case encompass respect for the rights of access to official documents according to the rules relating to personal data protection. A debate exists about the extent to which this principle applies to algorithmic systems used by public administrations, in so far as the source code of these algorithms (or at least their parameters) can be said to constitute information that falls under such access rights.Footnote ⁸²

This brings me to the observation that existing administrative law rules undoubtedly apply when public administrations use algorithmic systems, yet the enforcement of such rules is often rendered more difficult in this context precisely due to the nature and features of such systems, and civil servants’ unfamiliarity with the particular challenges they pose. Partly for this reason, more specific legal rules have been developed to protect individuals when their personal data is processed in an automated way, and when they are subjected to AI systems – two domains I will discuss next.

19.5.2 Data Protection Law

In the EU legal order, the right to privacy and personal data protection is enshrined respectively in Articles 7 and 8 of the Charter of Fundamental Rights of the EU (CFR) and in Article 8 of the European Convention on Human Rights. These fundamental rights were further concretized in other legal instruments. At the level of the Council of Europe, Convention 108 for the protection of individuals with regard to the processing of personal data was already opened for signatures in 1981, being the first legally binding international instrument in the field of data protection (later modernized through Convention 108+ in 2018).Footnote ⁸³ Unsurprisingly, the Council of Europe’s 2007 Code of Good Administration also included the principle of respect for privacy.Footnote ⁸⁴ At the level of the European Union, the most well-known legal instrument in this field is the General Data Protection Regulation or GDPR,Footnote ⁸⁵ which was largely inspired by Convention 108 and on which I will focus in what follows in a very succinct way, as it establishes directly invokable rights and obligations in EU Member States.Footnote ⁸⁶

The GDPR imposes obligations on private and public entities alike, and hence also applies to all personal data processing activities of public administrations (though a separate regime exists for the processing of data by law enforcement authorities in the context of criminal investigations).Footnote ⁸⁷ Its protective provisions are especially relevant in the context of algorithmic regulation, which almost by definition implies the processing of personal data.Footnote ⁸⁸ Public administrations must, pursuant to Article 6 of the GDPR, be able to justify each data processing activity through a legal basis that confers them such power, whether this be the data subject’s consent, the protection of the vital interests of a person, or the performance of a task carried out in the public interest or in the exercise of official authority vested in the data controller.Footnote ⁸⁹ In case a public administration seeks to rely on the latter ground, the legal basis must be laid down further in Union law or in domestic law which sets out the processing’s purpose, meets an objective of public interest and is proportionate to the legitimate aim pursued.Footnote ⁹⁰

The GDPR also mandates that administrations processing personal data do so in line with several principles, including the need to process data in a lawful, fair and transparent manner; to ensure it is collected only for specified, explicit and legitimate purposes, in a way that is adequate, relevant and limited to what is necessary in relation to such purposes; to ensure the data is accurate, kept up to date, and not stored for longer than necessary; and to ensure the data is processed with appropriate security measures, including protection against unlawful processing, accidental loss or damage.Footnote ⁹¹ Finally, administrations are not only responsible to make sure these principles are respected, but they must also be able to demonstrate compliance with them to foster accountability.Footnote ⁹²

In addition to these obligations, data subjects also have certain rights regarding their personal data, including the right to information about which data is being processed and in what way, as well as the right to rectify or erase such data.Footnote ⁹³ Moreover, at any time, Article 21 of the GDPR grants data subjects a right to object to the automated processing of their personal data based on “the performance of a task carried out in the public interest or in the exercise of official authority” on grounds relating to their particular situation. Administrations must demonstrate compelling legitimate grounds for the data processing which override the interests, rights and freedoms of the data subject (or for the establishment, exercise or defense of legal claims). Pursuant to Article 22 of the GDPR, data subjects also have a right not to be subject to a decision based solely on automated processing if it produces legal effects concerning them or similarly significantly affects them – which comes down to a general prohibition on such decision-making, subject to the exceptions listed in the article.Footnote ⁹⁴

The question of how much human intervention is needed to disqualify as “solely” automated processing is still not satisfactorily answered, though the European Data Protection Board (formerly the “Article 29 Data Protection Working Party”)Footnote ⁹⁵ issued interpretative guidelines in this respect.Footnote ⁹⁶ The Guidelines for instance clarify that this provision cannot be avoided “by fabricating human involvement. For example, if someone routinely applies automatically generated profiles to individuals without any actual influence on the result, this would still be a decision based solely on automated processing.”Footnote ⁹⁷ Indeed, to qualify as human involvement, the public administration should ensure that “any oversight of the decision is meaningful, rather than just a token gesture. It should be carried out by someone who has the authority and competence to change the decision. As part of the analysis, they should consider all the relevant data.”Footnote ⁹⁸ Of course, to meaningfully exercise their right to object to automated decision-making, individuals must first be made aware of the fact that a public administration is taking an automated decision which concerns them (though administrations would have an information obligation in this respect under the GDPR).Footnote ⁹⁹

19.5.3 AI Law

While data protection law provides important safeguards to counter the risks of algorithmic regulation,Footnote ¹⁰⁰ the GDPR’s entry into force also revealed that many legal gaps still remain. From 2020 onwards, these gaps were also more explicitly recognized by policymakers in Europe,Footnote ¹⁰¹ such as the Council of Europe’s Ad Hoc Committee on AI (in its Feasibility Study on a legal framework on AI),Footnote ¹⁰² and the European Commission (in its White Paper on Artificial IntelligenceFootnote ¹⁰³ and the work of its High-Level Expert Group on Artificial IntelligenceFootnote ¹⁰⁴). The realization that existing legal rules were insufficient to protect people’s rights against AI’s risks, and that the promulgation of nonbinding AI ethics guidelines did not provide a satisfactory solution either, prompted new legislative initiatives. At the level of the Council of Europe, in Spring 2022, negotiations were launched to adopt a new international “Framework Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law,” finalized in Spring 2024.Footnote ¹⁰⁵ At the level of the EU, the Commission proposed a new regulation laying down harmonized rules on AI in Spring 2021 – referred to as “the AI Act” – which after lengthy negotiations was adopted by the European Parliament and the Council in Spring 2024 as well.Footnote ¹⁰⁶ Since the former is very succinct and abstract, and still needs to be converted into national legislation by the States who wish to sign it, I will briefly focus on the latter.

As discussed more extensively in Chapter 12 of this book, the AI Act establishes mandatory requirements for AI systems that pose risks to people’s “health, safety and fundamental rights” and introduces prohibitions for several AI practices that are deemed incompatible with EU values.Footnote ¹⁰⁷ Being a regulation rather than a directive, the AI Act has binding legal force in every EU Member State without the need for any national transposition rules. Yet a first question to ask is whether algorithmic regulation by public administration falls under the scope of the AI Act. Rather than focusing on algorithmic systems, the AI Act applies to “AI,” which it defines as a “machine-based system designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.”Footnote ¹⁰⁸ In the recitals, the EU legislator made it clear that this does not encompass “simpler traditional software systems or programming approaches” or “systems that are based on the rules defined solely by natural persons to automatically execute operations.” This means that some applications of algorithmic regulation might not be captured by the AI Act, despite their potentially harmful consequences, merely because they are considered too “traditional” – which would be an unfortunate gap.Footnote ¹⁰⁹ That said, public administrations are increasingly jumping on the machine learning hype (often without a proper assessment of whether this is also more useful for the purpose they envisage), so it can be expected that ever more applications of algorithmic regulation will fall under the AI Act’s scope.

It is, however, not enough to merely fall under the scope of the regulation to also be subjected to its restrictive provisions. Taking a risk-based approach, the AI Act sets out five categories: (1) prohibited systems; (2) high-risk systems; (3) general purpose AI models; (4) systems requiring transparency measures; and (5) low-risk systems. The most relevant categories for the purpose of this chapter are the first two, since they pertain most frequently to the public sector.

The first category contains a list of several AI practices that are considered to pose an unacceptable level of risk to fundamental rights, and that are hence prohibited. For instance, AI systems cannot be used to deduce or infer people’s race, political opinions, trade union membership, religious or philosophical beliefs, sex life or sexual orientation based on their biometric data (so-called biometric categorization).Footnote ¹¹⁰ Public and private organizations are also banned from engaging in social scoring of individuals or groups based on their social behavior or based on known, inferred or predicted personality characteristics, if this scoring leads to their unfavorable treatment in unrelated contexts, or to a disproportionate detrimental treatment.Footnote ¹¹¹ Fully automated risk assessments of natural persons to predict the risk they commit a criminal offence based solely on their profiling is also prohibited,Footnote ¹¹² as is law enforcement’s use of real time facial recognition in public places, unless one of three exceptions apply and safeguards are foreseen.Footnote ¹¹³

The second category encompasses AI systems that are considered to pose a high risk to the health, safety and fundamental rights of individuals. Either they are already covered by existing product safety legislation (listed in Annex I) or they fall under the list of stand-alone high-risk systems (listed in Annex III). Many algorithmic regulation applications used by public administrations (to the extent they fall under the regulation’s “AI” definition) are included in Annex III, such as the use of AI systems to “evaluate the eligibility of natural persons for essential public assistance benefits and services, including healthcare services, as well as to grant, reduce, revoke, or reclaim such benefits and services.”Footnote ¹¹⁴ Annex III also lists several applications used by law enforcement and by migration and border control administrations, including the use of AI to assist in asylum application decisions, to profile individuals in the detection of criminal offenses, or to serve as a polygraph.

Before being put into service, high-risk systems must undergo a conformity assessment to ensure they respect the requirements listed in Articles 9–15 of the AI Act, taking into account the systems’ “intended purpose” and the “generally acknowledged state of the art on AI.”Footnote ¹¹⁵ That said, for virtually all high-risk AI systems public administrations can carry out this assessment themselves,Footnote ¹¹⁶ meaning there is no prior licensing or approval scheme before these systems are used.Footnote ¹¹⁷ Concretely, high-risk systems must be subjected to a risk-management process (“understood as a continuous iterative process planned and run throughout the entire lifecycle of a high-risk AI system”) that allows the identification of reasonably foreseeable risks the system can pose to “health, safety or fundamental rights when the high-risk AI system is used in accordance with its intended purpose,” on the basis of which “appropriate and targeted risk management measures” must be taken.Footnote ¹¹⁸ High-risk systems are also subject to data governance obligations for the training, validation and testing of systems, which include considerations regarding the relevant design choices of the model, the formulation of assumptions with respect to the information the data are supposed to represent, an assessment of the data availability and suitability and potential gaps, as well as the examination and mitigation of possible biases.Footnote ¹¹⁹

Besides obligations that pertain to their accuracy, robustness and cybersecurity,Footnote ¹²⁰ high-risk systems must also technically allow for the automatic recording of events for recordkeeping purposesFootnote ¹²¹ and set up a technical documentation of their compliance with those requirements, which national supervisory authorities can inspect if need be.Footnote ¹²² They must additionally be designed in a way that enables them to be overseen by natural persons. Such human oversight is meant to act as a supplementary safeguard to prevent or minimize risks (all the while taking into account the risk of so-called automation bias)Footnote ¹²³ and to enable the system’s user to decide not to use the system or to reverse its output.Footnote ¹²⁴ In the context of public administrations, civil servants should hence always have the possibility to deviate from the system’s suggested decision – though, as discussed earlier, this not only depends on a legal provision, but also requires an organizational environment that enables them to do so in practice.

As to the systems’ transparency, providers of high-risk systems must present deployers of their systems with the necessary information to “interpret the system’s output and use it appropriately.”Footnote ¹²⁵ This means that civil servants who procure AI systems should in principle receive information about the system’s “characteristics, capabilities and limitations of performance.”Footnote ¹²⁶ Note, however, that this information need not be shared with those who are subjected to (and potentially negatively affected by) the system, but only to the system’s users. The only information about high-risk systems that must be made publicly available is enumerated in Annex VIII of the AI Act and must be registered in the new “EU database for high-risk systems listed in Annex III,” which the European Commission must set up as per Article 71 of the AI Act. The most useful information that AI providers must register is “a description of the intended purpose of the AI system and of the components and functions supported through this AI system,” as well as “a basic and concise description of the information used by the system (data, inputs) and its operating logic.” Whenever a high-risk system is used by a public sector deployer, the system’s use must also be registered in the database, including a summary of “the findings of the fundamental rights impact assessment” which such deployers must conduct pursuant to Article 27 of the AI Act, and a summary of the data protection impact assessment they must carry out pursuant to Article 35 of the GDPR or Article 27 LED. The new EU database might hence become a valuable source for individuals seeking more information about public administrations’ use of algorithmic regulation, especially with a view of challenging it in case there are concerns about breaches of their rights.

At the same time, there are many shortcomings in the protection the AI Act intends to afford, which are discussed in more detail in Chapter 12 of this book.Footnote ¹²⁷ Particularly in the context of algorithmic regulation, many concerns remain unaddressed. The AI Act barely mentions the rule of law and has nothing to say about the normative influence that private actors can have on the public sphere through the procurement of algorithmic regulation tools. It also does not ensure that citizens get a say about whether certain algorithmic regulation applications should be used by public administrations in the first place, and its overly restrictive scope means it does not cover many harmful applications of algorithmic regulation, especially when based on more traditional systems. The AI Act also has many carve-outs which undermine its protection: AI systems deployed for research or for national security fall outside its scope, which risks constituting a significant gap. In addition, the safeguards against the use of live facial recognition (or biometric identification more generally) in public places only apply in the context of law enforcement and do not include borders, which leaves migrants – who already find themselves in a very vulnerable position – even more vulnerable.Footnote ¹²⁸

One could also argue that, by virtue of AI Act, the use of certain problematic applications is actually legitimized, since they can now be rubberstamped by claiming conformity with the AI Act’s rules. This also led some people to criticize the AI Act an instrument of “deregulation,” by potentially undermining safeguards drawn from other legal domains.Footnote ¹²⁹ Finally, the AI Act’s list-based approach – which exhaustively lists the systems that fall within its different categories – also means that some important applications are not covered, as the lists are underinclusive.Footnote ¹³⁰ Combined with the fact that providers of high-risk systems can largely self-assess their system’s conformity with the requirements, and the fact that they can even self-assess whether their system is truly high-risk,Footnote ¹³¹ the regulation leaves a lot of leeway to the very actors against which it allegedly seeks to protect individuals.Footnote ¹³²

That said, the AI Act does provide a number of new safeguards that can be invoked to counter some risks posed by public administrations’ use of AI. Moreover, the fact that it establishes a novel public enforcement mechanism both at the national and the European level also provides a strong signal that the EU takes AI’s challenges seriously. The AI Act should hence be seen as part of a broader legal framework that also includes other protective provisions, and that is complementary to the legal domains set out above. Its relationship with existing legislation is also clarified in the AI Act itself, which for instance states that it “does not seek to affect the application of existing Union law governing the processing of personal data,”Footnote ¹³³ and that “where an AI practice infringes other Union law” it can still be prohibited regardless of its inclusion in the exhaustive list of prohibitions of Article 5.Footnote ¹³⁴ Accordingly, one should look beyond the AI Act’s provisions to hold public administrations to account when they choose to rely on algorithmic regulation. Rather, the AI Act should be invoked along with provisions of constitutional law, administrative law and data protection law (as well as other relevant legal domains) to provide stronger protection against the challenges discussed in this chapter.

20.2.1 Brief Introduction to the History of IHL

The term armed conflict, which stands central to IHL, has a distinctive meaning in international law. The most commonly referred definitionFootnote ⁸ is the one provided by the International Criminal Tribunal for the Former Yugoslavia which describes it as existing “whenever there is a resort to armed force between States or protracted armed violence between governmental authorities and organized armed groups or between such groups within a state.”Footnote ⁹ The definition itself shows the armed conflicts are typically divided into those having an international and a non-international dimension.Footnote ¹⁰ For the sake of simplifying the scope of the present chapter, it will focus on the usage of AI solely in international armed conflicts. As a matter of fact, international armed conflicts were the only subject of regulation before the Second World WarFootnote ¹¹ which once again demonstrates the reactive nature of IHL.

IHL is considered to be born in the second half of the nineteenth century after the infamous battle of Solferino on June 24, 1859.Footnote ¹² The battle resulted in the victory of the allied French Army of Napoleon III and the Piedmont-Sardinian Army commanded by Victor Emmanuel II over the Austrian Army under Emperor Franz Joseph I. It was documented that over 300,000 soldiers participated in the fifteen-hour massacre from which 6,000 died and more than 35,000 were wounded or went missing.Footnote ¹³ A detailed testimonial regarding the fallout of the battle and the suffering of both combatants and civilians was given by the Swiss national Henry Dunant, who happened to be in the area and witnessed the gruesome picture of the battlefield, later described in his book named A Memory of Solferino.Footnote ¹⁴ Henry Dunant dedicated his life and work to the creation of an impartial organization tasked with caring for the wounded in armed conflicts and providing humanitarian relief. It became a reality on February 17, 1863, when the International Committee of the Red Cross (ICRC) was established in Geneva. This event, while significant, was just the beginning of the rapid development of modern IHL. In 1864, the Geneva Convention for the Amelioration of the Condition of the Wounded in Armies in the Field was adopted, becoming the first international treaty regulating armed conflicts in a universal manner, opened for all States to join.Footnote ¹⁵ That instrument, and the following Geneva conventions, established three key obligations for the parties: (1) providing medical assistance to the wounded regardless of their nationality, (2) respecting the neutrality of the medical personnel and establishments, and (3) recognizing and respecting the sign of the Red Cross on white background.

These first steps predefine the characteristics of modern IHL as part of the body of international law that governs the relations between states. Its subsequent development also broadens its scope to protecting “persons who are not or are no longer taking part in hostilities, the sick and wounded, prisoners and civilians, and to define the rights and obligations of the parties to a conflict in the conduct of hostilities.”Footnote ¹⁶ In addition, IHL evolved to serve as a guarantee for preserving humanity even on the battlefield by attempting to ease the suffering of the combatants. This development in the role of IHL occurred early on due to the rapid uptake of new weapons. In particular, the invention of the dum-dum bulletFootnote ¹⁷ in 1863 inspired the adoption of the first international treaty concerning weaponry (namely the St. Petersburg Declaration Renouncing the Use, in Time of War, of Explosive Projectiles under 400 gm weight, in 1868).Footnote ¹⁸ It is a pivotal instrument for IHL not only because it was the first of its kind but also because it recognized the customary character of the rule according to which using arms, projectiles, and materials that cause unnecessary suffering is prohibited.Footnote ¹⁹ This line of work was continued through the Hague Conventions from 1899 and 1907, governing the “laws of war” as opposed to the Geneva Conventions that focus on the right to receive relief.

The First World War indicated the end of this period of development of IHL, bringing forward the concept of total war, new weapons (including weapons of mass destruction such as poison gas), and the anonymization of combat.Footnote ²⁰ These novel technologies and the effects they had on people, as well as the outcome of the campaigns and individual battles, naturally resulted in the adoption of further legal instruments in response of the new treats to the already established law of war.Footnote ²¹ In addition, the Geneva Convention was overhauled by the 1929 Convention for the Amelioration of the Condition of the Wounded and Sick in Armies in the Field which further reflected the influence of the new technologies establishing, for example, protection of medical aircraft.

The Second World War brought another set of challenges for IHL besides the tremendously high percentage of civil causalities compared to the First World War.Footnote ²² The concept of total war which transforms the economies of the states into war economies fogged the distinction between civilians and combatants and also between civilian and military objects, which is one of the key principles of IHL. Other contributing factors were the civilian groups targeted by the Nazi ideology and the coercive warfare used by the Allied powers.Footnote ²³

The immediate response to the horrors of the biggest war in human history was the establishment of the United Nations and the International Military Tribunals of Nuremberg and Tokyo. In addition, four new conventions amended and reinforced the IHL framework. The 1949 Geneva Conventions on the sick and wounded on land; on the wounded, sick, and shipwrecked members of the armed forces at sea; on prisoners of war; and on civilian victims (complemented by the Additional Protocols from 1977) codified and cemented the core principles of the modern-day IHL, the way we know it at present days.

The second half of the twentieth century, and in particular the Cold War period and the Decolonization, contributed mostly to the development of the IHL rules regarding non-international armed conflicts. Nevertheless, the tendency of adopting treaties in response to technological advancement continued. Certain core legal instruments on arms control were adopted during that time, such as the Treaty on the Non-Proliferation of Nuclear Weapons from 1968,Footnote ²⁴ the Biological Weapons Convention from 1972,Footnote ²⁵ and the Chemical Weapons Convention from 1993.Footnote ²⁶ Another important legal treaty, directly connected to the regulation of new technologies used as weapons, concerns the Convention on Certain Conventional Weapons (CCW) from 1980.Footnote ²⁷ This Convention will be further discussed in Section 20.3.

The global tendencies of the twenty-first century signaled the decreasing political will of nation-states to enter into binding multilateral agreements beyond trade and finance due to their lack of effectiveness.Footnote ²⁸ This is extremely worrisome not only because of its effect on the international legal order, but also due to the consequences it has on the ambition of making law anticipatory rather than reactionary, especially in the context of new weapons such as LAWS. Therefore, the principles of IHL which have already been established during the last century and a half, need to be taken into account when interpreting the existing body of law applicable vis-à-vis technologies such as AI, regardless of the capacity it is used for in military context. The next section offers an exposition of these principles which should provide the reader with a better understanding of the IHL challenges created by AI.

20.2.2 Principles of IHL

To understand the effect of AI on IHL, six core principles of IHL need to be unpacked, as they serve both as an interpretative and guiding tool for the technology’s use in this area. This section will briefly discuss each principle in turn.

20.3.1 Lethal Autonomous Weapons Systems

LAWS have been in the spotlight for several years already due to the blooming of the tech industry and in particular the huge investments and reliance on AI systems. Naturally, realizing the capabilities of AI in everyday life poses the question of how it can be used in military setting. Combining technological development with justified ethical concerns and the catchy phrase “killer robots” served as a great source of inspiration for media and entertainment, with the unfortunate result of shifting the focus of the discussion from people and their behavior to the regulation of inanimate objects.

There are a number of problematic points related to the discussion about LAWS that prevent policymakers and other relevant stakeholders to move away from speculation and concentrate instead on the real dangers and immediate issues caused by using AI in armed conflicts.

First, despite the concept of LAWS being around for quite a while, we still lack a universal definition of what constitutes a lethal autonomous weapon.Footnote ⁶⁰ A definition is extremely important because “autonomy” is a scaled feature. Therefore, the answer to the question of whether a system is truly autonomous depends on where on the curve the definition positions LAWS. A similar issue appeared during the ongoing process of creating legislation on AI in several jurisdictions,Footnote ⁶¹ such as the AI Act in the EU, discussed in Chapter 12 of this Book.

Defining LAWS has been a key task for the High Contracting Parties and Signatories of the CCW, through the Group of Governmental Experts on emerging technologies in the area of LAWS (GGE on LAWS). However, their efforts to adopt a universal definition currently remain fruitless.Footnote ⁶² Hence, the lack of a universal definition is filled by a plethora of more regional definitions serving various purposes. For example, the United States Department of Defense characterizes fully autonomous weapons systems as systems that “once activated, can select and engage targets without further intervention by a human operator.”Footnote ⁶³

The European Parliament, by comparison, adopted another definition through a resolution referring to LAWS as “weapon systems without meaningful human control over the critical functions of selecting and attacking individual targets.”Footnote ⁶⁴ The two definitions, although similar, reveal some important differences. The definition provided by the European Parliament is based on the absence of meaningful human oversight, while the US definition only speaks about control. Additionally, Directive 3000.09Footnote ⁶⁵ also defines semi-autonomous systems, highlighting the scale-based nature of autonomy, while the resolution of the European Parliament does not mention them at all. The discrepancy between just these two examples of LAWS definitions demonstrates that a number of weapons systems might fall in or out of the scope of norms that should regulate LAWS based on the criteria applied by a certain State, which leads to uncertainty.

A second problematic point in the LAWS discussion is the speculative nature of their capabilities and deployment. While the general consensus is that the deployment of killer robots is a highly undesirable prospect for both ethical and legal reasons, such as the alleged breach of the principle of humanityFootnote ⁶⁶ and compromised human dignity,Footnote ⁶⁷ those arguments are based on what we imagine LAWS to be. Military technology has always been surrounded by a very high level of confidentiality, and this is also the case when it comes to LAWS. Until recently, their nature could only be speculated on based on more general knowledge about AI advances and robotic applications outside their military application. The first officially recognized use of LAWS was established by the UN Security Council’s Panel of Experts on Libya regarding an incident from March 2020. During that incident, a Turkish STM Kargu-2 drone, referred to as a lethal autonomous weapon, “hunted down and remotely engaged” logistic convoys and retreating forces in Libya.Footnote ⁶⁸ The drone was described as being “programmed to attack targets without requiring data connectivity between the operator and the munition: in effect, a true ‘fire, forget and find’ capability.”Footnote ⁶⁹ There is little to no information, however, regarding the measure taken to ensure compatibility of the drone with the norms and principles of international law.

The UN’s recognition of the use of autonomous drones in Libya reignited the public debate on the international rules applicable to LAWS. It also strengthened the calls for a total ban on killer robots, unfortunately without the participation of countries like the USA, Russia, Turkey, and others having the necessary resources for the research and development of autonomous weapons.

Nevertheless, relevant stakeholders such as states, NGOs, companies, and even individuals with high standing in societyFootnote ⁷⁰ did take some significant steps in two directions. First, they launched campaigns promoting a total ban on LAWSFootnote ⁷¹ and second, they started developing and applying rules that govern the existing LAWS in accordance with IHL.Footnote ⁷²

The first initiative, even though remaining popular in civil society, does not currently show any significant development on a global scale.Footnote ⁷³ The second one has a better success rate, being built around the application of the CCW, which was deemed to be the most suitable instrument to attempt to regulate LAWS. It is so because of its purpose to ban or restrict the use of specific types of weapons that are considered to cause unnecessary or unjustifiable suffering to combatants or to affect civilians indiscriminately. In addition, the structure of the Convention allows flexibility in dealing with new types of weapons, such as for example blinding laser weapons.Footnote ⁷⁴

Therefore, this particular forum was considered as best placed to discuss the technological development of LAWS and the legal and customary rules applicable to them,Footnote ⁷⁵ as well as to codify the results of this discussion. The GGE on LAWS adopted the eleven guiding principles in the 2019 Meeting of the High Contracting Parties to the CCW.Footnote ⁷⁶ The eleven guiding principles first and foremost address the need for LAWS to comply with IHL (guiding principle 1), as well as the human responsibility for the use of LAWS (guiding principle 2). This was important not only because of the ongoing trend of personification of machinesFootnote ⁷⁷ but also because of the existing debate regarding the so-called “responsibility gap” created by LAWS.Footnote ⁷⁸ This is further supported by guiding principle 7, which forbids anthropomorphizing LAWS. The guiding principles also elaborate on the need for human–machine interaction, accountability mechanisms, and a sound balance between military necessity and humanitarian considerations.

Despite the guiding principles showing some progress and common ground among States Parties to the CCW on the subject of LAWS, they are not a binding legal instrument and are not a lot to show after eight years of work on the side of the GGE. Indeed, the formulated principles could become useful by demonstrating convergence on elements of customary international law such as opinion jurisFootnote ⁷⁹ or serving as a starting point for a new protocol to the CCW. Currently, however, neither of those appear to be on the agenda.

20.3.2 Other Applications of Autonomous Systems in Armed Conflicts

As discussed previously, the precise extent of the use and development of LAWS remain mostly confined to speculations. While weapons with a certain degree of autonomy such as unmanned aerial vehicles are currently used in many armed conflicts, truly autonomous weapons remain a rarity. However, outside the context of LAWS, the more general use of AI systems in armed conflicts and for military purposes is becoming the new normal.

AI remains a great tool for supporting decision-making in the military, assisting personnel with going through large quantities of data and analyzing it, and making the “right” judgment regarding transport, logistics, communications, and others.

Furthermore, AI has many applications in data gathering, including as a surveillance tool, although such AI products may fall under the category of dual-use items.Footnote ⁸⁰ Other possible utilizations of AI systems are applications for predictive maintenance of military equipment,Footnote ⁸¹ unarmed vehicles such as ambulances, supply trucks or drones,Footnote ⁸² and medical aid.Footnote ⁸³ While for some of these purposes of IHL might still be a consideration, such as for example the special regime of medical vehicles, other uses might raise more fundamental concerns when aspiring conformity with human rights law. A typical example is dual-use technologies used for surveillance of individuals, which violate their right to privacy and often other rights too, without a plausible justification based on military necessity.Footnote ⁸⁴

Finally, AI is increasingly used in cyberwarfare, both for its attack and defense capabilities.Footnote ⁸⁵ Typically, this involves the spreading of viruses and malware, data theft, distributed denial-of-service attacks, but also disinformation campaigns.Footnote ⁸⁶ AI systems can enhance these activities, as well as help to circumvent defenses on the way. AI is, however, not only used for defense purposes beyond military infrastructure but also to protect civilian objects and infrastructure that is more susceptible to cyberattacks due to being part of IoT.Footnote ⁸⁷

While cyberwarfare has not been explicitly mentioned by IHL instruments, this does not mean that it falls outside its scope. On the contrary, the potential (and often the intent) of cyberattacks to be indiscriminate is a considerable reason for applying IHL to cyberwarfare in full force, especially when AI systems are involved given their ability to further increase the scale of the attack.