Imagine the following scenario (Schlicht 2025)

This scenario is reproduced from the main text of the author’s dissertation.

The situation described above is fictional but illustrates plausible occupational and ethical risks that may arise when AI tools are implemented to support healthcare personnel without having been designed in a manner that adequately considers human-centered design (HCD) criteria. Indeed, HCD criteria are currently not applied systematically throughout the development process of AI systems, raising concerns that the use of such technologies could weaken the relational and moral dimensions of healthcare by, for example, encouraging a shift from patient-centered care practices toward standardized interactions.

Healthcare has become a key sector for the advancement and integration of artificial intelligence.

Healthcare has become a key sector for the advancement and integration of AI (OECD 2019). AI-driven tools, including diagnostic, monitoring, and workflow optimization systems, are increasingly being integrated into clinical practice. Their implementation is intended to assist healthcare professionals (e.g., physicians, psychotherapists, physiotherapists, nurses) by automating routine tasks, enhancing diagnostic accuracy, streamlining treatment procedures, and enabling more personalized approaches to patient care (Sharma et al. 2022; Yakusheva et al. 2025). At the same time, they may potentially also undermine the holistic approach to care that is fundamental to ethically responsible healthcare. While the healthcare sector is generally regarded as less amenable to automation, given that it is highly dependent on contextual information, AI integration may nonetheless impact a wide range of clinical practices, including those integral to patient-centered care (Yelne et al. 2023).

Against this backdrop, this article addresses the following research requestion: What strategies and instruments can facilitate the sustainable integration of HCD criteria into the development of AI systems in healthcare? To answer this, the article first outlines key challenges in integrating HCD criteria into AI development processes. It then introduces participatory technology assessment (pTA) as a methodological approach capable of addressing these challenges. Finally, the article presents the methodology behind and findings from two multi-stakeholder dialogues, identifying concrete pathways toward human-centered AI development in healthcare.

While there is a growing consensus regarding the importance of considering occupational and ethical criteria during the design of AI systems, the practical incorporation of these criteria remains sporadic and limited (Tidjon and Khomh 2022; Vakkuri et al. 2020). Numerous frameworks, including international guidelines (Corrêa et al. 2023) as well as regulatory instruments (e.g., the EU Artificial Intelligence Act (Regulation (EU) 2024/1689)) and standards (e.g., ISO/IEC 42001:2023, ISO/IEC 38507:2022 or IEEE 7000:2021), underscore the importance of incorporating HCD criteria into AI system design. However, it remains largely uncertain what specific approaches could support the effective consideration of such non-technical design requirements – especially in complex sociotechnical domains like healthcare (WHO 2023).

Prominently, a central difficulty here lies in the typically generalized formulation of HCD criteria, which limits their applicability in concrete design and risk assessment processes (Sanderson et al. 2023). Another challenge stems from the adaptive nature of many AI systems, further complicating the criteria’s translation into practical technology development processes. As illustrated in the hypothetical scenario presented above, AI algorithms may evolve in response to dynamic data environments and user interactions, making it difficult – if not impossible – to fully anticipate the risks associated with their deployment during the technology design. However, traditional risks management methodologies typically lack iterative mechanisms for the continuous assessment and mitigation of emerging risks, i.e., they do not systematically incorporate observations of threats and hazards encountered in deployment contexts back into ongoing design and refinement processes (Siedel et al. 2021). As a result, within the scope of conventional methodologies, HCD criteria risk being rendered ineffective by the dynamic and context-specific challenges posed by adaptive AI systems.

In response to these limitations, the AI Act stipulates mandatory ongoing risk management processes to identify and minimize emerging risks – at least for AI systems classified as high-risk, such as patient triage systems and emotion-recognition technologies (Recital 65, Article 9, EU 2024).

In parallel, several standardization organizations have developed technical frameworks aimed at promoting a lifecycle-oriented approach to AI risk management (e.g., DIN SPEC 92001 series; IEEE 7000:2021; ISO/IEC 23894:2023; ISO/IEC 38507:2022; ISO/IEC 42001:2023).

The integration of HCD criteria, such as respect for autonomy or work-integrated learning (Beauchamp and Childress 2019; Ulich 2011), into verifiable measures that can be implemented and assessed throughout the development and implementation of AI systems faces distinct challenges. While existing risk assessment and mitigation strategies for technology development largely rely on quantifiable parameters, HCD criteria are difficult to express quantitatively (Poszler et al. 2024). Accordingly, there are to date only limited standardized procedures for evaluating the extent to which ‘soft’ criteria are considered to an effective degree during (re)design processes. Moreover, the consideration of specifically ethical criteria requires risk management procedures capable of considering the specific needs of individuals affected by AI systems (Mittelstadt 2019). Against this backdrop, it is clear that the integration of HCD criteria into verifiable measures cannot rely solely on technical expertise; properly considering these criteria requires multi-disciplinary expert knowledge from various disciplines, including human-computer interaction, occupational psychology, and applied technology ethics (Schmager et al. 2025).

Participatory technology assessment offers a particularly suitable framework for informing the development of approaches that consider human-centered design criteria.

PTA offers a particularly suitable framework for informing the development of approaches that consider HCD criteria – as well as their inherent socio-technical complexities – to an effective degree in technology (re-)development processes. By engaging experts across multiple disciplines as well as direct stakeholders, pTA enables the use of contextual insights to enhance the accuracy of risk assessment and mitigation efforts (Grunwald 2018; Grobe 2021). Importantly, the pTA format employed in the multi-stakeholder dialogues was designed to create a reflective space in which diverse perspectives could be articulated, negotiated, and constructively engaged, thereby fostering co-learning, knowledge integration, and the joint development of various actionable strategies and instruments.

To identify strategies for systematically assessing and mitigating the occupational and ethical risk presented by AI systems in the healthcare sector, we conducted two multi-stakeholder dialogues in March and May 2024.

A two-day format was employed for each dialogue to facilitate comprehensive exchange and iterative reflection.

Participants were selected through a targeted recruitment process to ensure the representation of a diverse array of stakeholder groups, disciplines, and institutional affiliations, supported by snowball sampling through existing professional networks. The final group of 25 participants included, among others, researchers (in the fields of applied machine learning, digital health engineering, occupational psychology, technology ethics, nursing science, technology assessment, and technology governance), policy officers from ministries and regulatory agencies, representatives of both employee and employer organizations, specialists from statutory insurances, and managers from companies in the field of nursing software and digital healthcare consulting (see Table 1).

The first dialogue was structured around the question: “At which phases of the AI lifecycle should action be taken to ensure the human-centered design of AI systems in healthcare?”. Following a brief introduction of the project’s objectives, participants collaboratively mapped existing and emerging AI applications in healthcare, reflecting on their technical functionalities, anticipated implications for work processes, and ethical dynamics. Participants were subsequently introduced to a set of HCD criteria based on (i) established criteria for human-centered work design aimed at promoting personality development, health maintenance, and employee performance (e.g., ISO 2016, 2019, 2024), and (ii) the principles of biomedical ethics by Beauchamp and Childress (2019), a widely recognized framework for ethical assessment within healthcare domains. The latter criteria also constitute the normative foundation of the High-Level Expert Group on AI’s “Ethics Guidelines for Trustworthy AI” (AI HLEG 2019), a central reference point for ethical AI design under the AI Act. Using selected examples of AI systems, participants explored how these criteria can be sustainably applied across the different stages of the AI lifecycle (i.e., from the specification of requirements to the operational monitoring). In this context, the goal was not to develop isolated strategies for each criterion but to identify approaches that enable the general integration of HCD criteria into verifiable measures.

In small-group follow-up sessions, the initial insights were deepened and specified using structured digital collaboration formats. The second dialogue, building directly on the results of the first session, was guided by the following question “What are particularly promising strategies and how can they be implemented?”. Drawing on key intervention points identified via finger voting, participants developed both individual and collective strategies for integrating HCD criteria into occupational safety and health assessments and broader AI governance mechanisms. Through iterative reflection formats, such as world cafés and listening circles, they then focused on implementation planning, clarifying necessary actions and responsible actors. In line with the participatory orientation of the overall approach, the design of these sessions deliberately avoided predefined outcome structures. Instead, the process was participant-driven, allowing the content and form of the results to emerge collaboratively. This openness also fostered joint ownership of the outcomes and encouraged participants to initiate follow-up activities within their own organizational and professional networks. All outcomes were recorded using Metaplan boards.

Participants developed a broad range of proposals for means of advancing human-centered AI risk management processes in healthcare settings. A central recommendation was the establishment of iterative feedback loops between AI system providers and developers – including for systems not classified as high-risk. The participants emphasized that such iterative risk management processes are closely linked to the twofold requirement to enhance transparency in system design and functioning and to ensure that system characteristics are communicated in an accessible manner for a wide range of stakeholders. On a similar note, special attention was paid to the post-deployment phase, where continuous monitoring and adaptive feedback mechanisms were considered to be essential to address emerging risks. The inclusion of both user and affected stakeholder perspectives was recognized as particularly important in this regard. As part of ongoing collaboration, the participants eventually also began to develop concrete instruments and other intervention points. The key insights – along with the associated follow-up activities – are summarized in Table 2.

The results of these multi-stakeholder dialogues underscore the value offered by participatory frameworks like pTA in developing effective strategies for human-centered AI design in healthcare. While involving diverse stakeholders is a complex and resource-intensive task, it proved to be highly beneficial in our workshops. The discourse among technical, social science, regulatory, and practice-oriented perspectives facilitated the development of various integrative pathways through which to embed work-related and ethical considerations into AI design – all in a manner tailored to the specific requirements of the healthcare sector. Through the follow-up activities, the participants began to collaboratively develop solutions, including measures to improve system transparency and comprehensibility, iterative prototyping methodologies, and checklists for psychological risk assessments. Moreover, further exchange formats were initiated, including with potential users of AI systems in healthcare, to promote ongoing dialogue and knowledge sharing.

… future activities may place greater emphasis on involving stakeholders directly engaged as healthcare professionals.

However, the developed tools have not yet been empirically tested in real-world development and deployment scenarios. Their practical viability and overall effectiveness therefore remain uncertain, particularly with regard to the organizational, technical, and ethical challenges that may arise in applied settings. Relatedly, although most participants brought expertise relevant to the healthcare sector, future activities may place greater emphasis on involving stakeholders directly engaged as healthcare professionals. Their inclusion would help to ensure that perspectives from clinical practice are more fully represented and that proposed measures are grounded in the realities of healthcare delivery.

Furthermore, future initiatives should investigate how context-sensitive, participatory frameworks for risk management can be scaled and institutionalized more broadly, including with a view to the national rollout of the EU AI Act. Ultimately, establishing such locally grounded risk management mechanisms will become increasingly critical as multi-agent AI systems (Moritz et al. 2025) are more frequently deployed in administrative processes as well as in clinical decision making. These systems are characterized by the interaction of multiple (semi-)autonomous AI agents, often incorporating large language models (e.g., GPT‑4, Gemini). Since such systems have a heightened potential to adapt to changing environments, there is an even growing need for iterative governance mechanisms – ideally developed in close collaboration with relevant stakeholders – that can flexibly address emerging occupational and ethical risks across diverse application contexts in the healthcare sector.