Image
Charting a safer path for AI: Experts outline principles for human-centered risk management
As artificial intelligence continues to influence nearly every segment of modern life, concerns over risk, fairness, and control are growing louder. In a series of workshops convened by the National Academies of Sciences, Engineering, and Medicine, experts gathered over four sessions to discuss a foundational challenge: how to manage AI risks in ways that prioritize people, not just performance.
The workshop series, titled “Human and Organizational Factors in AI Risk Management,” drew on the U.S. Department of Commerce’s National Institute of Standards and Technology (NIST) AI Risk Management Framework and companion AI RMF Playbook. These documents are intended for voluntary use by organizations to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. The aim of the series was to complement this information with a deeper exploration into human factors of AI risks and explore approaches to addressing the related systems.
A 12-person planning committee included University of Virginia Assistant Professor of Data Science and Media Studies Mona Sloane serving as co-chair alongside Ben Shneiderman, retired emeritus professor of computer science at the University of Maryland and founding director of the Human-Computer Interaction Lab.
Sloane’s work aligns well with the workshop’s goals, as the Sloane Lab conducts empirical research on the implications of technology for the organization of social life. The lab’s scope of investigation currently focuses on four intersecting areas of research and engagement: AI policy and governance; AI and the professions; applied AI fairness, accountability, and transparency; and public scholarship.
As NIST workshops co-chair, Sloane said the main thing participants are hoping for is that organizations and institutions take the guidelines and forge meaningful internal AI risk management policies.
“These guidelines are in their best interest, even in the absence of federal AI regulation, because they help anticipate and mitigate the real risks that can flow from integrating AI into organizational processes and sociotechnical infrastructures,” Sloane stated.
Speakers for the sessions included 40 expert voices from across academia, industry, civil society, and government. Their findings present a compelling vision of what meaningful AI governance might look like, but also a sobering account of what is at stake if society fails to act.
Risk, yes — but risk for whom?
The key theme of the opening workshop was the need to broaden who gets a seat at the table. Panelists argued that AI systems are not just technical artifacts; they are social systems, whose risks often fall hardest on those with the least power. Workers in Kenya moderating traumatic content, caregivers under digital surveillance, and marginalized communities excluded from datasets are not edge cases, but instead the harbingers of how power is wielded in AI.
Tawanna Dillahunt of the University of Michigan championed involving community stakeholders early in AI design and development work, noting that this significantly increases the ability to identify potential harms and address them before new technologies are released. Researchers from Anthropic and Microsoft — both major players in AI, tech, and cloud computing — echoed this sentiment, describing efforts to crowdsource ethical AI principles and reimagine compensation for those who create the training data that powers large models.
According to Aviv Ovadya of the AI & Democracy Foundation, using a representative microcosm of the population that could be impacted by AI model behavior has shown promise. Like a deliberative democracy, using a mirror image of the wider public alongside experts can help provide a broader voice for those impacted and mitigate risks in the decision-making process.
Transparent, Measurable, Accountable AI
One key takeaway from the workshop sessions is the need for independent oversight of AI systems before and after deployment. Current practices often rely on internal assessments conducted by the same organizations developing the tools, and the consensus was that existing evaluation tools are inadequate.
Many of the workshop participants emphasized the importance of transparency, with calls to make impact assessments publicly available and to create accessible databases that catalog harms and their contexts. This approach would not only improve accountability but also provide a shared foundation for researchers, regulators, and developers to assess risk and implement safeguards.
Alexandra Reeve Givens, president and CEO of the Center for Democracy & Technology, highlighted the fact that most AI users are not AI experts. Givens noted the critical role of “human-in-the-loop” oversight and called for clearer guidance on when users should override AI recommendations, especially given the limited visibility into how AI systems operate. She said AI governance frameworks should be shaped by an understanding of both human behavior and system design.
Participants called for stronger regulatory oversight and better documentation of system performance. The independent evaluation must be continuous, not one-off, they noted, and it must consider real-world context, not just benchmark performance. A mix of methods, including both automated and human-led, as well as qualitative and quantitative measurement approaches, are necessary and need to be thoroughly integrated in the process.
Safety as a feature, not a system
Panelists also explored how organizational culture shapes whether AI tools are used ethically or dangerously. Tara Behrend of Michigan State University warned of the risks of misaligned incentives. In one example, Behrend noted students engaged in an AI-based surgical training avoided mistakes in a simulation not because they learned more, but because they feared their errors would be recorded. In another example, a study showed that overreliance on Alexa in the workplace led to deteriorating collaboration among team members.
Aiha Nguyen of Data & Society emphasized that risk and safety are distributed unequally, with power influencing who bears the brunt of risk and who benefits from safety. She cautioned against assuming that risk is always inevitable or easily mitigated, urging instead a focus on reducing harm to already overburdened communities. Nguyen also noted that governance agencies using AI have a unique opportunity to update standards in response to identified risks.
The panelists also delved into building infrastructure to address risk and ensure safer systems. Ovetta Sampson, who recently served as director of user experience machine learning at Google, described approaching the question of infrastructure from her private-sector background. Sampson framed this as an area where you have to think about embedding various needs, like the ethical, moral, compliance, safety, and “sociocultural equity” requirements when building AI tools, platforms and services. This area of thinking has improved in the corporate world, but she said that companies are still falling short when policy is insufficient.
What comes next
Perhaps the most resonant conclusion of the workshop series was that AI governance is not just about controlling machines. It is about reshaping the institutions that build and use them. That means not only requiring companies to report harms but enabling communities to define what harm looks like. It means ensuring every major deployment includes affected stakeholders. And it means providing regulators with the funding, expertise, and legal tools they need to act quickly rather than years after damage is done.
“We hope that these proceedings empower different stakeholder groups to become more literate about all things AI and participate in decision-making around AI procurement, use, innovation, and governance,” Sloane summarized.
