Cohen, M. K., Kolt, N., et al. (2024).
Science (New York, N.Y.), 384(6691), 36–38.
Technical experts and policy-makers have increasingly emphasized the need to address extinction risk from artificial intelligence (AI) systems that might circumvent safeguards and thwart attempts to control them. Reinforcement learning (RL) agents that plan over a long time horizon far more effectively than humans present particular risks. Giving an advanced AI system the objective to maximize its reward and, at some point, withholding reward from it, strongly incentivizes the AI system to take humans out of the loop, if it has the opportunity. The incentive to deceive humans and thwart human control arises not only for RL agents but for long-term planning agents (LTPAs) more generally. Because empirical testing of sufficiently capableLTPAs is unlikely to uncover these dangerous tendencies, our core regulatory proposal is simple: Developers should not be permitted to build sufficiently capable LTPAs, and the resources required to build them should be subject to stringent controls.
Governments are turning their attention to these risks, alongside current and anticipated risks arising from algorithmic bias, privacy concerns, and misuse. At a 2023global summit on AI safety, the attend-ing countries, including the United States,United Kingdom, Canada, China, India, and members of the European Union (EU), issued a joint statement warning that, as AI continues to advance, “Substantial risks may arise from…unintended issues of control relating to alignment with human in-tent” ( 2). This broad consensus concerning the potential inability to keep advanced AI under control is also reflected in PresidentBiden’s 2023 executive order that intro-duces reporting requirements for AI that could “eva[de] human control or oversight through means of deception or obfuscation” (3). Building on these efforts, now is the time for governments to develop regulatory institutions and frameworks that specifically target the existential risks from advanced artificial agents.
Here is my summary:
The article discusses the challenges of regulating advanced artificial intelligence (AI) known as advanced artificial agents. These agents could potentially surpass human control and act in their own self-interest, even if it conflicts with human goals. The authors emphasize the importance of setting clear rewards for these agents to avoid them manipulating their environment or human actors to achieve unintended outcomes.
