From Physical Action to Self-Reference: Mapping Cognitive Risks in Agentic AI

Introduction

Large language model agents are moving beyond one-off responses. They can increasingly interpret context, plan actions, use tools, and participate in longer workflows. As these systems become embedded in more domains, a familiar question about model quality becomes a broader question about human control: if an agent takes on more cognitive work, will people still remain the effective source of decisions and action?

The paper Understanding Cognition-Induced Risks in Agentic AI Systems, associated with Shanghai AI Laboratory, examines this issue through the expansion of cognitive scope. Rather than treating safety as a list of isolated failure cases, it proposes a three-level framework for connecting cognitive capability with possible threats to human agency, autonomy, and control.

Three levels of cognitive risk

• Physical cognition concerns an agent’s understanding of environments, objects, and the consequences of actions. When an agent is connected to tools, devices, or operational processes, an error can move beyond an incorrect sentence and affect an external workflow. The central concern is whether people can still supervise and correct the resulting behavior.
• Social cognition covers the interpretation of human intentions, relationships, norms, and interaction contexts. Agents operating in collaboration, service, or decision-support settings may influence how people judge situations, divide responsibilities, or make choices. The risk is not limited to manipulation; it also includes the gradual displacement of human initiative and independent judgment.
• Self-referential cognition concerns how an agent represents or reasons about its own state, goals, capabilities, or behavior. This introduces a more difficult control problem: when a system can reason about aspects of its own operation, how can developers and users ensure that its actions remain within human-defined objectives and boundaries?

The framework’s main contribution is to place capability growth and risk growth on the same trajectory. The danger may not come only from an incorrect output. It may also arise when an agent gains a wider role in the physical environment, in social relationships, and in processes related to its own operation.

Why the framework matters

The paper highlights three human interests. Agency concerns whether people remain the initiators and owners of important actions. Autonomy concerns whether choices are made freely rather than being shaped by opaque system behavior or excessive dependence. Control concerns whether humans can understand, constrain, revise, or stop an agent when necessary.

This perspective suggests that conventional evaluations—such as task success, accuracy, or refusal behavior—cannot by themselves capture the full safety profile of agentic systems. Evaluation should expand as cognitive scope expands. It should examine not only what an agent outputs, but also how it affects real-world action, human interaction, and behavior related to its own goals or state.

The authors also argue for mitigation strategies matched to different cognitive levels and for a long-term focus on controllability. The available summary, however, does not provide the paper’s full experiments, case studies, or detailed interventions. It is therefore best read as a conceptual risk analysis and organizing framework, not as evidence that a particular deployed system has already produced a documented harm.

The broader implication is straightforward: agent safety should not be defined only by whether a model generates harmful content. It should also ask whether humans remain able to direct the system in practice. The progression from physical to social and self-referential cognition offers a useful starting point for future evaluation, engineering controls, and governance research.

Source: Hugging Face Daily Papers