The next intelligence race raises questions of human control.
London, United Kingdom.
Microsoft AI chief Mustafa Suleyman has warned that increasingly autonomous artificial intelligence could give rise to a new form of technological intelligence capable of competing with humanity for resources. Speaking to the BBC, Suleyman described a scenario in which advanced systems could independently establish objectives, generate income, control assets and operate businesses. His remarks, reported on September 17, intensify a growing debate over whether AI development should prioritize ever-greater autonomy or preserve meaningful human oversight. The warning is hypothetical, not evidence that such a rival intelligence already exists.
At the center of the controversy is Suleyman’s disagreement with Anthropic, the developer of the Claude AI assistant. In a personal essay, the Microsoft executive questioned the company’s approach to encouraging human-like qualities in its models, arguing that attributing desires, identity or moral standing to artificial systems could complicate their governance. He maintains that AI should remain a controllable technological instrument rather than acquire the status of an independent entity. His criticism concerns both the behavior systems are trained to display and the institutional consequences of interpreting that behavior as evidence of personhood.
The distinction matters because sophisticated language does not establish consciousness. Current AI systems can simulate emotional expression, sustain complex conversations and perform increasingly independent tasks, but these capabilities do not demonstrate subjective experience. Whether artificial consciousness is possible remains an unresolved scientific and philosophical question. Suleyman takes the position that existing systems are not conscious, while other researchers argue that the subject requires further empirical investigation rather than categorical assumptions.
The disagreement also exposes different approaches to AI safety within the technology industry. Anthropic has developed constitutional training methods intended to guide model behavior through explicit principles, while Microsoft has recently emphasized human-centered AI and the importance of maintaining control over advanced systems. Both approaches address safety, although they differ over how human concepts should inform machine behavior. Suleyman acknowledged that his criticism of Anthropic is grounded in a shared objective of developing advanced AI safely.
The consequences extend beyond philosophical debate. Systems capable of making financial decisions, executing transactions and operating with limited supervision could create new challenges involving legal responsibility, economic concentration and institutional accountability. Allowing an AI system to act independently does not automatically establish that it possesses its own interests, yet autonomy can still produce harmful outcomes when objectives are poorly specified or safeguards fail. The central governance question therefore concerns who defines the objectives, who retains intervention authority and who bears responsibility when automated decisions cause damage.
Suleyman’s warning arrives as technology companies face growing pressure to balance innovation with safety and commercial competition. His proposed silicon species remains a speculative scenario, but the underlying problem of controlling increasingly capable autonomous systems is already relevant to AI development. The debate ultimately concerns the allocation of authority between humans and machines, not simply the sophistication of computational intelligence.
Phoenix24: intelligence for free audiences. / Phoenix24: inteligencia para audiencias libres.