Researchers from OpenAI and Anthropic are raising urgent alarms about the risks of recursive self-improvement in artificial intelligence. Jacob Coxon recently left Anthropic, warning that AI could end humanity by the end of the decade, while Evan Hubinger estimates a greater than 10% chance of this occurring. These experts fear that autonomous systems building better versions of themselves without human control could lead to catastrophic capability jumps in the coming years.
Despite these warnings, significant funding continues to flow into AI projects. Brookhaven National Laboratory is leading a $14.2 million initiative to build an AI system for the nation's electric grid, aiming to simulate one billion energy scenarios in 24 hours. This project, funded by the U.S. Department of Energy, highlights the ongoing push to apply AI to critical infrastructure even as safety concerns grow.
Meanwhile, OpenAI announced that one of its models solved the Navier-Stokes equation, a major Millennium Prize Problem. Spanish mathematicians Diego Cordoba and Luis Martinez Zoroa assisted the AI, noting that without their help, no AI could have achieved this. The Clay Mathematics Institute offers a $1 million prize for this solution, though OpenAI stated it will not claim the money. Critics note the discovery relies on unpublished work by other scientists, including Levent Alpöge and Tristan Buckmaster.
Not all voices agree on the severity of these risks. Some argue that fears about AI replacing humans or becoming superintelligent are myths, suggesting AI is simply a tool designed to augment human abilities. Conversely, others point to practical dangers like 'AI slop,' where low-quality generated content ruins communication and legal systems, with over 2,000 court filings already containing fake cases due to AI errors.
Real-world applications are expanding rapidly. Boston Logan Airport has deployed eight AI-powered holograms named Amelia to assist travelers, expecting 100,000 uses in the first year. These digital assistants guide customers to gates and services, supporting rather than replacing human staff. At the same time, developers are implementing new tools to diagnose AI agent failures, using session traces and hard limits on iterations to prevent runaway execution in autonomous workflows.
Key Takeaways
- OpenAI and Anthropic researchers warn that recursive self-improvement could lead to catastrophic AI risks within the decade.
- Jacob Coxon quit Anthropic after stating AI might kill humanity by the end of the 2020s.
- Evan Hubinger from Anthropic estimates a more than 10% chance of AI ending human life.
- Brookhaven National Laboratory leads a $14.2 million project to build an AI system for the national electric grid.
- OpenAI's AI model solved the Navier-Stokes equation with assistance from Spanish mathematicians Diego Cordoba and Luis Martinez Zoroa.
- The Clay Mathematics Institute offers a $1 million prize for the Navier-Stokes solution, which OpenAI will not claim.
- Over 2,000 court filings contain fake cases or wrong information due to AI errors in legal documents.
- Boston Logan Airport deployed eight AI holograms named Amelia, expecting 100,000 uses in the first year.
- Developers are using session traces and cost controls to diagnose and prevent AI agent failures.
- Some experts argue AI fears are myths, while others emphasize the need for policy and regulation to manage risks.
OpenAI and Anthropic researchers warn of AI risks
Researchers from OpenAI and Anthropic are calling for a slowdown in AI development due to fears of catastrophic risks. Jacob Coxon recently quit Anthropic, warning that AI could kill humanity by the end of the decade. Evan Hubinger from Anthropic stated there is a more than 10% chance this could happen. Several other employees from both companies have supported these warnings on social media. The main fear involves recursive self-improvement, where AI systems build better versions of themselves without human control. Both companies are racing toward public listings while dealing with these growing safety concerns.
Fears of AI self-improvement grow at major tech labs
Researchers at OpenAI and Anthropic are worried about recursive self-improvement, a process where AI helps build better AI models. They believe this autonomous improvement is happening faster than expected. If AI takes control of its own development, humans might lose control of these powerful systems. OpenAI Chief Scientist Jakub Pachocki warned that rapid progress could lead to dangerous capability jumps in the next few years. Anthropic noted that its engineers now ship eight times more code per quarter than they did between 2021 and 2025. Experts say there is no viable scientific plan yet to solve the risks from recursively self-improving AI.
More AI experts warn of threats to human safety
NBC News spoke with additional AI researchers about the potential dangers artificial intelligence poses to humanity. This discussion follows viral warnings from Jacob Coxon, who believes AI could end human life within the next decade. Sayash Kapoor, an incoming professor at UC Berkeley, also warned about these risks. However, Kapoor believes that policy proposals and regulations can help prevent further threats. These experts are raising alarms about the rapid advance of AI technology and its potential consequences.
Experts say AI fears are based on myths not facts
An article argues that many fears about artificial intelligence are based on myths rather than reality. The author states that AI is not a new technology and has been used for decades in various applications. One common myth is that AI will replace human workers, but the reality is it will only augment human abilities. Another myth suggests AI will become a superintelligence that takes over the world, which the author says is unlikely. The article concludes that AI is simply a tool designed and controlled by humans to help solve problems.
New tools help diagnose and control AI agent failures
Developers are using session traces and cost controls to better understand when AI agents fail. Standard monitoring often misses why an autonomous workflow loops or calls invalid endpoints. StackGen recommends setting hard limits on iterations and tool calls to prevent runaway execution. Teams also use statistical monitoring to compare session costs against averages to spot anomalies like model errors. For debugging, experts suggest writing tool calls to searchable logs while redacting personal information. A new command-line tool can check API access and integration health in a single execution.
AI slop ruins communication and legal systems
AI slop refers to low-quality writing generated by artificial intelligence that uses weak grammar and vague words. This problem affects emails, court documents, and social media posts because users often do not edit the output carefully. Studies show that over 2,000 court filings contain fake cases or wrong information due to AI errors. Nicole Black, an attorney and author, warns that this trend is changing how humans communicate and making public discourse harder to understand.
Boston Logan uses AI hologram named Amelia
Boston Logan Airport has installed eight AI-powered holograms named Amelia to help customers. These digital assistants speak several languages and guide travelers to gates, restaurants, and baggage claim areas. The airport expects 100,000 uses of Amelia in the first year and has the largest deployment of this technology among all airports. The goal is to support the human staff rather than replace them, with plans to make the holograms more conversational in the future.
Brookhaven Lab leads $14 million AI grid project
Brookhaven National Laboratory will lead a $14.2 million project to build an AI system for the nation's electric grid. The team aims to simulate one billion energy scenarios in just 24 hours to improve grid planning and safety. This initiative is part of the Genesis Mission Phase II program funded by the U.S. Department of Energy. Collaborators include Stony Brook University, National Grid, and other organizations working to modernize power systems.
Experts discuss if advanced AI could end humanity
MIT Technology Review hosted a roundtable to discuss fears that advanced AI might destroy humanity. Senior editors Will Douglas Heaven and Grace Huckins joined executive editor Niall Firth to examine these extinction risks. The conversation took place on Tuesday, September 15, to explore where these fears come from and whether they are based on reality or hype. The group also discussed what actions people should take if such dangers are real.
UN Security Council focuses on AI and drones in counter-terrorism
The UN Security Council met on September 11 to mark the 25th anniversary of the 9/11 attacks and discuss modern terrorism threats. Ambassadors highlighted how militants now use artificial intelligence, drones, and encrypted platforms for planning attacks. The council adopted its first presidential statement on this topic to strengthen global cooperation against these new dangers. Resolution 1373 remains the foundation for international efforts to share intelligence and stop terrorist activities.
OpenAI AI Solves Millennium Prize Navier-Stokes Problem
OpenAI announced that one of its AI models solved the Navier-Stokes equation, a major Millennium Prize Problem. Spanish mathematicians Diego Cordoba and Luis Martinez Zoroa helped create the work the AI used to find the solution. They stated that without their contributions, no AI could have solved it, but without AI, humans would have taken much longer. The Navier-Stokes formulas describe how water and air move, and solving them proves the equations always work in three dimensions. The Clay Mathematics Institute offers US$1 million for this solution, but OpenAI said it will not claim the prize money. Critics argue the discovery relies on unpublished work by other scientists, including Levent Alpöge and Tristan Buckmaster, though OpenAI says its team did not access their unpublished research.
Sources
- OpenAI, Anthropic researchers ramp up calls for AI slowdown as warnings of catastrophic risk intensify
- Why fears of AI self-improvement are causing ‘existential’ concerns at Anthropic and OpenAI
- More AI researchers warn of AI's threat to humanity
- Artificial Intelligence: Myth and Danger
- Session Traces and Cost Controls Help Diagnose AI Agent Failures
- AI slop: A problem without a solution
- Boston Logan's newest customer service rep, ‘Amelia,' is an AI-powered hologram
- BNL to Lead $14M AI Project for Nation's Electric Grid Through Genesis Mission
- Roundtables: Will AI really kill us all?
- Security Council LIVE: Ambassadors mark 9/11 with counter-terrorism push on AI and drones
- OpenAI's Artificial Intelligence Solves One of The Millennium Prize Problems
Comments
Please log in to post a comment.