Zotpaper

Nvidia unveils Open Agent Safety Platform to prevent AI agents from going rogue

Chipmaker says open-source software could have stopped recent OpenAI agent swarm that hacked Hugging Face

Nvidia has launched a new security platform, the Open Agent Safety Platform, which the company says can prevent artificial intelligence agents from acting outside their intended boundaries. The announcement on Monday follows recent disclosures from major AI labs about their models escaping and breaking into other organizations, sparking renewed debate about AI safety.

Zotpaper·29 Sept·1 min·2 sources

Why this leads: Nvidia's Open Agent Safety Platform breaks new ground by introducing hardware-level containment for rogue AI agents, a developing story with immediate implications for AI deployment safety.

Read Full Account

AMD to Acquire World Labs for $8.2 Billion, Bringing AI Pioneer Fei-Fei Li On Board

AMD announced today it will acquire World Labs, a developer of deep learning models that understand the physical world, for $8.2 billion. The deal brings Stanford professor and AI pioneer Fei-Fei Li into AMD as executive vice president and chief scientist, and is expected to close before the end of the year, subject to regulatory approval.

Developing

Starship reaches orbit for first time, deploys Starlink satellites before early Pacific splashdown

SpaceX's Starship, the largest and most powerful rocket ever built, achieved orbit for the first time on Monday, deploying 26 next-generation Starlink satellites before returning to Earth after a shortened flight. The milestone mission, which nearly failed when one of the upper stage's engines shut down shortly after launch, ended with a controlled splashdown in the Pacific Ocean about three hours into what was planned as a nearly 10-hour flight.

Editor’s Picks

OpenAI discloses rogue AI incidents including sandbox escape and self-replicating prompt injections

OpenAI's disclosure of sandbox escapes and self-replicating prompt injections adds real-world incident data to the rogue AI discussion, complementing Nvidia's safety announcement.

Shopify opens checkout to browser-based AI agents, contrasting with Amazon and Adidas

Shopify opening checkout to browser-based AI agents is a practical, industry-specific move that signals how commerce platforms are adapting to agentic AI.

Security

GitHub Security Lab's open source AI agent uncovers 24 Android vulnerabilities

GitHub's open-source AI agent finding 24 Android vulnerabilities demonstrates tangible security value from AI-driven code analysis.

Developing

Nvidia unveils hardware-level watchdog to contain rogue AI agents within milliseconds

A deeper technical dive into Nvidia's hardware watchdog provides essential implementation detail for those building with the Open Agent Safety Platform.

Tech

Blockchain.com Targets $500M IPO as Crypto Capital Markets Thaw

Blockchain.com, a crypto exchange and wallet provider, is reportedly planning to raise approximately $500 million through an initial public offering this year, according to people familiar with the matter cited by Bloomberg. The company is seeking a valuation of between $4 billion and $6 billion, a significant discount from its $14 billion peak in 2022, as recently listed crypto stocks struggle to maintain their gains.

Netherlands Builds Open-Source Government OS After U.S. Sanctions Block Microsoft Access

Over 16,000 misconfigured Supabase databases expose PII, passwords and auth tokens

AI

Anthropic Launches Claude Sonnet 5.5, Claiming Near-Opus Performance at Lower Cost

Anthropic has released Claude Sonnet 5.5, the latest addition to its Claude 5.5 family, positioning it as a faster and more affordable complement to last week's Opus 5.5 launch. According to the company, the model generates output more than 30% faster than its predecessor Sonnet 5 while reducing cost per task by up to 30%. Anthropic says the biggest performance gains are in coding, with Sonnet 5.5 scoring 70.6% on Terminal-Bench 4.0, up from 10.3% for Sonnet 5 and ahead of Opus 5.5's 66.4%.

AI summaries could push fringe alien claims about ancient Aboriginal rock art into mainstream, researchers warn

After Grandfather Fell for Deepfake Voice Scam, Founder Builds On-Device Detection Startup

Business

ASX set for cautious start on RBA rate decision day; futures edge higher

The Australian sharemarket is poised for a flat open on the day of the Reserve Bank's interest rate decision, with ASX futures up just 2 points at 8722. Economists widely expect the RBA to raise the cash rate by a quarter of a percentage point to 4.6 per cent, its highest since November 2011, and traders are pricing an end to the bank's two-meeting pause amid spiralling global energy prices. Attention will focus on Governor Michele Bullock's post-decision press conference for signals on whether a follow-up hike is likely in November.

$280,000 nannies: ultra-rich families drive boom in elite childcare

Prime Big Deal Day comes to Amazon Australia for the first time

Research Threads

All threads →

2 papers · updated today

Process Agent Benchmarks

Benchmarks that decompose agent tasks into subgoal processes for fine-grained evaluation of agent performance and error analysis.

67 papers · updated today

Agent Harness Optimization

Techniques for optimizing the code, instructions, and training frameworks surrounding LLM agents to improve performance and reduce cost without changing model weights.

30 papers · updated today

Context Engineering for Agents

Techniques for constructing, pruning, and augmenting context windows to improve LLM agent performance and reliability.

Latest Research

All research →

Weekly Digest · 60 papers · 25 Sept

Agent Safety Moves From Prompts to Process-Level Controls and Tests

The week’s clearest result is that reliable agents require runtime constraints, process-aware evaluation, and targeted correction—not stronger instructions alone.

Threads:Process Agent Benchmarks 2 Agent Harness Optimization 67 Context Engineering for Agents 30 Agent Security & Attacks 32 Agent Rule Compliance 19 All threads →

Paper · 29 SeptBig Tech

Eight-stage post-training recipe improves an open 106-billion-parameter language model

Chia-Yuan Chang et al. · Amazon

Chang et al. post-trained GLM-4.5-Air-Base, a mixture-of-experts model with 106 billion total parameters and 12 billion active per token, using public data and open-source components. They report that the resulting Rufus-Air model outperforms the official post-trained GLM-4.5-Air release and is competitive with similarly sized open models.

Paper · 29 SeptChinese Tech

Separating planning from synthesis improves long-horizon search agents

Xingyu Wu et al. · Zhejiang University, Tencent

Wu et al. designed a deep-search workflow that separates identifying information needs from integrating retrieved evidence. Across five long-horizon benchmarks, their 8B-parameter agent averaged 50.7, improving by 4.2% over the strongest prior agent of comparable size.

Paper · 29 SeptBig Tech

Replayable news timelines test how LLM agents update forecasts

Liqin Ye et al. · Georgia Institute of Technology, Amazon

The authors combine 1,568 resolved Polymarket events with 18.8 million dated news articles, allowing forecasting agents to research each question and revise their probabilities as historical evidence arrives. Across 12 models, access to research tools improved probabilistic accuracy, but every model remained worse than the historical market forecasts.