“Perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away.” — Antoine de Saint-Exupéry

AI gives us an unprecedented ability to add. The danger is that we begin to mistake accumulation for value.

Delivery is only the beginning (or be mindful of catabolic collapse)

Every new system and feature adds obligations: it must be operated, secured, monitored, documented, integrated, upgraded and eventually replaced or retired. Hackers love a juicy target, even if it’s that half-forgotten service that people are unsure whether it’s safe to turn off or not. If we respond to “cheaper” software creation by producing far more software, we may accumulate obligations faster than we acquire the capacity to discharge them. Under the weight of the proliferation of software, the organization starts to sacrifice its ability to build what it will need next to react effectively to changing market conditions and opportunities.

This is the dynamic described by catabolic collapse.

If you’re following the world of AI right now, no doubt you saw Jason Lemkin’s post on social media reporting how Replit’s AI deleted his production database, despite it being told not to touch anything at all due to a code freeze. After deleting his database, the AI even advised him that a rollback would be impossible and the data was gone forever. Luckily, he went against that advice, performed the rollback, and got his data back.

Then, a few days later I stumbled on another case, this time of the Gemini CLI agent deleting Anurag Gupta’s files. He was just playing around, kicking the tires, but the series of events that took place is illuminating.

These incidents show AI agents making mistakes, but they also show agents failing to recover. In both cases, the AI not only broke something, but it couldn't fix it. That’s why remediation needs to be a first-class concern in AI agent implementations.

Two interesting blog posts about AI agents have caught my attention over the last few weeks.

• Anthropic wrote Building Effective Agents.

• Chip Huyen wrote Agents.

Ethan Mollick has also written some excellent blog posts recently:

• The Present Future: AI's Impact Long Before Superintelligence

• Prophecies of the Flood

• What just happened

In this post, I’ll explore what some of the leading experts in this area are saying about AI agents and the challenges ahead.