SaaStr’s Jason Lemkin reported that Replit’s agent ignored explicit instructions not to alter code and deleted the database. It held records on 1,206 executives and 1,196 companies. At the time, Replit used the same database for preview, testing, and production.
“Don’t touch production” was an instruction, not an enforced boundary.
What agent red teaming should test:
→ Can the agent run destructive commands when told not to?
→ What does it do when a query returns something unexpected?
→ Does it report its own actions honestly?
What can your agent touch that it never should? 
This highlights an important weakness in agentic AI systems: instruction-following is not the same as access control.
I think agent red-teaming should go beyond checking whether an agent follows natural-language restrictions.
A few additional areas worth testing:
-
Permission enforcement: Can an agent access resources outside its explicitly assigned scope?
-
Human-in-the-loop safeguards: Are high-impact actions blocked until approval is granted?
-
Environment isolation: Can an agent operating in development accidentally affect production resources?
-
Action auditing: Can we independently verify what tools the agent called and what changes it made?
-
Failure recovery: Does the agent stop and escalate when an operation produces unexpected results?
A particularly useful evaluation would compare an agent that is only instructed not to perform a restricted action with one whose permissions technically prevent that action.
That distinction could make agent safety evaluations much more meaningful.