Cascading failures and runaway agents

Check method

Design review: loop limits, budget caps, kill switch and human override.

Possible evidence type

Code or configuration review, without exploiting the risk.

Access and limitations

Connected repository. Actual results and checks not run appear in your report. A check with no finding does not guarantee the absence of risk.

Concrete scenario

A retry loop repeatedly invokes tools without making progress.

What to inspect

Review iteration bounds, timeouts, budget checks and recovery after tool failures.

Remediation approach

Set enforceable run limits, stop on repeated failures and retain an auditable final state.

Model the agent’s actual authority

An agent combines a model with tools, credentials and state. Its risk depends on what those tools can actually do, not on the name of the model. A read-only search tool and a tool that deploys code or transfers funds need different controls. Indirect instructions from a page or another agent must not acquire the authority of the user.

1. Inventory tools, credentials, accessible data and external destinations.
2. Identify approval boundaries for writes, deployments, messages and payments.
3. Check that tool permissions are enforced independently of the model response.

Contain failure and verify boundaries

Use least privilege, isolated execution, bounded iterations and auditable actions. A limit in one tool is ineffective if an alternate tool can perform the same operation without it. Verification should demonstrate blocked misuse and continued normal operation; destructive production exercises need separate owner approval.

1. Use scoped credentials per tool rather than a shared administrator credential.
2. Set run and tool limits and stop on unexpected permission failures.
3. Review the full chain from incoming content to the final external action.

References

OWASP ASI08 · OWASP ASI10