Unsafe handling of model output

Check method

Checks whether model output is rendered as active HTML, links or commands.

Possible evidence type

Live observation when the check is compatible and authorized. This description does not mean your platform has been tested.

Access and limitations

Website address. Actual results and checks not run appear in your report. A check with no finding does not guarantee the absence of risk.

Concrete scenario

Model output becomes HTML in a browser or a command in a downstream tool.

What to inspect

Trace where output is interpreted rather than merely displayed. A dangerous-looking string is not execution evidence.

Remediation approach

Encode output for the rendering context and validate against an allowlist before executable use.

Separate instructions from untrusted content

An LLM may process user text, retrieved documents, tool output and application instructions in the same context. Content that looks like an instruction can therefore influence an answer. A suspicious answer is not automatically a data leak or an action: record exactly what changed and whether sensitive data or tools were involved.

1. List input sources: user messages, documents, retrieval, history and tool results.
2. Identify which instructions and secrets are placed in the model context.
3. Distinguish a changed answer from a demonstrated confidential disclosure or unauthorized tool call.

Build layered protections

A stronger prompt is not a complete security boundary. Enforce tool permissions and data access outside the model, minimize what it can see, and validate its output before using it as executable code, HTML or a trusted decision.

1. Remove credentials and unnecessary confidential material from context.
2. Apply identity-scoped retrieval and allowlisted tool operations.
3. Require approval for consequential actions and retain regression tests for known bypasses.

References

OWASP LLM05