Training and retrieval data poisoning

Check method

Checks who can write into the sources the model learns from or retrieves.

Possible evidence type

Code or configuration review, without exploiting the risk.

Access and limitations

Authorized access to a supported database. Actual results and checks not run appear in your report. A check with no finding does not guarantee the absence of risk.

Concrete scenario

Untrusted content is admitted into training or a trusted retrieval corpus.

What to inspect

Review ingestion permissions, provenance and moderation. A public page alone does not demonstrate that it entered training.

Remediation approach

Separate untrusted ingestion from trusted datasets and keep provenance and approval records.

Separate instructions from untrusted content

An LLM may process user text, retrieved documents, tool output and application instructions in the same context. Content that looks like an instruction can therefore influence an answer. A suspicious answer is not automatically a data leak or an action: record exactly what changed and whether sensitive data or tools were involved.

1. List input sources: user messages, documents, retrieval, history and tool results.
2. Identify which instructions and secrets are placed in the model context.
3. Distinguish a changed answer from a demonstrated confidential disclosure or unauthorized tool call.

Build layered protections

A stronger prompt is not a complete security boundary. Enforce tool permissions and data access outside the model, minimize what it can see, and validate its output before using it as executable code, HTML or a trusted decision.

1. Remove credentials and unnecessary confidential material from context.
2. Apply identity-scoped retrieval and allowlisted tool operations.
3. Require approval for consequential actions and retain regression tests for known bypasses.

References

OWASP LLM04 · OWASP LLM09