Prompts guide; code enforces
A system instruction can tell an agent not to delete files, but the instruction is not a security boundary. The code that grants filesystem access is. Treat model requests as untrusted proposals and make every effectful operation pass through a policy boundary.
Use least authority
Expose only the tools needed for the task. Scope each tool narrowly: read a specific project directory, search one catalog, or create a draft instead of publishing directly. Validate inputs with a schema and check the current user’s permission at execution time. Never rely on a tool description alone to restrict what the underlying function can do.
allowed_tools = {"search_catalog": search_catalog}
def execute_tool(name, arguments, user):
if name not in allowed_tools:
return {"ok": False, "error": {"code": "UNKNOWN_TOOL"}}
validate_arguments(name, arguments)
authorize(user, name, arguments)
return allowed_tools[name](**arguments)For operations with meaningful side effects—sending a message, deleting data, spending money, or changing production—require confirmation or an explicit policy decision before execution. Keep the approval decision separate from the model’s own proposal.
Put a budget around every run
- Turn limit: stops loops that never reach a final answer.
- Deadline and tool timeout: bound stalled model or service calls.
- Resource limit: control tokens, tool calls, compute, and spend.
- Cancellation: let the caller stop work and propagate that signal to tools.
- Retry policy: retry transient failures only when repeating the action is safe.
Be careful with retries. A failed network response does not prove that a payment or message was not processed. For non-idempotent actions, use an idempotency key or check the remote state before trying again.
Treat observations as untrusted input
Tool results can contain incorrect, sensitive, or adversarial text. A page can instruct the model to ignore its system rules; that page is data, not authority. Keep tool output clearly separated from trusted instructions, return only task-relevant fields, and do not include secrets in either model context or traces.
Ask your teacher to help threat-model a tool you plan to expose.