There are two agents in this talk

1. The one that built the thing (hi Claude/Codex 👋)
2. The one that IS the thing (you're going to make one!)
   
   

(This is confusing. I didn't name them. Welcome to AI.)

Basically, we wanted this:

Is this an agent problem?

1. It recurs on a schedule
2. The inputs are machine-readable
3. The output has a fixed shape

4. A human was going to review it anyway

   Obviously.

Recipe card 1: the plan

ok so: every month someone at FPF reads everything that went wrong with AI and pulls out the privacy incidents for a digest. I want a small tool that does the reading and drafting, and a human does the editing.

Don't write code yet. Write docs/BUILD_PLAN.md. Lock the decisions: one container, one database file, one source to start (the AI Incident Database), the model drafts and never sends. Break it into phases I can build one at a time.

Sources: AIID only in v1, behind a pluggable SourceConnector interface so Federal Register / DPA feeds / arXiv / RSS can be added later.

Each source is a thing that breaks when nobody is watching. So: one first.

Recipe card 2: the build

Build Phase 0 from docs/BUILD_PLAN.md. Nothing from later phases. Tests must pass. Commit with a message that says what you did, then stop and show me.

Then I said "next phase" seven more times.

Burnt batch 1: the free filter

The plan said: don't pay a model to classify. Use the database's own "Privacy & Security" tag.

The tag flagged almost nothing.

The privacy tag from the database is catching almost nothing, maybe 3 of 30 incidents a month. Have the model screen every incident for a privacy angle instead. Keep the tag as a hint. Store the judgments so I'm not billed twice for the same month.

Burnt batch 2: the links

The model writes very convincing URLs. They don't resolve.

The model keeps inventing links that don't resolve. Change the design so it never writes a URL. It fills in four text cells per row; the date, the incident number and every link get joined from our own database when the page renders.

The fix was not a better prompt. It was taking the job away.

Recipe card 4: the runbook

Write docs/RUNBOOK.md for the person who will run this every month. Assume they have never seen the code and never will. Start with the monthly review, then what to do when the source is down.

Where the human stays

1. Read the member-mention banner first
2. Edit the four cells the model wrote; the rest is our data
3. Drop weak rows, pull stronger ones from the pool
4. Approve. That locks it.
5. Export, and paste into the real sending tool

Disclose twice

In the commit log, for whoever maintains it

In the footer, for whoever reads it

Add a one-line disclosure at the end of every export saying the table was drafted with an AI model from AI Incident Database records and reviewed by FPF staff.

The checklist ✅

1. Recurring task, machine-readable inputs, fixed output shape. If not, write a checklist instead.
2. Plan first. Lock the decisions. Write down what you are NOT building yet.
3. One source. Add the second when the first is boring.
4. Check the filter against a real month before you trust it.
5. Never ask the model for a fact you already have in a database.
6. Hard cost cap in code. Buttons that refuse to bill twice.
7. Hostile review before it touches the internet.
8. Decide where the human sits and make the tool stop there.

9. Disclose twice.

   
   (Hint: This is the slide to screenshot.)