Agent Oversight Mapping: The Quick Version
Free. No email required.
This is the quick version. The full version is The Agent Oversight Mapping Playbook.
This page is for a team whose AI agents can take actions without a person: merge code, deploy, send messages or spend money. In about thirty minutes you will have the five places where that matters most, and next to each one either a person's name or the word “nobody.” It runs on what the people in the room remember, so treat it as a first look.
Who is in the room. Bring two or three people who set up and run the agents every day, along with the people who are supposed to be watching what the agents do. Keep it to about eight. One person runs the meeting, keeps time, fills in the table and breaks ties.
Before you start, the person running it says this out loud: “Nobody” is a fact about how the work is set up, and it is not a mark against anyone.
- Write separate lists (5 minutes). Each person writes down the places they know an agent acts without a person and changes something. Nobody compares lists yet.
- Pick five (5 minutes). Each person reads out the two places on their list where a mistake would do the most damage. Take the five that get named most often.
- Ask two questions about each one (10 minutes). First, who is involved today? Write one person's name, “nobody,” or “don't know.” If the answer is a team, ask which person, and then ask that person when they last looked. If they can't say, write “nobody.” Second, how would you undo it, and how long would that take?
- Decide about the ones with nobody (10 minutes). Start with the one that would take longest to undo. Either a person in the room takes it and says what they will look at and how often, or the group decides on purpose to leave it as it is. Find out about any “don't know” after the meeting.
| Where an agent acts without a person | Who is involved today | How to undo it, and how long | What we decided |
|---|---|---|---|
| 1. | |||
| 2. | |||
| 3. | |||
| 4. | |||
| 5. |
When to stop early. If the people who run the agents can't come up with three places, the team probably doesn't need this yet, and you can say so and stop. If all five have a name and everyone agrees that person really does look, you can also stop.
No scores. Don't count up the names, turn the page into a percentage, or compare one team's page with another's. A score gives people a reason to leave places off the list.
Afterward. Keep the page, and do this again when the agents are given more to do.
From The Agent Oversight Mapping Playbook by Edward Schaefer. This page is licensed CC BY-SA 4.0: copy it, change it and share it, as long as you credit the source and keep the same license on what you share.