Skip to content

Run Every AI Tool You Pay For Through the Seat Test

In youth baseball, one of the first things a coach teaches the outfield is how to call the ball. A fly ball drifts into the gap between left and center, both kids run toward it, each one sure the other has it, and it lands on the grass between them. Nobody dropped it, exactly. It just had no owner.

Coaches fix that with one rule: whoever calls the ball gets it, and everyone else gets out of the way. The rule doesn't make anyone a better athlete. It decides, before the ball is in the air, whose ball it is.

Most teams I talk to have a few fly balls in the gap right now, and they're AI tools: a meeting note-taker somebody signed up for, a ChatGPT account a producer pays for herself, the assistant built into the software you already run, and a custom GPT someone built over a weekend that half the office now leans on. Each one is useful. When one of them gets something wrong, nobody's quite sure whose it was.

Four questions for every tool you pay for

The Seat Test is how I sort those tools, and it's four questions. Does it have a named role? Does it have a defined job? Does a human own it? Is there a decision it never makes? Four GOs and it earns the seat. Anything less is a subscription with ambitions.

The questions come straight out of what a Digital Crew is. A Digital Crew is a governed team of AI agents with named roles, defined jobs, a human manager, and decision rights that stay with people. The crew takes the busywork. Your best people get back to the work only they can do. Each clause in that definition becomes one question, which is why a tool that passes all four is a crew member and a tool that misses one is something else you're paying for.

A named role. A job you could say out loud, the way you'd introduce a new hire: this is our renewal Scribe, this is the Coach that answers new people's questions from our procedures. "It's AI" doesn't count. If you can't name it, nobody else on the team can find it either.

A defined job. You can say what it works from and what it hands back. The output can vary; the job doesn't. "Drafts the follow-up email from my appointment notes" is a defined job. "Helps with email" is a hobby.

A human owner. One person, by name, who reviews its work and answers for what it produces. A department doesn't count, and neither does whoever's account it happens to run on. In baseball terms, someone has called the ball.

A decision it never makes. You can name the line it doesn't cross, written down ahead of time with the reason attached. That written line is a Flight Rule. For an insurance agency, it might be that the tool never sends anything to a client until a licensed agent has read it. The tool may be perfectly capable of hitting send. You've decided it doesn't.

One agency's stack, polled

Here's how that plays out on the kind of stack I see in insurance agencies. These are composites, not anyone's real shop, but you'll probably recognize them.

The meeting note-taker usually passes the first two questions and misses the last two. It has a clear role and a clear job, turning a call into a summary. Nobody owns the summaries, though, so they pile up unread or get pasted into the agency management system without anyone checking them. And nobody has written down what it should never do with one, like send it to everyone on the invite.

The ChatGPT account a producer pays for herself often misses all four. It's a capable tool doing a dozen jobs, none of them written down, on a personal plan nobody at the agency manages, with no line it won't cross. That's Shadow AI, and it isn't her fault. Nobody gave her a better option.

The assistant built into the agency management system has a role and a job from the day it's switched on. It may also come with sensible guardrails from the vendor, but those are written around what the vendor thinks is risky. Your agency's rules are yours to write, so it's a NO-GO on question four until you write your own line. Question three stays open until someone on your team takes it.

The custom GPT that drafts quote emails comes closest. It has a name, one job, and an owner who built it and keeps it current. Write down the one thing it never does, and it earns the seat this afternoon.

Notice where the misses land. Most tools fail on the third and fourth questions, and those two cost nothing to fix: a person's name and one written sentence.

What to do with a NO-GO

A NO-GO is a list of what's missing. Sometimes the fix takes ten minutes: name an owner, write the line, narrow the job. Sometimes the honest answer is that nobody wants to own it, and that tells you it's a subscription you can let go.

Two limits are worth saying plainly. The Seat Test doesn't tell you whether a tool is any good. A tool can pass all four and still draft badly, and that's a different check, the kind in how to evaluate an AI tool before your team trusts it. The test also doesn't hold forever. A seat is held, not granted, so run it again when the model updates, when the workflow changes, or when the owner leaves, because an owner who's gone is a NO-GO on question three the day they walk out.

There's one more question I ask at renewal time, and it's blunter than the other four: if this subscription vanished tomorrow, would you rebuild it? For most of a typical stack, you already know the answer.

If you'd rather run the test with someone who's done it before, a Crew Launch is two live sessions for a team of up to 12. We pick the work worth fixing first, redesign it, and you leave with your first Digital Crew member working on it, with a written role description and a named owner on your team.

Questions people ask about the Seat Test

Is the Seat Test only for AI agents?

No. It works on any AI tool your team pays for, including chat assistants and the features built into software you already run. Agents make it more urgent, because they act without waiting for you.

What counts as a decision it never makes?

A specific action, written down with the reason, that the tool doesn't take even when it could, like sending something to a client or quoting a price. "Be careful" isn't one. If you couldn't check afterward whether the line held, it isn't written tightly enough yet.

Who should own a crew member?

The person closest to the work it does, not IT and not whoever bought the license. They're the one who'll notice it drifting before a client does.

Share with your network!
Mike Allton

Mike Allton

Mike Allton helps teams hire a Digital Crew of governed AI agents that take the busywork so their best people get back to the work only they can do. An AI keynote speaker, creator of the Digital Crew System and founder of The AI Hat, he's a practitioner, not a futurist, and runs his own business on one. He hosts The AI Hat Podcast and works the rooms he teaches, from national sales meetings to insurance and Medicare distribution conferences. He also helps organizations clear the Admin Drag burying their best people and bring Shadow AI into the light through governance and training.View Author posts