An automation that can send a message, change a record or move money should stop and ask a person before it acts, and that answer should be the last thing that happens before the act. Everything on the other side of the line, reading, sorting, drafting, filing a copy, should run without asking anybody. The line is not about how good the model is. It is about whether the result can be undone, because a human check placed after the send button is worth nothing.
This is the design rule we hold a workflow to after we have picked the one worth automating first. It is one sentence, and you can hold any vendor's product to it.
The acts that cannot be undone
A sent email is gone. There is no recall, and the recipient has already seen the subject line in a preview. When the message is marketing, the cost has a printed number. Under CAN-SPAM, each separate email in violation carries a civil penalty of up to $53,088, the FTC's inflation-adjusted amount for penalties assessed after January 17, 2025. The law counts individual emails, so one badly configured run is measured message by message. A recipient's opt-out must be honored within 10 business days. Count ten business days from an opt-out that arrives on a Monday, ignoring holidays, and it lands on a Monday two weeks later; an automated suppression job has to have caught up by then. Hiring a vendor, or an agent, to send for you does not move the legal responsibility off your firm.
A changed record overwrites what was true. A note written into the client file, a rescheduled appointment, a reclassified transaction in last month's books: the previous value is gone from the screen the moment the write finishes, and what you recover afterwards depends on whether the system kept a history you can read. The books are the worst case, because a reclassified transaction changes a report someone has already acted on.
A payment that has left the account comes back only if someone agrees to return it. The CFPB's guidance on a transfer sent with the wrong details turns on whether the funds have been released. If they have not, a correction is usually possible. If the money does not reach the account you intended, you may not be able to get it back. The useful window closes when the money leaves.
Exhibit 1
The acts that wait for a person are exactly the acts that cannot be undone
Why the person stays in the loop
A good model does not need a chaperone, goes the objection. The work behind the answer predates AI and comes from process plants, human factors research and accounting.
Lisanne Bainbridge's 1983 paper "Ironies of Automation" in Automatica argued that a designer who removes the operator leaves him the tasks the designer could not work out how to automate. In an automated system the person is left to monitor that the automatics are working, and to take over when they are not. Her subject was process plants and aircraft, and the paper names no failure rates. The mechanism transfers. An automation that handles the routine cases leaves the approver the strange ones, which are the cases that need judgment, and it removes the daily practice judgment normally comes from.
Raja Parasuraman and Victor Riley, in Human Factors in 1997, named the failure mode of leaning too hard on automation "misuse" and defined it as overreliance on automation that shows up as failures of monitoring. They gave the opposite name, "disuse", to neglect or underuse, which they attributed mainly to alarms that fire falsely. The risk runs both ways, and an approval step sits in the middle of it: too much trust and the step becomes a formality, too little and people route around it.
Google's site reliability book says that doing automation thoughtlessly can create as many problems as it solves. Its case study is a wipe run by automation that trusted an implicit signal, a machine whose first disk was unused, and it destroyed a multi-petabyte dataset in moments. The same chapter says that people lose direct contact with the system and cannot operate it well when the automation fails.
Accounting has its own version of this argument. The standard control is segregation of duties: separating who starts a transaction, who authorizes it, who records it and who reconciles it. Where a firm is too small to split those jobs fully, the guidance in a January 2021 Journal of Accountancy article on fraud programs is to add compensating controls such as reconciliations, supervisory review of audit trails and management oversight. On outgoing money, CPAs recommend instructing the bank to release funds only under dual authorization.
The ACFE's 2026 Report to the Nations, its 14th edition, covers 2,402 occupational fraud cases investigated between January 2024 and September 2025. In that study the median loss at organizations with fewer than 100 employees was $126,000, the highest of the four size bands it reports and about $22,000 above the all-sizes median in the same edition; the report notes that the same dollar figure is a much larger share of a small firm's revenue. One third of the reported cases came from a lack of internal controls, and 19% from an override of controls that already existed, which together cover just over half of the cases. The ACFE publishes cases reported to it, so these figures describe what a fraud looks like when it is found and say nothing about how often one happens in a firm of your size.
What the approval screen has to show
The automation does the entire job up to the act. It reads the enquiry, finds the account, writes the reply, fills the payment file. Then it stops and puts one screen in front of a person, and that screen answers three questions in the order the person asks them: who receives this, what changes as a result, and how much. Approve, edit and decline are one action each. Keep the record of who approved what, because an approval nobody can reconstruct is not a control.
OpenAI's guide to building agents says human oversight should trigger on an action that is sensitive, irreversible, or high stakes. Its worked examples are cancelling a customer order, authorizing a large refund, and making a payment. Its method is to rate each tool by reversibility, permissions and financial impact, then pause the agent before it uses a high-rated tool. Anthropic's engineering guidance says agents should be able to pause for human feedback at checkpoints, and when they hit a blocker, and describes the deployments that worked best as tasks with clear success criteria, feedback loops and meaningful human oversight built in. It also warns that an agent's autonomy brings higher costs and the potential for compounding errors, which is why it recommends testing in sandboxed environments with guardrails.
Exhibit 2
An approved automation is identical to an unattended one until the last step
Batch the approvals for work that recurs. If the same decision arrives all day, collect it and present the day's drafts to one person once, in the morning, as a list where each row shows recipient, subject and the amount if there is one. Approving a queue takes minutes. Approving the same decisions as separate interruptions takes the rest of the day.
Work that should not ask
Reading, sorting, summarizing, drafting, filing a copy and copying a value from one place to another should run unattended. Approval on those steps is a tax on work that has no consequence, and it teaches people to approve without reading. NIST's AI Risk Management Framework, published in January 2023, treats processes for human oversight as something to define, assess and document before deployment, and it says outright that some systems do not need human oversight while others specifically do. The framework is voluntary. The useful part is the triage, deciding which is which, tool by tool.
Failure modes
The rubber stamp is the approver clicking through without reading. Parasuraman and Riley's "misuse" is overreliance that shows up as failures of monitoring, and a screen that asks the same question all day long is engineered to produce it. The fixes are to cut the number of asks, to put the changed field where the eye lands, and to make the decline button as easy as the approve button.
The bottleneck is the owner as the only approver. Every draft waits for one person, and the queue grows until approvals get given without reading, which restores the rubber stamp from the other direction. Approval rights should follow the job that already owns the outcome: the front desk approves replies, whoever runs the books approves the payment run. The owner's calendar is the place to check this, and the test is how long a routine approval can sit unattended before it costs you anything.
Exhibit 3
Both failure modes keep the approval screen and lose the check
Worked examples
An inbound enquiry arrives. The automation reads it, finds the client record, drafts a reply in your voice and holds it. The front desk opens one screen, sees that the draft quotes a price from a rate card the client is not on, edits that line, and approves. The reply goes out. A drafting pass like the one in triaging your inbox and drafting replies is worth having; the send is the part that needs the person.
A vendor payment run. The automation matches invoices to purchase orders, flags the ones that do not match, builds the payment file, and stops. The person who runs the books sees vendor, invoice number and amount per row, approves the batch, and the file goes to the bank under a dual authorization instruction you have given the bank.
A record change in a client file. The automation proposes the change and shows the current value beside the proposed one, and the write happens only after a person accepts. Under HIPAA, a vendor engaged to perform functions involving protected health information is a business associate, and the covered entity needs satisfactory assurances in a written business associate agreement before disclosing that information. OCR's published examples of business associates include an AI chatbot running on a provider's patient portal. What HIPAA requires of a practice using general AI tools is a separate question from who presses save.
The published guidance on oversight
NIST asks organizations to have mechanisms, with named responsibilities, to supersede, disengage or deactivate a system that behaves inconsistently with its intended use. The EU AI Act makes human oversight a design duty. Article 14 requires that high-risk AI systems be designed and developed so they can be effectively overseen by natural persons while in use, and for a narrow class of high-risk biometric systems it goes as far as requiring that no decision be taken on the system's output unless two people with the competence, training and authority to do it have separately verified it. Those duties attach to high-risk systems placed on the EU market. A US service firm using an automation tool at home is outside them, so treat the Act as evidence of what a serious regulator thinks an approval step should look like. The dates have moved. Regulation (EU) 2026/1744 pushed the high-risk parts of the Act; the rules covering Annex III systems now apply from 2 December 2027 and Annex I product systems from 2 August 2028.
The test we apply is one line. If you can undo the result in a minute, let the automation run. If you cannot, it asks first, immediately before the act, and it shows the person what the act is. Deciding which of your tools fall on which side, and building the screen that goes with the ones that do not, is the part of AI enablement we spend most of our time on, and it is the same conversation a forward deployed engineer starts with on the first day.