Human in the loop automation means deciding, in advance, which actions your AI system may take on its own and which ones it must stop and hand to a person. The useful version is not a vague promise of “oversight” — it is a written rule, applied per action, based on four things: whether the action can be undone, how far the damage spreads if it is wrong, whether anyone outside the company sees it, and whether the facts behind it can be checked automatically. Actions that pass all four run unattended. Anything else pauses and asks.
Most teams get this backwards. They either put a human approval step on everything — which quietly kills the automation, because nobody approves forty items a day — or they remove approval entirely and discover the gap the hard way. This article sets out the tests we use at Codeora, with examples from systems we run ourselves every day.
What is human in the loop automation?
An AI workflow sits somewhere on a spectrum. At one end, the agent proposes and a person executes. In the middle, the agent acts but a person reviews before the result goes anywhere. At the other end, the agent acts and nobody looks. Human in the loop automation is the discipline of placing each individual action on that spectrum deliberately, rather than placing the whole system at one point.
That distinction matters because a single workflow almost always contains actions of very different risk. An agent that manages a website might fix a missing meta tag, rewrite a page title, and email a customer — three actions with nothing in common except the agent performing them. Treating them the same is the mistake.
The approval tax nobody budgets for
Every approval step costs attention, and attention is the scarcest thing in a small business. If your automation produces twenty items a day and each needs a tap, you have not automated anything — you have built a queue. Worse, approval fatigue makes the taps meaningless: by item twelve, the person is rubber-stamping, and the review that justified the whole design is no longer happening.
So the goal is not maximum oversight. The goal is oversight concentrated where it changes the outcome.
Which tasks should an AI agent do without asking?
An action is a good candidate for running unattended when all four of these hold:
- It is reversible. You can put it back the way it was in minutes, with no trace left on anyone else.
- The blast radius is small. If it is wrong, it affects one record, one page, one file — not a customer list, a billing run, or a live database.
- Nobody outside sees it before you do. Internal state, drafts, configuration. Nothing that lands in a stranger’s inbox or on a public page as a claim about your business.
- The facts are machine-checkable. The agent can verify its own inputs against a source of truth, rather than inferring them.
Housekeeping is the natural home for this: tidying metadata, regenerating a sitemap, filing documents, reconciling two lists, flagging anomalies. Boring, frequent, low-stakes, easy to undo. Our guide to admin tasks you can automate today covers more of these in detail.
When should an automation stop and ask a human?
The mirror image of the list above gives you the escalation triggers. In practice, four categories come up again and again.
Anything irreversible
Deleting, sending, paying, publishing to a third party, cancelling. The test is not “is the agent likely to be right” — it is “what happens on the day it is wrong.” An agent that is right 99% of the time and sends 200 emails a month sends two wrong emails a month, permanently.
Anything that becomes a claim about your business
Prices, capabilities, timelines, results. A generated sentence that overstates what your product does is not a content problem, it is a compliance problem — and it is the kind of error a model makes fluently and confidently. If the agent cannot trace the claim to a document you control, it should not make the claim.
Anything a real person is waiting on
An enquiry is not a task. Somebody asking about hiring you, about pricing, about a bug, about a partnership, has started a relationship, and the first reply sets its terms. This is the single clearest line we draw in our own systems.
Anything with a legal dimension
Decisions about people — applications, eligibility, account restrictions — carry rules that have nothing to do with how good your model is. Under UK GDPR, individuals have specific rights concerning decisions made by solely automated means where those decisions have legal or similarly significant effects; the ICO’s guidance on automated decision-making is the place to establish whether yours falls inside that. The NIST AI Risk Management Framework is a useful companion for thinking about the wider risk picture.
How do you write the escalation rule?
A good escalation rule is short enough to hold in your head and specific enough that the agent cannot argue with it. Four parts:
- A named trigger. Not “if unsure” — a model’s sense of its own uncertainty is not a reliable trigger. Use a category (“any message that mentions price”) or a threshold (“any invoice over £500”).
- A defined stop. What the agent does instead of acting: park the item, log it, notify, and — importantly — do not produce a placeholder action in the meantime.
- A route to a specific person, with the context that person needs to decide in under a minute.
- A resume path. The human’s decision goes back in and the item completes. Without this, escalation is just a dead-letter queue.
Write the allowlist, not the blocklist. Enumerating what the agent may do fails safe when something new appears; enumerating what it may not do fails open.
What does this look like in practice?
Three examples from systems Codeora runs, rather than hypotheticals.
The SEO agent that fixes ten things and refuses five
An agent audits this website every day. It may automatically fix a fixed allowlist of roughly ten site-level checks — sitemap generation, schema output, indexing flags, Open Graph tags, robots and permalink settings, and similar. Five recurring failures are deliberately excluded and left for a person, because each needs a judgement rather than a correction: choosing focus keywords that genuinely match a page’s intent, for example, or making a server configuration change. The agent reports them every day and never touches them.
The publishing system that never answers an enquiry
Our marketing engine writes and publishes to several channels unattended, and comments on other people’s posts. It is explicitly forbidden from answering a real enquiry. If a reply looks like hiring, pricing, a bug report, a partnership or a press request, the agent records it, notifies a human, and stops. The founder supplies the answer and the agent posts his words. That rule costs the system very little and removes the one failure mode that would actually damage the business.
The receipt scanner that extracts but does not decide
Receiply, our own app, reads a receipt and works out the merchant, total, date, category and line items — then shows them for confirmation. What it deliberately does not do is decide what is deductible or file anything with a tax authority. The extraction is mechanical and checkable; the tax judgement is neither, so it stays with the person. The boundary is a product decision, not a technical limit.
What are the common mistakes?
- Approval on the wrong step. Reviewing the agent’s draft while its data source goes unchecked reviews the wrong artefact.
- Escalating without context. A notification that says “needs review” and links to a dashboard will be ignored. Put the decision in the message.
- Silent fallbacks. An agent that degrades to a default instead of stopping hides the failure. If it cannot do the job, it should say so loudly.
- No log. If you cannot reconstruct what the agent did and why, you cannot tune the rule — and the rule is the product.
Frequently asked questions
Is human in the loop automation the same as human oversight?
No. Oversight is a general commitment to supervise a system. Human in the loop automation is a per-action rule that says which specific steps pause for a person. Oversight without that rule tends to mean nobody looks at anything in particular.
Does adding a human step defeat the point of automation?
Only if it is on every step. The aim is to remove approval from the high-volume, reversible work so that attention is available for the small number of actions that genuinely need it.
Can an AI agent decide for itself when to escalate?
Partly, but do not rely on it alone. Models are poor judges of their own uncertainty and can be confidently wrong. Escalation triggers should be defined by category or threshold in the surrounding system, with model-flagged uncertainty as an extra signal rather than the only one.
Where should a small business start?
List the actions one workflow takes, then mark each as reversible or not. Automate the reversible ones fully, escalate the rest, and review the log after a fortnight. That exercise is usually more valuable than choosing a tool.
What should never run unattended?
Payments, deletions, outbound messages to customers, anything that publishes a claim you cannot trace to a source document, and any decision about a person that carries legal weight.
Building this properly
The hard part of an AI project is rarely the model. It is deciding which actions the system is allowed to take, writing that down, and building the escalation path that makes the boundary real. That design work is most of what we do on a workflow automation engagement, alongside AI agents and copilots built to run inside those limits.
If you are weighing up an automation and are not sure where the line should sit, get in touch — describing the workflow out loud usually locates the boundary within a few minutes.