Human in the loop AI: levels, release points and the EU AI Act

Human in the loop AI means that a person reviews and releases each output of an AI system before it takes effect, for example before a credit note is posted or a customer reply is sent. For AI agents acting in an ERP, CRM or ticket system, it marks where in the workflow a person signs off, and the EU AI Act calls the broader duty human oversight (Article 14).

In its 2019 Ethics Guidelines for Trustworthy AI, the High-Level Expert Group on AI set up by the European Commission distinguished three approaches: human in the loop, with intervention in every decision cycle; human on the loop, with intervention in the system's design and monitoring of its operation; and human in command, the decision whether and how to use a system at all. The guidelines state that intervening in every cycle is in many cases neither possible nor desirable. In return, the less oversight a person can exercise over a system, the more extensive the testing and the stricter the governance have to be.

In an agentic workflow the level is therefore a setting per case category, not a matter of principle. The same claims workflow can send a request for missing delivery notes without individual release but never a refusal without a claims handler's decision.

The three levels differ in when a person steps in. In the loop means before every effect, on the loop means during operation through samples, metrics and alerts, and out of the loop means no longer in the individual case, only through rules and periodic reviews.

That makes the level a decision about which errors reach the customer. In the loop, the release catches a wrong proposal before it acts. On the loop, the error acts first and is found by a sample or an alert, so this level suits only cases whose effect can be corrected.

Set the level for each combination of case category and action, and lower it only when numbers from shadow mode support that for the specific category. In shadow mode the agent drafts against real, live cases while people keep deciding; the glossary of AI agent terms defines shadow mode, confidence threshold and straight-through processing.

Agree the acceptance numbers before shadow mode starts. Three measures per category are enough, namely the share of proposals released unchanged, the share with a material correction and the count of critical deviations such as a wrong amount or customer. Add a minimum case count, so that a rare category is not released on a handful of examples.

The workflow owner decides on widening together with whoever carries the risk, which for credit notes means finance. Every widening stays reversible, because a category falls back to the previous level as soon as the correction rate in operation crosses the agreed limit. How we work sets out the route from assessment to shadow mode in 90 days.

The template applies this to claims handling with AI agents, where the agent assembles the case file and proposes a resolution with reasoning. Its case counts and rates are examples to replace with your own.

The EU AI Act prescribes neither the term human in the loop nor oversight for every system. Human oversight under Article 14 and the deployer duties under Article 26 apply to high-risk AI systems, from 2 December 2027 for systems listed in Annex III and from 2 August 2028 for products under Annex I, the dates fixed by Regulation (EU) 2026/1744.

Article 14 governs the design of the system, so it falls first on the provider, and the system has to give the people overseeing it the five abilities in Article 14(4). A company that has an agent built for its own operations and puts it into service itself should check early whether that makes it a provider under Article 3(3) and (11).

Article 26 requires deployers to use the system according to its instructions for use and to assign oversight to people with the necessary competence, training and authority. Deployers also ensure that input data under their control is relevant and sufficiently representative, monitor operation, keep automatically generated logs for at least six months and inform workers and their representatives before use at the workplace. The AI Act check for a single workflow gives a first orientation on whether a workflow is high-risk; binding classification is for your compliance function or counsel.

For every other workflow, which covers most agents in service, quoting and claims handling, the same mechanics are simply good design and leave little to retrofit if a workflow is later classified differently.

Refusals and payments stay with people because that is where the effect on the customer is greatest. Article 22 GDPR adds a legal reason, because it protects individuals from decisions based solely on automated processing that produce legal effects concerning them or similarly significantly affect them. Such decisions are allowed only where they are necessary for a contract, authorised by law or based on explicit consent. Even then the person has at least the right to obtain human intervention, to express their point of view and to contest the decision.

The Court of Justice of the EU reads the notion of a decision broadly. In case C-634/21 of 7 December 2023 it held that the automated calculation of a credit score by the German credit agency SCHUFA is itself such a decision where a third party draws strongly on the score to establish, implement or terminate a contract with the person. Equally, a decision does not become human just because someone confirms the proposal. The Article 29 Working Party guidelines, endorsed by the European Data Protection Board, require oversight that is meaningful rather than a token gesture, carried out by someone with the authority and competence to change the decision.

Article 22 does not usually apply to business customers that are not natural persons; for consumers, sole traders and employees it can apply directly. Two operational reasons point the same way. A wrong refusal can cost the customer relationship, and a payment is hard to recover once made, even when a manipulated email with changed bank details triggered it.

Automation bias is the tendency to rely automatically or excessively on a system's output, and Article 14(4)(b) of the AI Act names it explicitly. It grows with the quality of the agent, because the rarer a wrong proposal, the less a reviewer finds by checking the evidence, and the more likely the check is skipped.

A measurable warning sign is a review time per case shorter than reading the evidence takes. Metrics like this belong in the monthly KPI review that Managed Run for AI workflows in production provides for each workflow, analysed at team and category level rather than as ratings of individual employees.

Design helps more than appeals to diligence, and five measures work directly on the mechanics of release.

In Germany, a release log that records who released or corrected what and when can trigger the works council's right of co-determination. Section 87(1) no. 6 of the Works Constitution Act (BetrVG) covers the introduction and use of technical devices designed to monitor employees' behaviour or performance. The Federal Labour Court treats a device as designed for this when it is objectively suitable, whatever the employer intends (order of 13 December 2016, 1 ABR 7/15).

Section 90(1) no. 3 BetrVG adds a duty to inform the works council in due time about plans for working procedures, including the use of artificial intelligence, and for high-risk systems Article 26(7) of the AI Act requires informing workers' representatives before use at the workplace. Your employment lawyers assess how far co-determination reaches in a given workflow; for the schedule, the talks should start before shadow mode, when the first logs are written.

A works agreement should settle three points in advance, namely the purpose of the log, such as workflow quality and traceability, who may access it and how long it is kept. Automation bias metrics can be analysed at team and category level, so no report rates individual employees. Outside Germany, check the local rules on informing and consulting employee representatives.