When engineers build an autonomous system, they automate the predictable tasks. What do they leave for the human?

Often, the most complex, chaotic, and dangerous situations. The ones they couldn’t figure out how to automate reliably.

In 1983, Lisanne Bainbridge published Ironies of Automation in Automatica. She examined industrial process control and the problems left for human operators as more work became automated.

More than four decades later, I find the paper remarkably relevant to software engineering. The systems have changed, but we are still making a familiar assumption: when automation reaches its limits, a capable human will be ready to take over.

These are four ways I see that tension playing out as we move from AI assistants to autonomous agents.

1. The better automation gets, the harder the remaining work can become

AI can help generate a feature, write tests, and fix straightforward errors. That can save substantial effort.

But consider what remains when a generated feature causes an intermittent production failure. Perhaps each component passes its tests, while a retry interacts badly with a transaction boundary under load.

The engineer now has to reconstruct decisions they may not have made, understand how the pieces interact, and investigate behavior the tests didn’t capture.

The routine work has disappeared. The difficult work still requires someone who understands the system.

This doesn’t mean automation makes every engineering job harder. It means the work left for people can demand more expertise even as the volume of manual work falls.

2. Automation can weaken the skills we need when it fails

Debugging develops through practice. You form a hypothesis, inspect evidence, discover that you were wrong, and try again. Over time, you learn where to look and which explanations deserve suspicion.

If an agent handles every investigation, an engineer may get the fix without going through that learning process.

An explanation of the solution helps, but reading an explanation is a different activity from deciding what evidence to collect next.

I don’t think using AI inevitably erodes skill. An engineer can use it to test hypotheses, explore unfamiliar code, or challenge an explanation. The risk appears when delegating the work also means delegating the thinking, consistently and over time.

What happens to our debugging judgment after years of rarely exercising it?

3. Reliable systems give us fewer chances to practise recovery

Suppose an automated workflow succeeds thousands of times before it fails. The person responsible for recovery may never have handled that failure before.

Reliability is valuable. We should want fewer incidents. But a quiet production environment doesn’t demonstrate that the team knows how to recover from an unusual failure.

A runbook can tell an engineer which command to execute. It may not tell them whether that command is appropriate when the system is already in a partially failed state.

That is a reason to rehearse recovery deliberately. Teams can practise stopping an agent, undoing its changes, and diagnosing a failed workflow in a controlled environment, before those actions become urgent.

4. Supervision requires understanding, and understanding takes time

We are asking engineers to supervise agents that can produce changes faster than a person can carefully examine them.

A large diff can contain individually reasonable decisions that combine into a poor design. Passing tests helps, but the reviewer still needs to ask whether the tests cover the important behavior and whether the implementation respects the system’s constraints.

If an agent generates thousands of lines in minutes, allocating a human reviewer doesn’t automatically create enough attention to understand them.

I think this should influence how we assign work to agents. Smaller changes, explicit assumptions, and clear acceptance criteria give reviewers a manageable decision to evaluate. Review capacity needs to shape generation capacity.

Preserving the ability to intervene

The central irony is uncomfortable: we automate everyday work to reduce dependence on human expertise, while continuing to rely on that expertise for exceptional situations.

For engineering teams, I would turn that concern into a few concrete habits:

• Keep engineers involved in diagnosis. Ask for a hypothesis and supporting evidence before accepting an agent’s proposed fix.
• Practise recovery. Rehearse rollback and manual intervention, including situations where the agent’s account of events is incomplete.
• Make changes reviewable. Keep tasks small enough that a person can understand the consequential decisions.
• Give learning explicit space. Let less experienced engineers investigate problems with support, even when an agent could produce an answer sooner.

These practices take time. That time belongs in the cost of operating an autonomous system.

The less humans participate in routine work, the harder it may become for them to handle exceptional work.

I think this is one of the most important challenges we need to solve as we move toward autonomous agents. Alongside asking how much work we can automate, we need to ask how people will continue to develop judgment, maintain expertise, and intervene effectively when automation fails.

The original paper is a short, worthwhile read: Lisanne Bainbridge, Ironies of Automation, Automatica, 19(6), 775–779, 1983 (PDF).