Systems in Control

In this episode

The connection

A driver sits behind a wheel that turns itself, feeling their focus slip away as the machine takes over. Writing for Knowable Magazine on July 13th, Amber Dance looks at why humans naturally switch off when automation feels reliable. It’s a quiet kind of surrender. But there’s a sharper side to these systems. In a piece from STAT News published this week, Joy Evers describes the wall she hit as an emergency doctor trying to get care for her husband. While one person drifts because a system works smoothly, the other is wide awake because a rigid insurance logic won't let her act. It’s worth pausing on how these structures either lull people to sleep or lock them out.

Amber Dance leaves the listener with the image of a driver sitting behind a wheel, hands off, slowly losing the ability to react because the machine feels too reliable. It’s a state of forced passivity, where the biggest risk is a mind that has simply drifted away. But notice what happens when that systemic control moves from a car’s dashboard to a hospital’s billing office. Where Dance describes the danger of humans falling asleep at the switch, Joy Evers, up next, shows the agony of a professional who is wide awake and screaming for the switch to work. Evers is an emergency physician who knows every lever of the medical world, yet she finds herself trapped by a different kind of automation: the rigid, unyielding logic of insurance protocols. The two pieces share a chilling observation about what happens when a system is designed to prioritize its own internal rules over the messy, unpredictable needs of a person. In the first case, the system lulls the human into a false sense of safety; in the second, it becomes a wall that no amount of professional expertise can break through. The tragedy Evers describes isn't about a machine failing to drive, but a system succeeding at following its own rules until it’s too late.

Hands resting in a lap while a car steers itself looks like a quiet trust in the sensors. That trust turns into a wall when an ER doctor finds her medical degree carries less weight than an insurance company’s software. Whether it’s a driver drifting into a daydream or a spouse fighting a rigid billing code, the friction remains the same. It’s the gap between what a person knows and what the machine allows. When a system is designed to be flawless, where is the room left for a human to change its mind?