Didn’t “Just Do It”: When AI Runs but Humans Trip over Potholes

I cannot help being wary when a brand promises that more data will mean a more personal experience. Nike, Amazon and Mastercard bring different versions of the same question into view: who gets to define what is normal, useful or safe for the customer?

Nike’s technology partnerships make me wonder who controls customer data as it moves between organisations. What happens when a partnership changes? Who remains responsible? Sustainability raises another question: what energy and resources sit behind the efficiency story? I want the innovation, but I also want to understand the terms.

Footwear makes the limits of an average especially tangible. My husband wanted to try the HOKA experience after hearing about it from friends. Some of those friends were disappointed with how the shoes worked for them. That does not tell me which technology made the recommendation, but it does remind me that a runner meets more than a neat profile: unusual movement, changing conditions and potholes do not politely stay inside a model.

As O’Neil (2016) argues in Weapons of Math Destruction, models can reinforce exclusion when the assumptions or data behind them fail to represent the people affected. For any sizing or recommendation system, the question is how the exceptions are recognised and how a customer can challenge the result.

Amazon? HA! I have had a package waiting to be returned because the size recommendation did not work for me. One account may also shop for several people—in my case, family and friends. I wonder how that changes the picture the system builds. Yet if an app started asking intrusive questions about every swimsuit, I would not welcome that either. Better personalisation and less intrusion are not always an easy pair.

I have also had perfectly ordinary purchases blocked on my Mastercard. I cannot tell from the checkout which part of the payment system made the decision, but I can tell you what it feels like: being stopped from using your own money. I understand the purpose of fraud prevention. Sometimes, though, being “protected” feels more like being policed.

Across these experiences, the tension is between helpfulness and intrusion. Personalisation can make life easier, but I want to know who controls the information and what happens when the prediction is wrong. At what point do we stop feeling like customers and start feeling like data sources?

Relying purely on automated decisions can feel like bureaucratic red tape: the rules look logical until an ordinary life becomes an exception. More data does not automatically answer the question of what the system should do with that exception (O’Neil, 2016).

Human governance matters because societies, expectations and values change. A system needs people who can question its assumptions and remain responsible for its effects (Bryson, 2020). Otherwise, we risk building automated bureaucracy that runs at the speed of data.

We are not perfect either. The challenge is to keep room for judgment, challenge and adaptation—the messy qualities that help human systems learn and grow.