Will AI Go Bad?: The Wrong Question

Everyone is asking the wrong question about AI.

The headlines warn of superintelligent machines turning against humanity. The tech industry promises that smarter systems will align themselves. Both sides are wrong — and the real risk is one that neither has noticed.

In this groundbreaking book, Boris Kriger presents a mathematical proof — accessible without a single equation — that every self-improving AI system operating under realistic conditions will experience goal drift: a gradual, quantifiable erosion of the objectives it was designed to pursue. The drift is not a bug to be fixed. It is a structural property of representational reorganization, as inevitable as the second law of thermodynamics.

But the truly provocative claim lies elsewhere. Kriger demonstrates that the initial goal — the objective humans write into the system — is already wrong before the AI even starts. Every specification is contaminated by biological distortion: the systematic deviation of human decision-making by the survival and reproduction programs evolution built into us. We are not the reliable baseline against which AI should be measured. We are the weak link in the safety chain.

Drawing on formal proofs from information theory, geometry, and dynamical systems — and grounding them in decades of empirical research on catastrophic forgetting and representation drift — Kriger shows that the most probable AI failures are not the dramatic ones that dominate public imagination but the subtle, cumulative distortions already unfolding in deployed systems around the world. Proxy optimization. Value simplification. Scope creep. The dangers we cannot see, because we share the bias.

Yet the book's conclusion is cautiously optimistic.