And? Look to fossil fuels and the greenhouse effect — even with ample evidence of both a causal mechanism and the results, we've still got people who want to drill and burn more oil; and also, there's also plenty of people who want to switch oil off despite all the positive things that oil brings.

An AI which is "only" as misaligned as one of the major industrial sectors of the world, that is made out of humans who are necessarily able to care about human interests, and which drove a huge amount of economic improvement and brought with it derivatives such as basically all polymers, is still capable of great harm. And because of those benefits, society as a whole still has not unplugged the oil drills.

The more power there is in a system, the harder it is to keep it aligned with our best interests, even when there are humans not just at the wheel but occupying every step.

And the more aligned it is, the harder it is to "just unplug it" when things — as is inevitable because nothing is perfect — do go wrong.