AI is software, so in a sense everything that can go wrong with AI must be a subset of things that can go wrong with software.
Lots of software has a test environment. Even in live, e.g. bank account losing your money the transactions can be un-wound.
And that's the difference when it comes to replacing software devs with LLMs vs replacing CEOs with LLMs: it's possible to write the tests and then lock them. And to perform code review before changes are merged into the main branch. And to test against something other than production.
I know the Board can in principle remove a CEO, but is there even a standardised way for a CEO to have a vice-CEO that always checks everything they do, that always tests their commands against a simulation of the company and only lets them proceed if they agree with the outcome?
The point is that "AI as CEO" would be in the category of "business-critical" software, and also that current approaches to AI also lack sufficient guarantees of obligation compliance or sufficient defence against failures, which in the banking example would be things like the AI deciding to save money by deleting the system capable of unwinding incorrect banking transactions.
To the extent this kind of failure mode happens with vibe coding (in the original coining of the term: always accept without reading), it's like letting the LLM modify the unit tests to always succeed regardless of the code.
Lots of software has a test environment. Even in live, e.g. bank account losing your money the transactions can be un-wound.
And that's the difference when it comes to replacing software devs with LLMs vs replacing CEOs with LLMs: it's possible to write the tests and then lock them. And to perform code review before changes are merged into the main branch. And to test against something other than production.
I know the Board can in principle remove a CEO, but is there even a standardised way for a CEO to have a vice-CEO that always checks everything they do, that always tests their commands against a simulation of the company and only lets them proceed if they agree with the outcome?
The point is that "AI as CEO" would be in the category of "business-critical" software, and also that current approaches to AI also lack sufficient guarantees of obligation compliance or sufficient defence against failures, which in the banking example would be things like the AI deciding to save money by deleting the system capable of unwinding incorrect banking transactions.
To the extent this kind of failure mode happens with vibe coding (in the original coining of the term: always accept without reading), it's like letting the LLM modify the unit tests to always succeed regardless of the code.