“This is not philosophy, this text is following in the footsteps of Alan Turing” (paraphrasing) is both incredibly humble (/s) and incredibly dismissive of philosophy as a structured form of generating knowledge.
Putting that to the side - i don’t think I’ll read this fully soon, but the core thesis of “imitation is intelligence” can be easily disproven by a process that exists in society. An actor acting to be a genius is in fact, if they are a good actor, indistinguishable in their appearance to a genius. Yet they are not, in fact, a genius, they’re just good at memorisation. This is a clear showcase that imitation of a level of intelligence does not mean that this level of intelligence is present.
We have fallen into a trap of thinking that answering in plausible sentences is what makes humans intelligent. While in reality we are observing an actor responding from an infinitely large script. What makes humans intelligent (reasoning from first principles and pattern recognition across all the sensory inputs of the world) is still very much out of grasp.
> An actor acting to be a genius is in fact, if they are a good actor, indistinguishable in their appearance to a genius.
An actor will be distinguishable from a genius in their ability to answer questions and generate new insights. If the imitation was actually perfect, the actor would be able to do these things, and would in fact be a genius.
Except google and facebook have locked in numbers at times of virtually no competition before they started scaling up ads. If Open AI starts scaling ads next year they will churn people at a rate that will not be offset by growth and will either plateau or more likely lose user numbers, as their product has no material edge to alternatives in the market.
I disagree with Zitron’s analysis on many points, but I don’t see Open AI achieving the numbers it needs. Investors backing it must have seen something in private disclosure to be fronting this much money. Or more precisely, I need to believe they have seen something and are not fronting all this money just based on well wishes and marketing.
Most people don't choose by blind taste test. How intrusive do those ads have to get before it overwhelms habit and familiarity? OpenAI might be betting on enough of its 800m and growing weekly users sticking around long enough to moot churn until a liquidity event pays everyone off.
Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page.
If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of information and thus this is not a problem. Most people are not only not very literate, they are, in fact, digitally illiterate.
Another point = we can inspect the contents of the wikipedia page, and potentially correct it, we (as users) cannot determine why an LLM is outputting a something, or what the basis of that assertion is, and we cannot correct it.
The problem is that the good websites are constantly scraped/botted upon by these LLM's companies and they get trained upon and users ask LLM's and not go to their websites so they either close it or enshitten it
And also the fact that its easy to put slop on the internet more than ever so the amount of "bad" (as in bad quality) websites have gone up I suppose
Unfortunately, the Gen AI hypesters are doing a lot to make it harder for people to attain literacy in this subdomain. People who are otherwise fairly digitally literate believe fantastical things about LLMs and it’s because they’re being force fed BS by those promoting these tools and the media outlets covering them.
Of course there are many illiterate people, but the interesting fact is that many, many literate, educated, intelligent people don't understand how tech works and don't even care, or feel they need to understand it more.
The real world use cases for LLM poisoning is to attack places where those models are used via API on the backend, for data classification and fuzzy logic tasks (like a security incident prioritization in a SOC environment). There are no thumbs down buttons in the API and usually there's the opposite – promise of not using the customer data for training purposes.
The question was where should users draw the line? Producing gibberish text is extremely noticeable and therefore not really a useful poisoning attack instead the goal is something less noticeable.
Meanwhile essentially 100% of lengthy LLM responses contain errors, so reporting any error is essentially the same thing as doing nothing.
Reporting doesn't scale that well compared to training and can get flooded with bogus submissions as well. It's hardly the solution. This is a very hard fundamental problem to how LLMs work at the core.
Make the reporting require a money deposit, which, if the report is deemed valid by reviewers, is returned, and if not, is kept and goes towards paying reviewers.
You're asking people to risk losing their own money for the chance to... Improve someone else's LLM?
I think this could possibly work with other things of (minor) value to people, but probably not plain old money. With money, if you tried to fix the incentives by offering a potential monetary gain in the case where reviewers agree, I think there's a high risk of people setting up kickback arrangements with reviewers to scam the system.
... You want users to risk their money to make your product better? Might as well just remove the report button, so we're back at the model being poisoned.
Your solutions become more and more unfeasable. People would report less or anything at all if it costs money to do so, defeating the whole purpose of a report function.
And if you think you're being smart by gifting them money or (more likely) your "in-game" currency for "good" reports, it's even worse! They will game the system when there's money to be made, who stops a bad actor from reporting their own poison? Also who's going to review the reports and even if they finance people or AI systems to do that, isn't that bottlenecking new models if they don't want the poison training data to grow faster than it can be fixed? Let me make a claim here: nothing beats fact checking humans to this day or probably ever.
You got to understand that there comes a point when you can't beat entropy! Unless of course you live on someone else's money. ;)
RAG still needs model training, if the models were to go stale and the context drifts sufficiently, the RAG mechanism collapses.
Sure, those models are cheaper, but we also don’t really know how an ecosystem with a stale LLM and up to date RAG would behave once context drifts sufficiently, because no one is solving that problem at the moment.
I find this take naive. First, to have a zero sum game or indeed a positive sum game you have to be playing with perfect information with rationally behaving actors. Given most organisations have high levels of uncertainty and are resource constrained you can’t rationally make positive sum game decisions as the interpretation of uncertainty is cardinal to it - and additionally the resource constraint means different views of that uncertainty will tend to bias towards the thing they know best - engineers will find more certainty in build, marketers in marketing, designers in design - take your pick.
This necessitates collaborative information synthesis to resolve uncertainty uniformly to then be able to play a positive sum game under constraints. This is possible but it necessitates exchange of information between different business functions.
As informational clarity is a communicative process with repetitive feedback cycles, it will tend to have a big delay in the overarching system of decision-making. Therefore a shortcut is to influence, i.e. use conviction processes to shorten the cycle, rather than repeat to arbitrary infinity in order to drive perfect information alignment.
Therefore influencing is a necessary component even in an otherwise perfectly healthy and incentive aligned positive sum system of rational actors - and politics are influencing.
The problem becomes when conviction isn’t used as shortcut for informational clarity but as a method of exploitation of irrationality of human actors - this is bad politics.
What I do agree with is that putting in place right incentives, processes and organisational structure minimises politics - and in an org with rational actors this is the goal.
But good luck hiring perfectly rational actors in each function, that will still behave rationally in an economic downturn :).
If your c-suite is idiotic or nepotistic you can absolutely still influence them with good politics, you just need to understand their incentives and frame your arguments that way. You need to understand that you’re not playing meritocracy and get your outcomes done in the system you are playing.
In this case that means being in that golf game or figuring out a way how you can use corruption to get good outcomes done.
Or, more likely if your moral compass is sound, quit and find an organisation that isn’t like this.
While I agree with you that random corporate world does behave this way, companies where founders are still around - don’t - because they’re mission driven.
Great post. I’d just take it a step further and point out that this doesn’t stop at software or work.
A person can not “not be in politics”. You can only choose to have politics that affect you happen without your input. That’s how you end up with bad governments (in your mind).
The most important thing to learn about passivity is that it’s not a neutral position of exclusion. It is an active choice to not participate and be at the receiving end of the outcome.
I came here to write the same comment you did. What I’d suspect (I don’t work in self driving but I do in AI) is the issue is that this mode of operation would happen more often than not as the sensors disagree in critical ways more often than you’d think. So going “safety first” every time likely critically diminishes UX.
The issue is not recognising that optimising for Ux at the expense of safety here is the wrong call, motivated likely by optimism and a desire for autonomous cars, more than reasonable system design. I.e. if the sensors disagree so often that it makes the system unusable, maybe the solution is “we’re not ready for this kind of technology and we should slow down” rather than “let’s figure out non-UX breaking edge case heuristics to maintain the illusion of autonomous driving being behind the corner”.
Part of this problem is not even technological - human drivers tradeoff safety for UX all the time - so the expectation for self driving is unrealistic and your system has to have the ethically unacceptable system configuration in order to have any chance of competing.
Which is why - in my mind - it’s a fools endeavour in personal car space, but not in public transport space. So go waymo, boo tesla.
Putting that to the side - i don’t think I’ll read this fully soon, but the core thesis of “imitation is intelligence” can be easily disproven by a process that exists in society. An actor acting to be a genius is in fact, if they are a good actor, indistinguishable in their appearance to a genius. Yet they are not, in fact, a genius, they’re just good at memorisation. This is a clear showcase that imitation of a level of intelligence does not mean that this level of intelligence is present.
We have fallen into a trap of thinking that answering in plausible sentences is what makes humans intelligent. While in reality we are observing an actor responding from an infinitely large script. What makes humans intelligent (reasoning from first principles and pattern recognition across all the sensory inputs of the world) is still very much out of grasp.