That sounds like a similar failure rate for using newer models in technical contexts. Especially for popular technologies where there are enough resources such that someone other than the original creators could reasonably write a book about it.

I think that's an oversimplified metric when comparing the ability of an AI to analyze and describe code. Imaging you're reading a book on a challenging new topic. Even if the book is 100% right, there will be many passages you don't understand, and may even misinterpret. A book won't correct you, offer alternate explanations, or break down concepts any further than what was already written.

An LLM, for all its flaws, can do these things exceedingly well (provided it has good source material, of course). This gives you the ability to get answers to your questions and correct your mistakes. I'd argue that these benefits far outweigh the errors an LLM makes in breaking down and explaining material, and make an LLM better than a book in many (not all) cases.

> Even if the book is 100% right, there will be many passages you don't understand, and may even misinterpret.

Which is why no one reads only one book on a particular subject, especially if one is new to the domain. Often you bring in an alternate, but correct perspective from another book. The risks of hallucination with LLMs' responses outweighs the advantage of multiple perspectives (even if a broken clock is right twice a day, you still want a working one)

No one reads only one book on a subject? People act on mere headlines all the time. Reading one whole book would be a huge win over the status quo

> The risks of hallucination with LLMs' responses outweighs the advantage of multiple perspectives

You can ask a model a question multiple times. You can ask different models the same question. You can ask the same question different ways. You can do this automatically with agentic frameworks or reasoning models. "Multiple perspectives" are compatible with language models.

> You can ask a model a question multiple times. You can ask different models the same question. You can ask the same question different ways. You can do this automatically with agentic frameworks or reasoning models.

LLMs are for generating text, not producing knowledge. The actual knowledge come from the signal/noise ratio of the training data, not the training method.

It's like recording a violin in a noisy environment, encoding it with a low-bitrate lossy encoder and trying to get the PCM of only the violin back.

We've tried to use a better environment for recording specific notes from the violin (better training data), adjust the parameter of the lossy encoder (tuning the model), mix other violin samples to the respone (RAG, MCP), and redefining what a violin should sound like (LLMs marketing).

But the simpler solution is to record it correctly and play it later (AKA going to the source).

I would guess it's been around 1-2 years since you last tried seriously using a language model for any practical use cases, right? It makes sense to not use them if that is your experience, but it's hard to imagine anyone saying this who has actually been following along for the last few years to now. SOTA from 2 years ago now comfortably runs on a single GPU. A high end GPU can easily out perform SOTA from then both in (general!) knowledge reproduction, and capabilities (including saying things like "I don't know" when you ask beyond the encoded knowledge).