Do LLMs actually know anything?

For example, can an LLM know if dogs are hairy without knowing what dogs are or what hairiness means?

Altman and Sutskever are amusingly coy on the subject.

“Well, they certainly learnt something” - Sam Altman

“Saying what understanding means is hard, and so we measure prediction instead” - Ilya Sutskever

This is part of “intelligence equals prediction”, which maybe explains how OpenAI thought of all this in the first place.

Representing ideas

What did Sutskever mean when he said “an LLM is great for learning about the world (including representations of ideas), but not so good for producing output”?

Here is Gemini’s answer.

Transformer Multitasking

Here is another Sutskever quote:

during training the transformer is forced to multitask a huge number of tasks

And here are Gemini’s thoughts on what he meant.