ChatGPT has been taught about sailfish - it has had access to a corpus of several centuries worth of knowledge regarding sailfish - and it will try to persuade you that it is a mammal, making up some outlandish facts along the way.

Yea, I lean on "at present" because the question in the paper is whether LLMs (GPT 3.5 specifically) can reason about medicine now.