AI researchers and CS people and the rest of us are human brain users and so have some familiarity with them even if they haven't studied neuroscience.

You can make some comparisons between how they perform without really understanding how LLMs or brains work, like to me LLMs seem similar to the part human minds where you say stuff without thinking about it. But you never really get an LLM saying I was thinking about that stuff and figured this bit was wrong, because they don't really have that capability.