I don't even like the word "hallucination", because it's an anthropomorphism, further suggesting/misleading into an impression that LLMs have anything to do with how a human would reason.
Fancy autocomplete. Useful, yes. Powerful, definitely. But it's just another tool, limited in applications the same way a hammer is - this is the part that many humans seem to struggle accepting.
Philosophy of language, and Wittgenstein in particular, has more specific words for the output insofar as they mean anything to humans: "senseless" or "nonsense".
But I don't think those terms are accurate. If I say "John Doe was born in 1987", that's a perfectly sensible sentence, even if I don't actually know when he was born and even if it turns out he was actually born in 1971. A nonsense sentence would be something like "John Doe cat five green explicate" or "h4stga3jkui7rutjdyrst".
You've just re-presented one difference between sense and nonsense in philosophy of language. We might call that first phrase wrong or an error or inaccurate if that was the case. We don't need to use a word like "hallucinate" for your second phrase --- we can just call it nonsense.
"Hallucinate" is far more inaccurate and confusing. It paints a false picture that melds 2 distinct things, as you've described, which need not happen at the same time: errors and nonsense.
I think the term forst originated with the dream-like images of google deepmind, these were really similar to human visual hallucinations, i guess the term stuck
When a human hallucinates, that is due to the brain operating out of spec. When an LLM "hallucinates", it's performing exactly as intended, it just has delivered a nonsensical result which is the consequence of the tech behind the LLM.
Instead of "hallucinating" I would have preferred the term "bullshitting" -- in the Harry G. Frankfurt sense of not caring about the truth of one's utterances. But it's too late for that.
Using "bullshit" would be interesting, but to me would introduce a backdoor anthropomorphism to describe the output. The picture is still too human.
Isn't Frankfurt's concept of bullshit made up of 2 parts: 1) a distinction between lying and telling the truth, AND 2) the absence of caring about either when speaking when normally it's assumed present?
Part 1 seems to apply, but part 2 wouldn't. It doesn't make sense to talk about GPT "caring" about its output beyond anthropomorphism. No one talks about their computer caring about having correct or accurate output and neither is it assumed. People would think your imagining a demon in the box. Really, it's even odd to say "GPT lied" outside of very specific circumstances.
I think "bullshitting" fits better than "hallucinating" - just keep spitting out words rather than admit ignorance, but maybe the best human analogy is freestyle rapping where one has to keep the flow of words coming regardless!
Maybe we just need to coin a new word for it - "LLM-ing" perhaps ?!
Fancy autocomplete. Useful, yes. Powerful, definitely. But it's just another tool, limited in applications the same way a hammer is - this is the part that many humans seem to struggle accepting.