Thanks for writing this so clearly... I hear wrong/misguided arguments like we see hear every day from friends, colleagues, "experts in the media" etc.
It's strange because just a moment of thinking will show that such ideas are wrong or paint a clearly incomplete picture. And there's plenty of analogies to the dangers of such reductionism. It should be obviously wrong to anyone who has at least tried ChatGPT.
My only explanation is that a denial mechanism must be at play. It simply feels more comfortable to diminish LLM capabilities and/or feel that you understand them from reading a Medium article on transformer-network, than to consider the consequences in terms of the inner black-box nature.
You write: "it's just that at every step the alligator is lurking in the logits because it directly derives from the prompt" - but isn't that the whole point: at the moment the model writes "an", it isn't just spitting out a random article (or a 50/50 distribution of articles or other words for that matter); rather, "an" gets a high probability because the model internally knows that "alligator" is the correct thing after that. While it can only emit one token in this step, it will emit "an" to make it consistent with its alligator knowledge "lurking". And btw while not even directly relevant, the word alligator isn't in the prompt. Sure, it derives from the prompt but so does every an LLM generates, and same for any other AI mechanism for generating answers.
> While it can only emit one token in this step, it will emit "an" to make it consistent with its alligator knowledge "lurking".
It will also emit "a" from time to time without issue though, but will never spit "alligator" right after that, that's it.
> Sure, it derives from the prompt but so does every an LLM generates, and same for any other AI mechanism for generating answers.
Not really, because of the autoregressive nature of LLMs, the longer the response the more it will depend on its own response rather than the prompt. That's why you can see totally opposite response from LLM to the same query if you aren't asking basic factual questions. I saw a tool on reddit a few month ago that allowed you to see which words in the generation where the most “opinionated” (where the sampler had to chose between alternative words that were close in probability) and where it was easy to see that you could dramatically affect the result by just changing certain words.
> "an" gets a high probability because the model internally knows that "alligator" is the correct thing after that.
This is true, though it only works with this kind of prompt because the output of the LLM has little impact on the generation.
Globally I see what you mean, and I don't disagree with you, but at the same time, I think that saying that LLMs have a sense of anticipating the further token misses their ability to get driven astray by their own output: they have some information that will affect further tokens but any token that get spit can, and will, change that information in a way that can dramatically change the “plans”. And that's why I think using trivial questions isn't a good illustration, because it pushes this effect under the rug.
I don't understand how OpenAI claims it would have happened. The weights are closed and as far as I read they are not complaining Deepseek hacked them and obtained the weight. So all they could do was to query OpenAI and generate test data. But how much did they query really - I would suppose it would require a huge amount done via an external, paid-for API? Is there any proof of this besides OpenAI saying it? Even if we suppose it is true, I suppose this must have happened via the API so they paid per token etc. So they paid for each and every token of training data. As I understand, the requester owns the copyright on what is generated by OpenAI's models and is free to do what they want.
You have to give it a chance. He is first building up an argument about why consciousness cannot depend just on the physical substrate itself but rather on the "interpretation" of this. It is very important to understand this part/argument. What follows is something that resonates with you namely how our consciousness is now 'tuned' to the current physical laws.
I'm wondering how deep the hack is... it seems sending a frame is just setting some registers and waiting for an interrupt. This suggests (though I'm not an expert!) that they are talking to another layer of firmware that does the actual stuff? Reminds me a bit of the Raspberry pico board which has the main RP2040 SoC but where the WiFi is a separate WiFi/BT module (CYW43xx) with its own Arm cores. Not even the external register interface to the WiFi module is documented publicly, but there is an open source driver (https://github.com/georgerobotics/cyw43-driver/tree/cf924bb0...) so one can infer the specification. However, this driver yet again talks to software running on Arm codes inside the module, the code for which is supplied as big firmware binary blobs by the manufacturer (the blobs are actually in the linked repo, defined inside header files in the firmware directory). I'm wondering how this ESP32 hack corresponds to this?
It's strange because just a moment of thinking will show that such ideas are wrong or paint a clearly incomplete picture. And there's plenty of analogies to the dangers of such reductionism. It should be obviously wrong to anyone who has at least tried ChatGPT.
My only explanation is that a denial mechanism must be at play. It simply feels more comfortable to diminish LLM capabilities and/or feel that you understand them from reading a Medium article on transformer-network, than to consider the consequences in terms of the inner black-box nature.