I see this as a temporary problem. A human brain can be trained on way less than the entire corpus of everything humans have ever written. Ultimately, this will apply to LLMs (or whatever succeeds them) too.
There's another aspect to this too: China and DeepSeek. While this was released by a private company, I think there's a not-insigificant chance that it reflects Chinese government policy to "commoditize your complements" [1]. Companies like OPenAI want to hide their secret sauce so it can't be produced. Training an LLM is expensive. If there are high-quality LLMs out there for free you can just download, then this moat completely evaporates.
There's another aspect to this too: China and DeepSeek. While this was released by a private company, I think there's a not-insigificant chance that it reflects Chinese government policy to "commoditize your complements" [1]. Companies like OPenAI want to hide their secret sauce so it can't be produced. Training an LLM is expensive. If there are high-quality LLMs out there for free you can just download, then this moat completely evaporates.
[1]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/