shelf life of autoregressive LLMs
"I have a bold prediction which is that the shelf life of autoregressive LLMs is very short. My prediction is that five years from now nobody in their right mind would use them. So enjoy it while it lasts. They'll be replaced by things that are better." - Yann LeCun, Chief AI Scientist at Meta AI (FAIR)
It feels like current hype around Generative AI is rushing the world to over-rely on an incredibly limited AI architecture. Essentially the longer your conversation (with AI) is, "you sort of diverge away from the set of correct answers exponentially" (16:54).
He also points out that we are easily fooled by their fluency, but LLMs don't actually know how the world works.
Auto-Regressive LLMs are good for (16:03):
- Writing assistance, first draft generation, stylistic polishing.
- Code writing assistance
What they are NOT good for (16:08):
- Producing factual and consistent answers (hallucinations)
- Taking into account recent information (anything after the last training)
- Behaving properly (they mimic behaviors from the training set)
- Reasoning, planning, math
- Using "tools", such as search engines, calculators, database queries...
So please use AI responsibly! Every business integrating AI into their workflows and every developer building on these LLMs need to be fully transparent and honest to themselves and their users/customers about its limitations and risks.
He DOES propose a Modular Architecture for Autonomous AI, which would have an actual "understanding" of the world and theoretically allow us to hard code certain guiding values and fail safes. But it appears that we're still perhaps 5-10 years away from seeing this in action.
So while the rest of the world is still adapting to the impact of the ChatGPT meteor, let's not give LLMs more credit than its due, nor lose sight of the even bigger meteor to come.