If I said Model Autophagy Disorder (M.A.D.), you’d be forgiven for guessing its meaning lies with the biomedical community.
Not so.
This catchy three letter acronym resides firmly with the Artificial Intelligence community and potentially represents a large problem for the Large Language Models (LLM’s) we use today.
It goes something like this:
Large Language Models (LLM’s), are trained on large chunks of language, usually, the web. When AI’s generate web content, as they are increasingly doing, that AI content is used to train a subsequent model. The new model gets worse, failing completely back to gibberish in only a few iterations.
This is not speculation, it’s been demonstrated experimentally [1.] [2.]
I’ve been “working the web” as they say since 1997 and one of my consistent fascinations has been the miraculous self generation of Wikipedia, by humans. That humanity would voluntarily write, curate and maintain such a monumental and useful opus without payment was not at all obvious in 1999. Quite the reverse.
And yet it happened.
That early, pre 2006 (web 1.0 , ugh!) was where the real content creation heavy lifting emerged. Few bots, low or no automation. No or difficult Content Management. There was still a sufficient barrier to entry to focus the mind and think before publishing. Phones were still phones.
Cast your mind forward.
The next few years will see exponential growth in LLM generated web content (No LLMs were sued in writing this, I swear!), and this content is already feeding quietly and silently into the LLM’s we will be using tomorrow.
Autophagy is the act of self consumption.
AI will increasingly eat itself.
By Phil Blything
