Most of us are familiar with the humble photocopier. Introduced by Xerox as a product in 1949, the “Xerox Machine” was the high-tech of its day. Xerox went on to fund it’s Palo Alto Research Center (PARC), which gave rise to the mouse, the windowed operating system, object oriented computer, and Ethernet. Xerox copies have one specific weakness – they are imitations of the original. Make a copy of a copy of a copy, and, eventually, you lose the quality of the original. This experience of copying copies is familiar to those of us of a certain age, but camera photos of camera photos have a similar effect.
The paper “Levels of Organization in General Intelligence“, which appears in the book “Real AI: New Approaches to Artificial General Intelligence”, published around 2002, may contain the first use of the term “Recursive Self-Improvement”, or RSI. At the time, RSI meant AI training itself – getting better and better over time. This is sort of the opposite of the photocopy problem, where the machine spits out better copies over time, sort of like the classic use of “Enhance” in government, military, and police drama TV shows.
This makes for good tv, but is very limited in practice. The questions are: What does that really mean for AI, is it practical, and to the extent it is possible, what does that really mean for us – what should we do?
RSI Redefined
Fundamentally “Levels of Organization” was a “Philosophy of AI” paper. The author of the paper, Eliezer Yudkowsky, the author of the paper, did not lay out exactly how the AI improve itself. At least, he did not lay anything out beyond metaphorical comparisons to optimizing compilers, or the superiority of the linked list to the array as a way to implement a stack. According to Yudkowsky, the machine itself would go on to create better and better algorithms. Bengio and others were looking at combining neural networks and statistical word processing as early as 2000, but Yudkowsky has said when he came up with the term, he was NOT thinking about neural networks, but instead cognitive science, neuroanatomy, and decision making. Bengio’s work is not cited in the 100-odd references at the end of “Levels of Organization” paper. It wasn’t until Word2Vec, in 2013, that computer science was able to make text as vectors (distance and direction, linear algebra in N-space) efficient enough to do real work. Word2Vec didn’t even use graphics processors (GPUs) with their specialty in calculating multi-dimensional space; that would become popular just a year later. In 2017, Google finally published a paper that suggested the transformer architecture that all LLM’s use today[1].
Short version: The RSI that Yudkowsky was talking about was related only in the vaguest of philosophical ways to the actual LLMs we are working with today. Yudkowsky’s example was more like what James Bach was mocking in his 2007 blog post “The future will need us to reboot it.” RSI is not really recursive, either, as a recursive function calls itself over and over, eventually “backing out” and returning a result. Here is a classic recursive function in Python:
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)RSI isn’t really calling itself recursively, it is just improving itself over and over, with the new version taking over.
Lately, when I hear people talk about RSI for LLM’s, they mean the AI working as its own continuous integration framework, setting up the variables, gathering the data, running the processing and training the new model.
RSI: Big Lie or Just Wrong?
The first problem with RSI as “just training the model” is it is misleading. It currently takes several hundred million of dollars to train an AI frontier model. Each increasing generation drastically increases processing and electrical cost while having diminishing returns. As a result, the largest frontier runs are predicted to cost over a a billion dollars each in 2026[2]. A peer-reviewed research paper for Transactions of the Association for Computational Linguistics recently ran the numbers. The paper calculated this as not a coincidence, but a foregone conclusion, as LLM transformer technology has decreasing returns, and calculated hard limits. You can always push a hard limit; 1.99 is closer to 2 than 1.9, and 1.999 is closer still. Yet, for all intents and purposes, at some point, you are so close that it does not make a difference.
So yes, you could have a LLM create a more-expensive LLM that takes longer to train and is incrementally better, approaching some hard limit. I suppose. But that isn’t what people mean when they hear “Recursive Self Improvement.” Using a word and knowing people will take as one definition, but secretly having a separate, more defensible definition, is called the logical fallacy of equivocation. Equivocation to investors as a recurring pattern is part of why Elizabeth Holmes is currently in prison. In other words, equivocation is a very fancy lie.
Except of course, it could be worse.
LLM’s are trained on data. We put humans in the loop to train the neural nets, to look at output and say “that’s not quite right.” Without the humans int the loop, how would the LLM decide if the output is correct? It is possible to have the AI use other tools or generate its own checks, but then we have the photocopy problem. Another term for the photocopy problem is model collapse, and it is already demonstrated and documented. Axios is saying that about half the content produced for the web today is AI/LLM/Generated. If input to your LLM contains LLM content, and we eliminate the humans in the loop, and the current LLM content is below the standard for humans, then we get garbage-in, garbage-out. This is the photocopy problem all over again, where copies of copies lose fidelity with each successive generation. While there is some hope in specific small domains, the bull case was for Autonomous General Intelligence (AGI). Right now, we don’t even see that in text-only models!
This makes the current approaches to RSI, where we redefine RSI as “Continuous Integration Builds for AI” not just an equivocation, but fundamentally flawed.
Of course, I could be wrong. Perhaps there is some new breakthrough technology that makes our understanding of limits and costs obsolete. Right now, what I can say is that I just do not see the evidence for these claims, and extraordinary claims, the kind that has the world investing $1 Trillion Dollars in AI just this year, require extraordinary evidence, I’m just not seeing it.
Why am I, the oddball tester who marches to the beat of a different drummer, the person to figure this out? Shouldn’t this be more obvious?
Which does bring me to one bit of hope.
Moving Forward
Lossy and imperfect as they are, you’ve got to admit the LLM tools and agents add a lot of new capabilities. It is true that five years ago, writing even a simple mobile app for the phone required a significant amount of effort, one many people were unwilling to work through. Over the years I’ve probably taught a dozen sections of CS-360 at Southern New Hampshire University, Mobile App Development. When you look at the effort students have to get through to create the capstone project, and how small that project really is, it is enough to make you … sad. Tools like Claude can make mobile programming easy in ways not-that-different than Java made programming easier than hand-coding hexadecimal. Exactly what domains these new tools will be valuable in, how good they will get, how fast they will get good, what value they will have once they are no longer subsidized, these are all interesting questions.
Assuming this near era of programming means something, the skills may have shifted a bit. It used to be the hard technical expert who could learn a new programming language in minutes was the core accelerator of the team. Many of us experienced a different environment, pre-Agile perhaps, when the technical experts were stuck because the requirements were so poor, the bottleneck was in systems analysis, not coding. Tomorrow may see a resurgence of the importance of logic, rhetoric, subject matter expertise, math, and a little bit of tech. Perhaps – just perhaps – there was a reason I’m the one to be figuring this out. Perhaps we are in some sort of Asch conformance experiment and I’m socially awkward enough to call out the emperor’s lack of clothes. It is hard to tell.
But finally, it is worth saying …
Earlier I said RSI was not truly recursive, in the sense that it is not a function calling itself over and over, then backing out to form some sort of total. If you do too much recursive programming, you eventually run out of memory, and get something called a “stack overflow error.” Traditionally, with a stack overflow, your program blows up and ends. That’s it. If you are lucky you get useful error messages. Later languages like Java provide catch, so you can exit out of the routine but not break the program entirely. Still, you’ll lose all your work.
Maybe [attempts at] recursive self improvement isn’t a terrible title, after all. We just got the reason why wrong.
Endnotes
[1] There is one name that keeps popping up in these papers, and it is “Jeff Dean“; he might be one to watch and take seriously.
[2] Part of why this paper is so long is I kept running into somewhat-unreasonable misunderstandings of what I was writing. For example, I changed that model collapse was “demonstrated and documented” instead of “already happening”, because a somewhat-unreasonable person could claim it is not demonstrated to be happening in the current frontier models. But of course it isn’t, as they still have human in the loop feedback and oversight. My first draft said “Researchers at MIT”, but, of course, the publication is by MIT Press, not employees of MIT. So I put “Transactions of the Association for Computational Linguistics” in the piece. All this made the paper longer and longer. Eventually I decided it was good enough. There is tension in writing between comprehensive and comprehensible. I strove for technical accuracy without equivocation here, but I have my own hard limits for what the reader would tolerate. If you think I am “wrong” about a “fact”, I’d politely ask you to consider the possibility that English is vague and ambiguous, and is there another reasonable interpretation that does not have those problems? I think 1,600-odd words is about the limit of my readers, so I have to stop now.