I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.
I repeat, the LLM is not doing machine learning while users are using it.
We agree here.
And we agree here too, but to trust an LLM to tell you the truth on your question that you don’t know the answer is like trusting some random drunk at the pub, because you don’t know whether the answer is from an LLM hallucination, a random lie/error on reddit or an expert’s contribution to wikipedia.
And to trust an LLM when there’s a trained programmer or professional journalist is stupid. Sure, an LLM might even sometimes write as good or better code than an intern, but again, the LLM is not learning from its mistakes as you correct it. The intern gradually becomes an expert. The LLM does not. Paying interns is an investment in future programmers, who get more expensive the more experienced they are.
The LLM is currently cheaper than the intern, but LLM pricing needs to go up by a factor of about ten to cover running costs let alone pay off the vastly more immense debts of buying all that hardware.
Like I said before, humans sleep every night, with rare exceptions. LLMs do not get retrained every night. The human brain adapts to feedback loops during everyday interactions, not just overnight. It’s a silly analogy and this is a silly point to defend.
There are plenty of textbooks that say that volatile running RAM is like short term memory and hard disks and SSDs are like long term memory, but it would be silly to reverse the analogy as you are doing and claim that sleep is pressing the save button on the day’s learning, or that this makes your word processor the same as your human intelligence because, and this is the central point you’ve been trying to argue around and about and against, they’re doing fundamentally different things, and telling me one was inspired by the other doesn’t change that. An LLM is fundamentally a stochastic regurgitator whose training is designed primarily to make it sound right. A human brain just doesn’t work that way.
If you truly believe that the LLM is learning like a human or intelligent like a human, you are confusing analogies for reality.
This is a small terminology misconception. The LLM is not doing “training” during inference. It’s still a “machine learning” system.
In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That’s why some systems seem to be able to “remember” things from previous “conversations”. Your message is vectorised and then that vector used to look up similar past interactions. The model isn’t fine tuning on that, so it’s not “long term” memory, but the model can take it into account for future interactions.
AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.
They’re a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn’t be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it’s dangerous to underestimate them.
Depends which drink guy at the pub you randomly pick. The attribute that they share with the drink guy at the pub is their reluctance to admit that they don’t know or have no expertise or can’t help you. Clever and experienced people know where their expertise ends and express self doubt when appropriate. LLMs don’t. They can’t. They’re making literally everything they say up. It’s probably right, but they are the script kiddie of conversationalists.
It’s dangerous to underestimate their ability to sound good enough to convince executives to fire humans. It’s dangerous to underestimate the scale of substitution of plausibility over knowledge that will only accelerate with further adoption. It’s dangerous to assume that the interactions that middle and senior management have with staff that do actual work cannot be replicated already with a suitably trained LLM.
Those slashes are doing a lot of work in that sentence!
Humans learn by generalising from examples. Humans learn when you ask them well-designed questions. Humans learn by practising skills repeatedly. Holland park from their mistakes. Humans learn because they are in a constant state of feedback loop. Humans learn by watching other people. Humans learn by experimentation. Humans learn through playing with new things. Humans learn by talking to each other. Humans learn by sitting and thinking things through. Humans learn through thought experiments. Humans learn through seeing and hearing and reading more quickly than doing any of them alone. Humans learn by explaining things to other people, crystallising their experience into verbal solidity. Humans learn through discussion. Humans learn by learning who to trust and how to weight different input by source. Humans learn by learning how to learn more effectively.
LLMs do none of any of those things.
Machine learning is a very, very narrow form of “learning” and you’re conflating the use of a neural network with actual learning, which you then compound by confusing the resulting LLM with the neural network that was used in its creation.
Pulling the wool over people’s eyes about what an LLM is is at least as harmful as underestimating the ability of AI to be so plausible as to disrupt absolutely everything about how money moves around society.