he wants other AI snuffed out so anthropoic remains one the only ones around, thats what he really meant, additionally trying to buy time how to leave the AI bubble unscathed when it bursts.
In a blog post published Thursday, the company cited its own internal data as evidence that modern AI systems are nearing the point of “recursive self-improvement”—i.e., being able to refine their capabilities without a human in the loop. “AI that can build itself would be a major development in the history of technology—one that could bring enormous good for the world in science, healthcare, and beyond,” the post, which was written by company cofounder Jack Clark and Anthropic Institute lead Marina Favaro, reads. “But full recursive self-improvement also might increase the risks of humans losing control over AI systems.”
Are they? When our own researchers barely have a grasp on how LLMs work, can’t really pinpoint why LLMs make certain choices, can you really expect it to improve itself when it has no sense of understanding? Or will it just make it self even more convoluted and nonsensical until it all falls apart like a house of cards?
I don’t know shit about LLMs, so maybe I’m lacking understanding here.
If researchers barely have grasp on how LLMs work how did they create Claude?
They created the model and trained it, but they don’t know why it gives what it gives when you ask it a question. Which is why they still haven’t solved the hallucination issue.
They understand the theories and underlying principles, but the sheer amount of data makes it impossible to actually verify it.
An ELI5 comparison would be a hill of stones: you know when you throw more stones onto it, a “landslide” will occur and rearrange the hill. For a very small hill of 10 stones you may even be able to know input and output (“if I throw a stone there, the stones will be like this after the landslide”). But you cannot predict the same for a hill of 1000000 stones, even tho the “rules” are the same. You know what will happen, but you have no way to predict the outcome, or verify that everything went as expected.
The theory / math is not the problem. The scale is.
This is their usual marketing scheme - “it’s too powerful to release!”
it means thier underlying business strategy is a failure, another one of my favorite is “we grown too big or famous we have to close down some of our services”
Also the same thing that they do when they suddenly realize that they are about to be obsolete.
Once AIs can do a better job of creating themselves than AI companies, then there’s not much need for AI companies anymore is there?
Market leader wants to freeze the market.
While I’m strongly opposed to the “move fast break things” mindset, this is not an altruistic or even benevolent move by Anthropic.
its to buy time so they can try to snuff other AIs, or determine how to escape the bubble bursting.
They’re trying hard to look like “good guys” ahead of their IPO.
Everybody heard their recent breakup with Pentagon but ignoring they were happily engaged with the War Machine before the breakup. They’re still serving Palantir, Blackrock, NSA and god knows who else.
I’ll take them seriously when they take their own advice seriously and pause their operations.
Here’s the thing. We should absolutely not trust any companies with this much power about anything they say, buuuuuuut to play the devil’s advocate here, let’s pretend they are absolutely altruistic.
They are still a company and must compete. Their market edge isn’t dominant enough that they could really take any moral stand like “stop research and development” when there are three other companies who will gladly step into the market leader position, so even if they do believe their message, without any assurances that everyone will agree to pause, they’d just be letting someone else take over their spot for a moral stance for an arguably worse situation.
And realistically, if they weren’t in the lead, people would be saying “loser wants leaders to slow down lol” and if they’re in the lead people will say, “lol market leader wants competition to stop, I wonder why?”. There is really no position they can be in and make the claim without catching flak for it. A year or two ago Anthropic made a similar blog when they weren’t in such a dominant position and comments were exactly like that.
All that said, obviously they’re nearly a trillion dollar company and we shouldn’t take anything they say at face value, but I really do think the message here isn’t terrible (like the article says). We really should be slowing down and looking at the effects of AI. It has too much potential to fuck everything up to not think about it a bit, especially if recursive self-improvement is on the horizon.
We really should be slowing down and looking at the effects of AI. It has too much potential to fuck everything up to not think about it a bit, especially if recursive self-improvement is on the horizon.
IMO, the notion that AI systems are intelligent in any meaningful capacity is venture capitalist marketing. AI systems are especially capable of pooping out convincing bullshit. Otherwise, these systems are not particularly more dangerous than other machines. In particular, the notion that we are even converging towards AGI is marketing. The focus of criticism should be on the concrete applications of these machines. IMO, statistical learning theory has excellent theoretical backing and lots of practical applications, and the mathematics is honest (if complicated) about the limitations of the field. The statistical learning community is still working on developing the theoretical backbone for LLM systems at scale.
IMO, the main issue is that capitalists are wrecking our environment, stealing all our private information, and stealing our tax dollars to build systems that are then used to kill, oppress, harass, demean, and propagandize the working class. Again, AI technologies are especially capable at helping the worst monsters in all of history achieve these goals. But it’s not because of “intelligence”. It’s because of what they are using these systems to do to us.
I.e., we should absolutely scale down AI system deployment to a sustainable level, and we should burn anyone who uses AI systems for oppression … but people who are organically interested in statistical learning systems should at least be allowed to continue research into the mathematics of “AI” (preferably under a different title, because people who end up using AI do not need nor want intelligent systems, just useful ones!) as long as they’re not doing anything with the technology to oppress people.
We really should be slowing down and looking at the effects of AI. It has too much potential to fuck everything up to not think about it a bit, especially if recursive self-improvement is on the horizon.
This is Anthropic’s entire marketing strategy: get people thinking that LLMs can do more than they actually can through fear. This is what they constantly do and have done for at least the last two years. They also tend to stretch or bury the truth in their “studies” with these goals in mind.
Fear, especially of the unknown, is very powerful. If you fear it the way they want you to, that feeling will convince you that the LLMs are going to take over any day now, and once enough people think that, it pumps Anthropic’s valuation.
I do agree that we should slow down, but not necessarily because of any recursion (they claim they can already do this with “agents”, another term they stole to make LLMs seem more capable). I think we should stop because it’s a huge bubble, and the more we pump into it the louder the pop will be. Not only that, but is a monumental waste of resouces, all to generate half-baked slop most of the time.
I’m not even talking exclusively about capabilities, I’m talking about the effects of using LLMs on people. Cases of psychosis and the effects on the newer generation in their education, the atrophy of critical thinking skills, the ability to subtly influence public opinion, those are things we should be thinking about even if you don’t think that LLMs are capable of doing anything.
Yeah, I definitely include that in my reasoning why it should stop. Anthropic is not, unfortunately.
Anthropic once again trying to brand themselves as the good guys, while actively cooperating with the NSA.
They ran out of unsuspecting VCs and need time to find fresh ones, the Money Furnace demands more sacrifices!
