Interesting read and feels intuitively plausible. Also matches my growing personal sense that people are using these things wildly differently and having completely different outcomes as a result. Some other random disconnected thoughts:
I’m surprised they’re publishing this, it seems to me like a pretty stark condemnation of the technology. Like what are the benefits they anticipate that made them decide this should be published, vs. quietly kept aside “pending further research”? Obviously people knowing how to use the tools better is good for longevity, but that’s just not what our idiotic investment cycles prioritize.
I’m no scientist or expert in experimental design, but this seems like way too few people for the level of detail they’re bringing to the conclusions they’re drawing. That plus the way it all just feels intuitively plausible has a very “just so” feeling to the interpretation rather than true exploration. I mean, cmon - the behavioral buckets they are talking about range from 2-7 people apiece, most commonly just 4 individuals. “Four junior engineers behaved kinda like this and had that average outcome” MIGHT reflect a broader pattern but it sure doesn’t feel compelling or scientific.
Nonetheless I selfishly enjoyed having my own vague subconscious observations validated lol, would like to see more of this (and anything else that seems to work against the crazy bubble being inflated).
For 1: as a software company, they have a vested interest in ensuring that software engineers are as capable as possible. I don’t know if anthropic as a company uses this as a guiding principle, but certainly some companies do (ex Jane Street). So they might see this as more important than investment cycles.
The quality of software engineers and computer scientists I’ve seen coming out of undergraduate programs in the last year has been astonishingly poor compared to 2-3 years ago. I think it’s almost guaranteed that the larger companies have also noticed this.
I completely agree and appreciate sincerely that they released this. It’s unfortunate, the way the obviously nonsense claims made by the industry at large - “LLMs are AI and can do everything!” - have polluted a lot of devs’ ability or willingness to see the tools for what they are, and maybe official acknowledgements like these can help.
It also seems likely to me that the major players know a lot of negative truths about all this stuff, you’re probably right about hiring observations. I don’t follow any of their marketing really so I have to admit I’m even just assuming that releasing this is out of character.
If I’m being honest, I’m mostly just on the edge of my seat waiting for the hype bubble to burst, lol, and curious about how it’ll unfold. Probably just kind of hoping this marks a step toward that.
Interesting read and feels intuitively plausible. Also matches my growing personal sense that people are using these things wildly differently and having completely different outcomes as a result. Some other random disconnected thoughts:
I’m surprised they’re publishing this, it seems to me like a pretty stark condemnation of the technology. Like what are the benefits they anticipate that made them decide this should be published, vs. quietly kept aside “pending further research”? Obviously people knowing how to use the tools better is good for longevity, but that’s just not what our idiotic investment cycles prioritize.
I’m no scientist or expert in experimental design, but this seems like way too few people for the level of detail they’re bringing to the conclusions they’re drawing. That plus the way it all just feels intuitively plausible has a very “just so” feeling to the interpretation rather than true exploration. I mean, cmon - the behavioral buckets they are talking about range from 2-7 people apiece, most commonly just 4 individuals. “Four junior engineers behaved kinda like this and had that average outcome” MIGHT reflect a broader pattern but it sure doesn’t feel compelling or scientific.
Nonetheless I selfishly enjoyed having my own vague subconscious observations validated lol, would like to see more of this (and anything else that seems to work against the crazy bubble being inflated).
For 1: as a software company, they have a vested interest in ensuring that software engineers are as capable as possible. I don’t know if anthropic as a company uses this as a guiding principle, but certainly some companies do (ex Jane Street). So they might see this as more important than investment cycles.
The quality of software engineers and computer scientists I’ve seen coming out of undergraduate programs in the last year has been astonishingly poor compared to 2-3 years ago. I think it’s almost guaranteed that the larger companies have also noticed this.
I completely agree and appreciate sincerely that they released this. It’s unfortunate, the way the obviously nonsense claims made by the industry at large - “LLMs are AI and can do everything!” - have polluted a lot of devs’ ability or willingness to see the tools for what they are, and maybe official acknowledgements like these can help.
It also seems likely to me that the major players know a lot of negative truths about all this stuff, you’re probably right about hiring observations. I don’t follow any of their marketing really so I have to admit I’m even just assuming that releasing this is out of character.
If I’m being honest, I’m mostly just on the edge of my seat waiting for the hype bubble to burst, lol, and curious about how it’ll unfold. Probably just kind of hoping this marks a step toward that.