In the 1980s, economist Robert Solow made an observation that reminded economists of today’s AI boom: “You can see the computer age everywhere but in the productivity statistics.”
I feel like the big mistake they continue to propagate is failing to distinguish among the uses of AI.
A lot of hype seems to be the generative uses, where AI creates code, images, text, or whatever, or the agentic uses where it supposedly automates some process. Safe uses in that way should involve human review and approval, and if the human spends as much time reviewing as they would creating it in the first place, then there’s a productivity loss.
All the positive cases I’ve heard of use AI like a fancy search engine - look for specific issues in a large code base, look for internal consistency in large document or document sets. That form lets the human shift from reading hundreds or thousands of pages to reading whatever snippets the AI returns. Even if that’s a lot of false positives, it’s still a big savings over full review. And as long as the AI’s false-negative rate is better than the human, it’s a net improvement in review.
And, of course, there’s the possibility that AI facilitated review allows companies to do review of documents that they would otherwise have ignored as intractable, which would also show up as reduced productivity.
I’ve used AI plenty of times to help troubleshoot some weird error message. Sometimes just an old-fashioned Google just isn’t enough. There needs to be added context, which would just screw up the Google results.
I treat talking to AI for advice (in any category) roughly the same as asking an IRC channel…because that’s basically what it is. It’s taking in data from tons of sources and summarizing it.
Some of those sources might be legitimate and knowledgeable, some of them might be a negative-scored stack overflow comment.
If you have no domain-specific knowledge, you won’t know how to identify an issue in its response, and you shouldn’t be blindly copying code. Trust…but verify.
In that scenario where AI is used to find specific code snippets or other matching text blocks, the false positives aren’t really the problem. The false negatives are the issue.
I’ve run into that myself a few times when trying to use AI. You give it a very clear prompt to find something and it sometimes just falls flat on its face. It’s easy for the AI evangelists to just blame the human who wrote the prompt, or say “you didn’t give it enough context!” But anyone who’s tried using AI and is being objective about it will tell you that’s a weak excuse that doesn’t hold water a good chunk of the time. You can give it plenty of context, and be very clear, and it still doesn’t find all the examples that clearly match the prompt.
Ultimately, you often have better luck using a well-crafted regular expression to search for text than using AI.
And that seems like the crux of the issue (which you also highlight). While there are some very good use cases for AI, it’s being waaaay over-used. And too often its faults are dismissed or glossed over.
I have a “prosumer” internet setup at home for various reasons. It’s UniFi gear, which is highly configurable, and configs are centrally managed. They provide a pretty robust web UI to manage it all, but the configuration all resides in plain text files that you can also hand edit if you want to do anything really advanced.
While troubleshooting an issue recently I came across a post on their support forum from somebody who had used Claude to analyze those config files and make recommendations. Since I have access to Claude through my employer I decided to give that a try. I was pleasantly surprised with the recommendations it made after it spent a few minutes analyzing my configuration.
The application you’re talking about might not be a bad one at all. Network configuration is a relatively narrow scope with very well-defined rules and protocols.
I feel that people often ask LLMs to do things that are far too broad in scope for them to have a lot of hope of doing a good job, but I think you might have found a solid one. Obviously you want to double check everything for sanity, but that’s not so hard to do either.
To me, that’s the ‘fancy search engine’ mode of AI where it works well and basically focuses the human effort. A needle-in-haystack problem. It might still be missing things, but they’re things you’ve already missed yourself, so no loss.
It’s different from asking Claude, for example, to create a new guest VLAN with limited internet access and access to only a specific service on the private network. For that, you have to 1) trust Claude because you lack the expertise to review, 2) spend time learning the config system well enough to review, or 3) already know the system well enough to check it. 1) just sounds bad. 2) sounds like Claude isn’t saving much time, but maybe helps focus the human where to study, and 3) seems like the human might have been able to just do the job in similar or less time than writing the prompt + reviewing the result.
I feel like the big mistake they continue to propagate is failing to distinguish among the uses of AI.
A lot of hype seems to be the generative uses, where AI creates code, images, text, or whatever, or the agentic uses where it supposedly automates some process. Safe uses in that way should involve human review and approval, and if the human spends as much time reviewing as they would creating it in the first place, then there’s a productivity loss.
All the positive cases I’ve heard of use AI like a fancy search engine - look for specific issues in a large code base, look for internal consistency in large document or document sets. That form lets the human shift from reading hundreds or thousands of pages to reading whatever snippets the AI returns. Even if that’s a lot of false positives, it’s still a big savings over full review. And as long as the AI’s false-negative rate is better than the human, it’s a net improvement in review.
And, of course, there’s the possibility that AI facilitated review allows companies to do review of documents that they would otherwise have ignored as intractable, which would also show up as reduced productivity.
You know…this.
I’ve used AI plenty of times to help troubleshoot some weird error message. Sometimes just an old-fashioned Google just isn’t enough. There needs to be added context, which would just screw up the Google results.
I treat talking to AI for advice (in any category) roughly the same as asking an IRC channel…because that’s basically what it is. It’s taking in data from tons of sources and summarizing it.
Some of those sources might be legitimate and knowledgeable, some of them might be a negative-scored stack overflow comment.
If you have no domain-specific knowledge, you won’t know how to identify an issue in its response, and you shouldn’t be blindly copying code. Trust…but verify.
In that scenario where AI is used to find specific code snippets or other matching text blocks, the false positives aren’t really the problem. The false negatives are the issue.
I’ve run into that myself a few times when trying to use AI. You give it a very clear prompt to find something and it sometimes just falls flat on its face. It’s easy for the AI evangelists to just blame the human who wrote the prompt, or say “you didn’t give it enough context!” But anyone who’s tried using AI and is being objective about it will tell you that’s a weak excuse that doesn’t hold water a good chunk of the time. You can give it plenty of context, and be very clear, and it still doesn’t find all the examples that clearly match the prompt.
Ultimately, you often have better luck using a well-crafted regular expression to search for text than using AI.
And that seems like the crux of the issue (which you also highlight). While there are some very good use cases for AI, it’s being waaaay over-used. And too often its faults are dismissed or glossed over.
I have a “prosumer” internet setup at home for various reasons. It’s UniFi gear, which is highly configurable, and configs are centrally managed. They provide a pretty robust web UI to manage it all, but the configuration all resides in plain text files that you can also hand edit if you want to do anything really advanced.
While troubleshooting an issue recently I came across a post on their support forum from somebody who had used Claude to analyze those config files and make recommendations. Since I have access to Claude through my employer I decided to give that a try. I was pleasantly surprised with the recommendations it made after it spent a few minutes analyzing my configuration.
The application you’re talking about might not be a bad one at all. Network configuration is a relatively narrow scope with very well-defined rules and protocols.
I feel that people often ask LLMs to do things that are far too broad in scope for them to have a lot of hope of doing a good job, but I think you might have found a solid one. Obviously you want to double check everything for sanity, but that’s not so hard to do either.
To me, that’s the ‘fancy search engine’ mode of AI where it works well and basically focuses the human effort. A needle-in-haystack problem. It might still be missing things, but they’re things you’ve already missed yourself, so no loss.
It’s different from asking Claude, for example, to create a new guest VLAN with limited internet access and access to only a specific service on the private network. For that, you have to 1) trust Claude because you lack the expertise to review, 2) spend time learning the config system well enough to review, or 3) already know the system well enough to check it. 1) just sounds bad. 2) sounds like Claude isn’t saving much time, but maybe helps focus the human where to study, and 3) seems like the human might have been able to just do the job in similar or less time than writing the prompt + reviewing the result.