Support this channel on Patreon ► https://www.patreon.com/zoranhorvatGenerative AI can write code, but it cannot develop software on its own. Here is why the…
Support this channel on Patreon ► https://www.patreon.com/zoranhorvatGenerative AI can write code, but it cannot develop software on its own. Here is why the…
I don’t have great experiences trying to get LLMs to write code, but I’ve found it to be incredibly valuable for checking my code for mistakes and oversights. Sure, it will spit out nit picks, false positives, and straight up nonsense, but at a rate that is quite a bit lower than other tools I’ve used. I not infrequently end up going through a number of iterations, rewriting code and tests based on its feedback, until I’m happy with the code and the LLM no longer points out actual problems.
It probably doesn’t help my productivity, in terms of lines of code committed or whatever companies might use, but it certainly helps improve the quality of my work
It’s only really good if you are totally confident in what is and is not a mistake though, as it’ll do things like replace a struct with an int and be like “use a vec of indices and access the actual array with that since it’s smaller to copy and pass around!” when you have no need to copy and pass it around, and it basically created a layer of indirection for no reason.
It’ll then make up some reason as to why you DO need to pass it around and copy it, so you have to be REALLY sure about what the code should be doing.
It probably depends on what model you are using, but what you describe is more akin to the kind of advice I’ve gotten when I’ve asked for suggestions for optimizations, rather than asked the LLM to identify (not solve) problems in the code. When I’ve asked the LLM to identify problems, the overwhelming majority of issues raised where true positives, though most of them weren’t very serious either
Yeah I would say it’s pretty good at catching most mistakes the majority of the time, it’s just the few times when it does make a mistake that I’m not sure is a mistake or not, it’ll argue in a way that seems plausible and forces me to rethink and verify the code that was already correct. I probably still save time with it, but it can be incredibly annoying and time consuming when it fails.
True. Most of the false positives were easy to dismiss, but I did spent a significant amount of time of time on a couple of them, since it wasn’t immediately clear to me that the code was correct. In one case the agent had missed a precondition that was verified elsewhere, and in the other it had misrepresented what the C++ standard actually said