Yet another opinion on LLMs · Hasufell's blog

hasufell:

I’m starting to believe that not everyone actually rigorously questions the output. There have been cases where they suggested me to adjust tests (as in: actually break them) to fix a problem.

They really really should question it. Always, because I have noticed this too - I think it is an artifact of how models have been specifically trained to make unprompted suggestions almost no matter what, because if the problem space is sufficiently constrained (ie, you show it an actually ‘perfect’ piece of code of sufficient complexity), it will still try to fill the suggestions void in one of several ways:

• copy and paste snippets of your code verbatim and tell you it made changes and improvements
• make meaningless syntactic changes that have no effect on program structure
• suggesting increasingly bizarre nomenclature
• actually start breaking shit like you said