Coding assistant failure example
This post discusses an example failure in using AI coding assistants that demonstrates two common problems: 1) they get tunnel vision, 2) they don't actually understand what they are talking about!
You can read the actual transcript
Too summarize: I give ChatGPT a problem and it generates a solution that is flawed. When I explain that it doesn't work, ChatGPT acts like it knows what the problem is (failed pattern matching) and proposes a corrected solution. That one also fails and ChatGPT tries again but it has developed tunnel vision on the pattern matching and that really isn't the problem.
After studying the code myself I suggest a different diagnosis, to which ChatGPT replies: "prompt() collapses newlines. When you paste multi-line text into a prompt(), Firefox silently replaces line breaks with spaces, which makes any line-based parsing fail.That explains everything."
This is the critical point in the conversation. ChatGPT generated the code to use 'prompt()' which wasn't going to work. When I pointed out why it didn't work it was able to explain the failure, but it wasn't clever enough as it generated the original solution to realize it would fail.
That's because it doesn't really understand what it's generating. It is producing code based on thousands of examples it has been trained on, not on any fundamental conceptual understanding of the meaning of that code.
I think this is a pretty major limitation of coding assistants and I have doubts that there's an easy way to overcome it.