I noticed the agent was working on the code and the file got bigger, and bigger, say 3k, then 6k, then 9k.
...Then at some point the file was 2 or 3k again. Basically the agent just deleted everything it had done and rewrite the file.
I think that code files should not be thought of as "files" to the agent.
Instead, whether the agent writes the code, or ingests it, the agent should determine how to split up the file (into a series of functions for example) and should have a corresponding test(s) for each function, and should only make changes to that specific function.
The entire file if possible (or maybe the header in the case of C++) should still be in the context, but the only part actually being edited should be the 1 specific function, which should be stored in its own place. Then when the agent writes the file to disk, it "compiles" it from those chunks into the final text file that is written. The agent should not have the power to rewrite the entire file at once and re-save it. Otherwise that situation prevents it from systematically working through the problem set, and instead causes it to loop endlessly starting over at the beginning with a rewrite and losing work that it had already done and even debugged.
This happened tonight with Claude-Sonnet-3.5 BTW. But I think I've also seen it with GPT-4o, deepseek-coder and/or qwen2 72b instruct.
BTW -- you may already be doing this, I saw hints of something like this in the log. But I still saw the problem, so maybe I just need to understand the code better so I can help prevent this sort of issue from happening.
I noticed the agent was working on the code and the file got bigger, and bigger, say 3k, then 6k, then 9k.
...Then at some point the file was 2 or 3k again. Basically the agent just deleted everything it had done and rewrite the file.
I think that code files should not be thought of as "files" to the agent.
Instead, whether the agent writes the code, or ingests it, the agent should determine how to split up the file (into a series of functions for example) and should have a corresponding test(s) for each function, and should only make changes to that specific function.
The entire file if possible (or maybe the header in the case of C++) should still be in the context, but the only part actually being edited should be the 1 specific function, which should be stored in its own place. Then when the agent writes the file to disk, it "compiles" it from those chunks into the final text file that is written. The agent should not have the power to rewrite the entire file at once and re-save it. Otherwise that situation prevents it from systematically working through the problem set, and instead causes it to loop endlessly starting over at the beginning with a rewrite and losing work that it had already done and even debugged.
This happened tonight with Claude-Sonnet-3.5 BTW. But I think I've also seen it with GPT-4o, deepseek-coder and/or qwen2 72b instruct.
BTW -- you may already be doing this, I saw hints of something like this in the log. But I still saw the problem, so maybe I just need to understand the code better so I can help prevent this sort of issue from happening.