Singer getting stuck when there's frequent file rotation #513
Description
We’re using Singer as a sidecar container in our pod to read logs from the main container. However, we’re facing an issue when frequent log file rotations occur. Singer throws the following error:
ERROR [2025-10-15 08:29:36,691] com.pinterest.singer.processor.DefaultLogStreamProcessor:
The position: LogPosition(logFile:LogFile(inode:11666108), byteOffset:10400428) from watermark is invalid in LogStream{tutorial.text_log:payload.log}.
Even though a new payload.log file is created successfully, Singer continues to throw this error at regular intervals until we manually restart the process.
Your help in resolving this issue would be greatly appreciated. 😊