RouterOS `/tool traceroute` (and `/ping`) fails with `generic_error` when a hop does not answer

[vadvolo]

Summary

Running an interactive, self-refreshing RouterOS command such as
/tool traceroute count=1 <ip> through the gnetcli gRPC Exec API returns:

rpc error: code = Internal desc = generic_error

whenever the traced path contains an intermediate hop that does not answer (a
timeout row). The destination is reachable (/ping shows 0% loss) and the full output
is produced by the device — gnetcli captures it but then discards it and reports the
opaque generic_error.

Destinations whose hops all answer quickly usually succeed, so the bug looks intermittent.
The distinguishing factor is not reachability but whether the device performs an in-place
cursor-up redraw of its live table (which a stalled/non-answering hop forces).

Environment

• Device: MikroTik RouterOS 6.49.3 (RB2011 / hEX)
• Transport: SSH streamer, program: shell

Reproduction

Exec(host=<routeros>, cmd="/tool traceroute count=1 172.21.4.204")

Destination	Path	Result (before fix)
8.8.8.8	all hops answer	usually OK (intermittent)
172.22.2.4	2 hops, all answer	OK
172.21.4.204	hop 3 = timeout	always generic_error

/ping 172.21.4.204 count=3 → sent=3 received=3 packet-loss=0% (reachable).

Root cause

RouterOS renders /tool traceroute / /ping as a live table that it redraws in place
using ESC[<n>A (cursor up), ESC[K (erase line) and \r. With a wide login terminal,
every line — including the prompt — is right-padded with ~1000 spaces and the prompt is
re-emitted several times via \r. A non-answering hop stalls ~1 s, which forces at least
one full cursor-up redraw cycle; fast, all-answering paths often finish in a single frame
and avoid it — hence the "intermittent" appearance.

Three independent defects combine so that this output is thrown away:

Bug 1 — pkg/streamer/ssh/ssh.go: data lost on (n>0, io.EOF)

chanReader checked the read error before delivering the bytes:

readLen, err := reader.Read(readBuffer)
if err != nil {           // returns here on io.EOF...
    close(tmpBuffer); _ = wg.Wait(); return err
}
tmpBuffer <- readBuffer[:readLen]   // ...so these bytes are never delivered

io.Reader.Read is allowed to return data and a non-nil error (typically io.EOF)
in the same call, and SSH channels do exactly this when the peer closes right after its
final write — which RouterOS does when a finished traceroute returns to the prompt. The
last chunk (the prompt) is silently dropped.

Bug 2 — pkg/streamer/streamer.go: no final match on EOF

In GenericReadX, the "channel closed" branch returns EOF without trying the
expression one last time, so a buffer that already ends in a valid prompt is discarded:

if !ok {
    return NewReadXRes(EOF, buffer, nil, []byte{}), ...   // buffer may already match!
}

Bug 3 — pkg/terminal/parser.go: unhandled ESC[<n>A (CUU / cursor up)

Parse handled the CSI finals D K m H J but not A (CUU) / B (CUD). On the redraw
sequence it hit the default and returned:

unknown esc A ...

which the exec path turns into generic_error. This is the final blocker: with bugs 1
and 2 fixed the prompt matches, but the parser then rejects the cursor-up embedded in the
captured output.

Why the error is opaque

cmd/gnetcli_server/server.go connectionErrorInterceptor maps every non-connect error
to ErrorTypeGeneric and stores the real cause only in ErrorInfo.Metadata["err"], which
the client did not surface. The real causes (sessionStdoutReader error EOF, then
unknown esc A ...) were only visible in the server DEBUG log.

Patch

Four files. Bugs 1–3 are the core fix; the ros change is prompt-matching robustness for
the padded/redrawn prompt.

1) pkg/streamer/ssh/ssh.go — deliver bytes before acting on the read error

 	for {
 		readBuffer := make([]byte, defaultReadSize)
 		readLen, err := reader.Read(readBuffer)
-		if err != nil {
-			// flush
-			close(tmpBuffer)
-			_ = wg.Wait()
-			return err
-		}
-		logger.Debug("read", zap.ByteString("data", readBuffer[:readLen]))
-		tmpBuffer <- readBuffer[:readLen]
+		if readLen > 0 {
+			// Read may return data together with an error (notably io.EOF when the
+			// peer closes the channel right after its last write, as RouterOS does
+			// after a finished traceroute/ping). Deliver the bytes before acting on
+			// the error, otherwise the final chunk (e.g. the prompt) is dropped and
+			// the command is reported as generic_error.
+			logger.Debug("read", zap.ByteString("data", readBuffer[:readLen]))
+			tmpBuffer <- readBuffer[:readLen]
+		}
+		if err != nil {
+			// flush
+			close(tmpBuffer)
+			_ = wg.Wait()
+			return err
+		}
 	}

2) pkg/streamer/streamer.go — final expr match before returning EOF (GenericReadX)

 			if !ok {
+				// The stream closed. The final chunk may already contain a full match
+				// (device printed the prompt and immediately closed the session), so try
+				// once more before reporting EOF instead of discarding valid output.
+				if regExpr != nil {
+					if mRes, matched := regExpr.Match(buffer); matched {
+						var underlyingRes ReadRes
+						if mRes.Underlying != nil {
+							underlyingRes = NewReadResImpl(buffer[:mRes.Underlying.Start], buffer[mRes.Underlying.End:], mRes.Underlying.GroupDict, buffer[mRes.Underlying.Start:mRes.End], mRes.Underlying.PatternNo)
+						}
+						res := NewReadResImplWithUnder(buffer[:mRes.Start], buffer[mRes.End:], mRes.GroupDict, buffer[mRes.Start:mRes.End], mRes.PatternNo, underlyingRes)
+						after := buffer[mRes.End:]
+						return NewReadXRes(Expr, buffer, res, after), after, buffer[len(inBuffer):], nil
+					}
+				}
 				return NewReadXRes(EOF, buffer, nil, []byte{}), buffer, buffer[len(inBuffer):], nil
 			}

3) pkg/terminal/parser.go — handle CUU (A) / CUD (B) like CUP/ED (strip)

 	SGR     = 'm'
 	CUP     = 'H'
 	ED      = 'J'
+	CUU     = 'A' // cursor up
+	CUD     = 'B' // cursor down
 )

-			case CUP, ED: // not implemented
+			case CUP, ED, CUU, CUD: // cursor positioning we don't render - strip it
 				m.data = sliceEdit(m.data, escStart, m.pos+1)
 				m.pos = escStart - 1
 				continue

4) pkg/device/ros/device.go — prompt-match robustness

The RouterOS prompt is padded to terminal width and redrawn via \r, so the real prompt
can sit far from the tail and/or be followed by a truncated redraw remnant. Widen the
prompt match window and relax the anchored tail so it still matches.

-	visiblePrompt = `\[(?P<login>\S+)@(?P<hostname>\S+)\]\s{1,2}(?P<cfg_path>\/[\/\w\s-]+)?(<(?P<safe_mode>SAFE))?> $`
+	visiblePrompt = `\[(?P<login>\S+)@(?P<hostname>[^\]]+)\]\s+(?P<cfg_path>\/[\/\w\s-]+)?(<(?P<safe_mode>SAFE))?>[^\n]*$`

 	cli := genericcli.MakeGenericCLI(
-		expr.NewSimpleExprLast(1500).FromPattern(promptExpression),
+		expr.NewSimpleExprLast(65536).FromPattern(promptExpression),
 		expr.NewSimpleExprLast(2500).FromPattern(errorExpression),

Note: the visiblePrompt diff is shown against upstream; adapt to your current regex.
The >[^\n]*$ tail requires the literal [login@host] > prefix, so it does not create
false positives inside command output (a prompt only appears at the true end, and
[^\n]* cannot cross a newline).

Testing

• /tool traceroute count=1 172.21.4.204 → succeeds, returns all hops incl. the
  timeout hop and the final destination. Repeated single runs are stable.
• 8.8.8.8, 172.22.2.4, 10.129.18.38 traceroutes → OK.
• /ping, /system resource print, /interface print, /ip route print → unchanged.
• go build ./... clean.

Known cosmetic limitation

Because cursor-up is stripped rather than rendered, a successful traceroute may contain
an intermediate redraw frame (the table can appear twice). The final frame is complete
and correct. Rendering the redraw into a single clean frame would require a screen-buffer
terminal emulator (track cursor row, overwrite in place); stripping keeps the parser simple
and safe for config-gathering output and matches how CUP/ED are already handled.

Related, not addressed here

• Opaque generic_error. Consider surfacing Metadata["err"] (or distinguishing an
  EOF/parse failure from other internal errors) so callers can diagnose without server logs.
• Unbounded runs. Without count=/duration=, RouterOS traceroute/ping refresh
  forever and never return a prompt (the call runs to the deadline). pagerExpression
  also does not match the monitor footer -- [Q quit|D dump|C-z pause] (only …|down]),
  and the generic pager auto-answer sends a space, which does not quit the RouterOS
  monitor. A RouterOS-aware interactive-monitor handler (recognize the footer, send q)
  would be the complete solution; callers should meanwhile always pass count=/duration=.