Skip to content

FIX: incorrect finalOffset and positionIncrement when stopwords filtered for issue#921 - #1145

Closed
kin122 wants to merge 1 commit into
masterfrom
issue921
Closed

FIX: incorrect finalOffset and positionIncrement when stopwords filtered for issue#921#1145
kin122 wants to merge 1 commit into
masterfrom
issue921

Conversation

@kin122

@kin122 kin122 commented Apr 3, 2026

Copy link
Copy Markdown
Contributor

Fix FVH highlighter offset misalignment for multi-value fields (issue#921)

Problem

When a remote stopword dictionary contains a word (e.g., "value"), and a multi-value field has a value that is completely filtered as a stopword, IKTokenizer.end() returns finalOffset of 0 instead of the actual text length. This causes incorrect cumulative
offsets when ES indexes multi-value fields, leading the FVH highlighter to insert <em> tags at wrong positions.

Changes

A three-layer fix with minimal changes:

  1. AnalyzeContext.java — Track maxConsumedEndPosition (max end position of all consumed lexemes, including filtered stopwords) and lastSkippedCount (number of skipped stopwords) in getNextLexeme()
  2. IKSegmenter.java — Save maxConsumedEndPosition before context.reset(), accumulate skippedCount on each next() return, expose both via getters
  3. IKTokenizer.java — Use accumulated skippedCount to set correct positionIncrement in incrementToken(); use max(endPosition, savedMaxConsumedEndPosition) in end() to ensure correct finalOffset even when all lexemes are filtered

Effect

  • finalOffset correctly reports text length when all lexemes are stopword-filtered (was 0)
  • Skipped stopwords are correctly reflected in positionIncrement (was always 1)
  • Multi-value field offset accumulation no longer drifts, FVH highlight positions are correct
  • No behavior change for non-stopword scenarios

Tests

7 new test cases + 17 existing tests all pass (24/24)

@kin122 kin122 closed this Apr 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant