Skip to content

Deduplicator Logic Updates For New Schemas and Cleanup#8

Merged
drewjj merged 25 commits intodevelopfrom
fix/ode-schema-updates
Aug 8, 2025
Merged

Deduplicator Logic Updates For New Schemas and Cleanup#8
drewjj merged 25 commits intodevelopfrom
fix/ode-schema-updates

Conversation

@drewjj
Copy link
Copy Markdown

@drewjj drewjj commented Aug 1, 2025

Includes logic updates for all deduplication topologies and cleans up the project to reduce technical debt.

  • Updated deduplication topologies to match the current schema structures for the latest jpo-ode and jpo-geojsonconverter:
    • OdeBsmJson
    • ProcessedBsm
    • OdeMapJson
    • ProcessedMap
    • ProcessedMapWKT
    • ProcessedSpat
    • OdeTimJson
  • Removed the OdeRawEncodedTimJson deduplication topology due to it no longer being necessary for the jpo-ode with the micrometer metrics
  • Updated all test examples to only have a base reference message example. All additional test messages are programmatically modified versions of each message's base reference example. A few message types still have a 2nd example message intended to represent an entirely unique additional message.
  • All Pair POJOs have been removed due to their lack of usage.
  • The GeoUtilsTest class has been fixed so that Maven will find the test cases and include them when running tests.
  • Basic serialization and JsonSerdes class tests have been added for each topology to isolate serialization issues.
  • The deduplication topology processors for ProcessedMap and ProcessedMapWKT have been made consistent to utilize the properties OdeGeneratedAt for timing calculations.

Copy link
Copy Markdown

@Michael7371 Michael7371 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, just a few comments.

Checks:

  • Unit tests pass
  • Docker build succeeds
  • topic.ProcessedMap -> topic.DeduplicatedProcessedMap
  • topic.ProcessedMapWKT -> topic.DeduplicatedProcessedMapWKT
  • topic.OdeMapJson -> topic.DeduplicatedOdeMapJson
  • topic.OdeTimJson -> topic.DeduplicatedOdeTimJson
  • topic.OdeBsmJson -> topic.DeduplicatedOdeBsmJson
  • topic.ProcessedSpat -> topic.DeduplicatedProcessedSpat

Comment thread docs/Release_notes.md
Comment thread jpo-deduplicator/src/main/resources/application.yaml Outdated
Copy link
Copy Markdown

@Michael7371 Michael7371 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few questions.

Comment thread jpo-deduplicator/src/main/resources/application.yaml Outdated
Copy link
Copy Markdown

@Michael7371 Michael7371 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks Good!

Comment thread Dockerfile
Comment thread README.md
Comment thread docs/Release_notes.md
Comment thread jpo-deduplicator/pom.xml Outdated
@drewjj drewjj requested a review from John-Wiens August 8, 2025 07:19
@drewjj drewjj merged commit 6733ae1 into develop Aug 8, 2025
3 checks passed
@drewjj drewjj deleted the fix/ode-schema-updates branch August 8, 2025 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants