Skip to content

feat: GTFS-RT feed validation & E012 compliance fix#64

Open
diveshpatil9104 wants to merge 3 commits intoOneBusAway:mainfrom
diveshpatil9104:feat/gtfsrt-feed-validation
Open

feat: GTFS-RT feed validation & E012 compliance fix#64
diveshpatil9104 wants to merge 3 commits intoOneBusAway:mainfrom
diveshpatil9104:feat/gtfsrt-feed-validation

Conversation

@diveshpatil9104
Copy link
Copy Markdown
Contributor

@diveshpatil9104 diveshpatil9104 commented Mar 16, 2026

Summary

  • Fix E012: header timestamp now guaranteed >= all entity timestamps
  • Add bearing (0-360) and speed (>= 0) ingest validation per GTFS-RT spec
  • Add 21-test spec-validation suite covering all 12 MobilityData validator rules
  • Fix pre-existing test quality issues (realistic timestamps, assert.Eventually)

Why

Milestone 1 exit criteria requires the feed to pass the https://github.com/MobilityData/gtfs-realtime-validator without errors. An audit found two violations:

  1. E012 — Header timestamp could be less than entity timestamps when vehicles report future times (within 5-min ingest window). Fixed with max(now, maxEntityTimestamp).
  2. E027 — No bearing/speed validation, so invalid values could reach the feed. Fixed with ingest range checks.

What Changed

  • handlers.go: E012 fix in buildFeed(), bearing/speed validation in validate(), skip zero-timestamp vehicles
  • feed_validation_test.go: New file — validateFeedCompliance helper + 21 tests for E001, E012, E026, E027, E038, E039, E048, E049, E050, E052, W001, W002
  • handlers_test.go: E012 assertion, 6 rejection cases, boundary acceptance test, fix Timestamp: 100 and time.Sleep

Known Limitations

@diveshpatil9104 diveshpatil9104 marked this pull request as ready for review March 16, 2026 13:21
Copy link
Copy Markdown
Member

@aaronbrethorst aaronbrethorst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Divesh, this is a well-structured approach to GTFS-RT compliance — the E012 header timestamp fix is exactly right, the validateFeedCompliance test helper is a great reusable asset for catching regressions, and the test quality improvements (replacing Timestamp: 100 with time.Now().Unix() and time.Sleep with assert.Eventually) are welcome cleanups. The 21-test validation suite organized by MobilityData rule number makes it easy to verify coverage at a glance.

Before this can merge, there are a few things to address.

Critical Issues (2 found)

  1. Merge conflicts with main. GitHub reports this PR as CONFLICTING. You'll need to rebase against main to resolve.

  2. Branch is stale against PR #45 (*float64 pointer migration) — code will not compile after rebase. PR #45 changed Bearing, Speed, and Accuracy from float64 to *float64 on both LocationReport and VehicleState. This PR uses bare float64 throughout:

    • Validation (handlers.go:67-71): r.Bearing < 0 || r.Bearing > 360 and r.Speed < 0 won't compile when these are *float64. After rebase, these checks must be conditional on non-nil — a nil bearing/speed (field not provided) should be allowed, not rejected. The validation should be:

      if r.Bearing != nil && (*r.Bearing < 0 || *r.Bearing > 360) {
          return fmt.Errorf("bearing must be between 0 and 360 (inclusive)")
      }
      if r.Speed != nil && *r.Speed < 0 {
          return fmt.Errorf("speed must be non-negative")
      }
    • buildFeed() (handlers.go:201-202): proto.Float32(float32(v.Bearing)) and proto.Float32(float32(v.Speed)) use bare value access, but on main these are pointers with conditional emission (see PR #45's buildFeed changes that use nil guards). After rebase, you'll need to adopt the same nil-guard pattern.

    • feed_validation_test.go: All VehicleState literals use bare float64 fields (e.g., Bearing: 180, Speed: 8.5) which won't compile against *float64. These need to use the float64ptr() helper.

    • handlers_test.go: Same issue — the new validation test cases use Bearing: -1, Speed: -5, etc. on LocationReport, which will need float64ptr().

Important Issues (1 found)

  1. E050 tension with ingest window is documented but may need a follow-up. The ingest window allows timestamps up to now+300s, but E050 rejects >now+60s. The PR correctly documents this in TestFeedValidation_E050_TimestampsNotFarFuture and in the "Known Limitations" section. However, this means a vehicle with a timestamp 2 minutes in the future will pass ingest validation, appear in the feed, and fail the MobilityData validator for E050. Since the goal is to pass the validator without errors, this gap should be tracked for follow-up — either tighten the ingest window to 60s (breaking change) or clamp entity timestamps in buildFeed() to min(ts, now+60).

Suggestions (1 found)

  • Consider float64ptr() boundary tests for the pointer migration. Once you rebase and adopt *float64, you'll want test cases that verify nil bearing/speed passes validation (no rejection), explicit zero passes (bearing=0 is north, speed=0 is stationary), and out-of-range values are rejected. The current test cases cover the range checks well but will need the nil-is-valid case added.

Strengths

  • E012 fix is clean and correct: headerTimestamp = max(now, maxEntityTimestamp) is a simple, correct solution
  • Zero-timestamp vehicle skip prevents W001 violations in the feed with an appropriate slog.Warn
  • validateFeedCompliance helper is a great reusable asset — it tests buildFeed() through the lens of every applicable MobilityData rule
  • Test quality improvements: replacing hardcoded Timestamp: 100 with realistic time.Now().Unix() and flaky time.Sleep with assert.Eventually
  • 21 well-organized tests covering E001, E012, E026, E027, E038, E039, E048, E049, E050, E052, W001, W002
  • Protobuf round-trip test (TestFeedValidation_FeedSerializesCleanly) verifies the feed survives marshal/unmarshal

Recommended Action

  1. Rebase against main to resolve merge conflicts
  2. Adapt all bearing/speed code to the *float64 pointer types from PR #45
  3. Add nil-bearing/speed validation test cases
  4. Consider tracking the E050/ingest-window tension for follow-up
  5. Re-run review after fixes

@diveshpatil9104 diveshpatil9104 force-pushed the feat/gtfsrt-feed-validation branch from 9c83b54 to a14bfd5 Compare March 26, 2026 07:47
@diveshpatil9104
Copy link
Copy Markdown
Contributor Author

hii aaron! I Rebased against main and resolved all conflicts. Changes:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants