feat(docs): add telemetry backend streaming architecture design#1092
feat(docs): add telemetry backend streaming architecture design#1092yakshithkd23 wants to merge 1 commit into
Conversation
WalkthroughThis PR adds three documentation files for the telemetry backend: an end-to-end README, a short architecture design introduction, and a schema document introduction for structured JSON communication between the client and ingest framework. ChangesTelemetry Backend Documentation
Estimated code review effort: 1 (Trivial) | ~3 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/telemetry-backend/architecture-design.md`:
- Around line 1-3: The document "Architecture Design: Telemetry Streaming
Pipeline" is currently a stub; expand it to provide the promised end-to-end
blueprint by adding sections for a system component diagram, a clear data flow
sequence (client → ingestion → Kafka → Spark/stream processing → datastore →
dashboard), component responsibilities and interfaces (ingester, Kafka topics,
stream processor, storage, visualization), deployment architecture (k8s/VMs,
networking, configs), and scaling/fault-tolerance strategies (partitions,
consumer groups, checkpointing, retries, monitoring); if you intended this as a
placeholder, replace the body with a brief “TODO/Planned” note referencing issue
`#719` and an expected content outline.
In `@docs/telemetry-backend/data-schemas.md`:
- Line 1: Replace the placeholder intro in
docs/telemetry-backend/data-schemas.md with concrete JSON schema definitions:
add event payload schemas for performance metrics, user interactions, and crash
logs (with required vs optional fields, data types, and validation constraints),
include example JSON payloads, an error response format, and a schema versioning
strategy for evolution; ensure each schema is named and discoverable (e.g.,
"PerformanceEvent", "InteractionEvent", "CrashEvent") and document required
fields, types, and sample payloads so the Flutter client and backend ingest
service can implement compatible serialization/deserialization, or if this is a
deliberate stub, add a clear "planned work / TODO" note with expected
deliverables and timeline.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 804c4a83-bd64-46af-8016-21e7983ea6ca
📒 Files selected for processing (3)
docs/telemetry-backend/README.mddocs/telemetry-backend/architecture-design.mddocs/telemetry-backend/data-schemas.md
yakshithkd23
force-pushed
the
backend_Server_1
branch
from
c263c3d to
bda923c
Compare
yakshithkd23
changed the title
|
i was started to create backend logic as per the architecture in my personal repository , since i was planned backend of telemetry such way that it will become a new service or module. |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/telemetry-backend/README.md (1)
11-11: ⚡ Quick winClarify the "Zero-Overhead" characterization for Vector.
Vector does consume CPU and memory resources. Describing it as "zero-overhead" could mislead readers about the actual resource footprint during deployment. Consider replacing with "lightweight" or "minimal-overhead" to set accurate expectations.
💡 Suggested edit
- * **Zero-Overhead Log Shipping:** A lightweight **Vector** sidecar container monitors the shared volume, parses completed JSON files, and instantly streams them to a distributed message queue. + * **Lightweight Log Shipping:** A lightweight **Vector** sidecar container monitors the shared volume, parses completed JSON files, and instantly streams them to a distributed message queue.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/telemetry-backend/README.md` at line 11, The bullet point describing log shipping uses the term "Zero-Overhead Log Shipping" which is misleading since Vector does consume CPU and memory resources during operation. Replace "Zero-Overhead" with a more accurate characterization such as "Lightweight" or "Minimal-Overhead" in the bullet point title to set proper expectations about Vector's actual resource footprint while keeping the rest of the description intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@docs/telemetry-backend/README.md`:
- Line 11: The bullet point describing log shipping uses the term "Zero-Overhead
Log Shipping" which is misleading since Vector does consume CPU and memory
resources during operation. Replace "Zero-Overhead" with a more accurate
characterization such as "Lightweight" or "Minimal-Overhead" in the bullet point
title to set proper expectations about Vector's actual resource footprint while
keeping the rest of the description intact.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 8e48c50b-e09d-4e94-b4a9-648dd559ef25
📒 Files selected for processing (3)
docs/telemetry-backend/README.mddocs/telemetry-backend/architecture-design.mddocs/telemetry-backend/data-schemas.md
✅ Files skipped from review due to trivial changes (2)
- docs/telemetry-backend/architecture-design.md
- docs/telemetry-backend/data-schemas.md
|
Updated the README file based on Architecture 3, which is the updated version of Architecture 2. We have also started the implementation in the registration client TUSD server branch and are proceeding according to the new architecture: registration_client_tusd-server (regi_client_test_yak branch) |
|
Updated the README file based on Architecture 3, which is the updated version of Architecture 2. We have also started the implementation in the registration client TUSD server branch and are proceeding according to the new architecture: reference: https://github.com/yakshithkd23/registration_client_tusd-server/tree/regi_client_test_yak Team TuluTech |
yakshithkd23
force-pushed
the
backend_Server_1
branch
from
79d195d to
47c0308
Compare
Signed-off-by: Yakshith K D <yakshithkd97@gmail.com>
yakshithkd23
force-pushed
the
backend_Server_1
branch
from
47c0308 to
7462e7f
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/telemetry-backend/README.md`:
- Around line 7-15: The README’s telemetry architecture description skips the
local Hive-backed offline buffer, causing the documented flow to diverge from
the intended design. Update the telemetry pipeline summary to explicitly mention
Hive queueing in the client-side buffering step and show it as the layer between
Android event capture and TUS upload, using the existing telemetry backend
architecture sections as the place to reflect this. Keep the wording aligned
with the current terms like client buffering, TUS protocol, and offline
operations so the documented flow matches the implementation target.
- Around line 78-79: The storage guidance in the README mixes up
high-cardinality indexes with low-cardinality filter fields; update the wording
in the telemetry backend section to avoid recommending log_level as a
high-cardinality index target. Use the existing storage/indexing guidance near
the structured tables description to distinguish selective identifiers like
client_id and session_id from low-cardinality dimensions such as log_level, and
adjust the phrasing to reflect that only the selective fields are suitable for
indexing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: b8fb26e6-5ea3-4da3-9753-cd08215aa09d
📒 Files selected for processing (3)
docs/telemetry-backend/README.mddocs/telemetry-backend/architecture-design.mddocs/telemetry-backend/data-schemas.md
✅ Files skipped from review due to trivial changes (2)
- docs/telemetry-backend/data-schemas.md
- docs/telemetry-backend/architecture-design.md
1 participant
Overview
This Pull Request introduces the foundational architectural and design documentation for the real-time telemetry processing backend under a new dedicated directory:
docs/telemetry-backend/.Changes Included
README.md: Outlines the high-level system components, decoupling logic (offline buffering vs. server processing), and the technical stack (Apache Kafka, Apache Spark Streaming, and analytical datastores).architecture-design.md: Provides the blueprint for the end-to-end data pipeline feeding the real-time operator monitoring dashboard.data-schemas.md: Details the structured JSON data contract between the Flutter client application and the backend streaming services.Context & Goals
This architecture supports the development of the server-side infrastructure required for issue #719. It addresses scaling requirements by offloading telemetry workloads from the main client application through local Hive queueing, high-throughput message streaming, and windowed analytical processing.
Summary by CodeRabbit