Skip to content

feat: time-series download analytics backed by TimescaleDB - #2027

Open
gnugomez wants to merge 21 commits into
eclipse-openvsx:mainfrom
gnugomez:feat/download-analytics-timescale
Open

feat: time-series download analytics backed by TimescaleDB#2027
gnugomez wants to merge 21 commits into
eclipse-openvsx:mainfrom
gnugomez:feat/download-analytics-timescale

Conversation

@gnugomez

@gnugomez gnugomez commented Jul 31, 2026

Copy link
Copy Markdown
Member

Related to #2025

Adds opt-in download analytics to the registry: per-extension download counts over time, ingested from CDN access logs into a TimescaleDB hypertable, exposed through a public API and surfaced as a weekly-downloads card on the extension detail page.

Disabled by default (ovsx.analytics.enabled, default false).

Screen.Recording.2026-08-24.at.11.08.49.mov

@gnugomez
gnugomez force-pushed the feat/download-analytics-timescale branch 2 times, most recently from fcae59e to d66b01f Compare August 24, 2026 09:06
@gnugomez
gnugomez marked this pull request as ready for review August 24, 2026 09:11
@gnugomez
gnugomez force-pushed the feat/download-analytics-timescale branch from d66b01f to 35e5126 Compare August 26, 2026 09:12
@gnugomez
gnugomez requested a balanced review from Copilot August 31, 2026 08:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds opt-in TimescaleDB-backed download analytics, CDN log ingestion, a public time-series API, and weekly download statistics in the web UI.

Changes:

  • Introduces analytics storage, ingestion, API, metrics, and tests.
  • Adds the weekly-downloads extension card and frontend service types.
  • Expands frontend extension points and shared UI infrastructure.

Reviewed changes

Copilot reviewed 111 out of 114 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
webui/vite.config.mts Inlines MUI test dependencies
webui/test/unit/support/test-providers.tsx Extends provider test helpers
webui/test/unit/support/menu-queries.ts Adds menu query helper
webui/test/unit/pages/user/avatar.spec.tsx Tests desktop menu contributions
webui/test/unit/pages/admin-dashboard/admin-dashboard.spec.tsx Tests contributed admin pages
webui/test/unit/main.spec.tsx Tests user-loading context
webui/test/unit/default/menu-content.spec.tsx Tests mobile menu contributions
webui/test/unit/components/weekly-downloads.spec.tsx Tests weekly downloads card
webui/src/pages/user/avatar.tsx Supports contributed menu entries
webui/src/pages/search/search-header.tsx Reuses compact control styling
webui/src/pages/extension-detail/weekly-downloads.tsx Implements analytics card
webui/src/pages/extension-detail/use-extension-download-series.ts Fetches annual download series
webui/src/pages/extension-detail/extension-detail.tsx Updates chrome context reference
webui/src/pages/extension-detail/extension-detail-overview.tsx Mounts weekly downloads
webui/src/pages/admin-dashboard/nav-types.ts Defines contributed admin pages
webui/src/pages/admin-dashboard/admin-dashboard.tsx Integrates contributed pages
webui/src/page-settings.ts Adds consumer extension points
webui/src/main.tsx Exposes user-loading state
webui/src/layout/scroll-restoration.tsx Renames scroll helper
webui/src/layout/app-navbar.tsx Supports extended navbar blur
webui/src/layout/app-layout.tsx Uses navbar chrome provider
webui/src/index.ts Expands public frontend exports
webui/src/extension-registry-types.ts Adds analytics response types
webui/src/extension-registry-service.ts Adds analytics API client
webui/src/default/theme.tsx Updates popup styling
webui/src/default/menu-content.tsx Supports mobile menu contributions
webui/src/context/navbar-chrome-context.tsx Consolidates navbar chrome state
webui/src/context/extension-tint-context.tsx Removes superseded context
webui/src/context.ts Adds userLoading
webui/src/components/pill.tsx Adds reusable pill primitive
webui/src/components/page-primitives.tsx Adds shared style primitives
webui/src/components/extension-searchfield.tsx Reuses slash primitive
webui/src/components/category-pill.tsx Reuses generic pill
webui/src/app-providers.tsx Updates provider documentation
webui/CHANGELOG.md Documents frontend changes
server/src/test/resources/org/eclipse/openvsx/storage/log/fastly.log Removes obsolete fixture
server/src/test/resources/org/eclipse/openvsx/analytics/ingestion/aws/fastly.log Adds Fastly ingestion fixture
server/src/test/resources/org/eclipse/openvsx/analytics/ingestion/aws/cloudfront.log Extends CloudFront fixture
server/src/test/resources/application.yml Limits test connection pools
server/src/test/java/org/eclipse/openvsx/storage/StorageUtilServiceUploadFileTest.java Updates ingestion dependencies
server/src/test/java/org/eclipse/openvsx/storage/StorageUtilServiceTest.java Tests ingestion-aware counting
server/src/test/java/org/eclipse/openvsx/storage/log/FastlyLogFileParserTest.java Removes obsolete parser tests
server/src/test/java/org/eclipse/openvsx/storage/log/CloudFrontLogFileParserTest.java Removes obsolete parser tests
server/src/test/java/org/eclipse/openvsx/repositories/RepositoryServiceSmokeTest.java Covers renamed queries
server/src/test/java/org/eclipse/openvsx/RegistryAPITest.java Updates test wiring
server/src/test/java/org/eclipse/openvsx/eclipse/EclipseServiceTest.java Updates test wiring
server/src/test/java/org/eclipse/openvsx/analytics/ingestion/RawDownloadRecordTest.java Tests raw records
server/src/test/java/org/eclipse/openvsx/analytics/ingestion/jobs/IngestionJobsTest.java Tests job scheduling
server/src/test/java/org/eclipse/openvsx/analytics/ingestion/DownloadIngestionMetricsTest.java Tests ingestion metrics
server/src/test/java/org/eclipse/openvsx/analytics/ingestion/CountryCodesTest.java Tests country normalization
server/src/test/java/org/eclipse/openvsx/analytics/ingestion/aws/FastlyLogFileParserTest.java Tests Fastly parsing
server/src/test/java/org/eclipse/openvsx/analytics/ingestion/aws/CloudFrontLogFileParserTest.java Tests CloudFront parsing
server/src/test/java/org/eclipse/openvsx/analytics/ingestion/aws/AccessLogRecordTest.java Tests download extraction
server/src/test/java/org/eclipse/openvsx/analytics/DownloadEventTest.java Tests event validation
server/src/test/java/org/eclipse/openvsx/analytics/DownloadAnalyticsServiceTest.java Tests series aggregation
server/src/test/java/org/eclipse/openvsx/analytics/DownloadAnalyticsEndpointTest.java Tests end-to-end analytics
server/src/test/java/org/eclipse/openvsx/analytics/DownloadAnalyticsDisabledTest.java Tests disabled configuration
server/src/test/java/org/eclipse/openvsx/analytics/DownloadAnalyticsAPITest.java Tests public endpoint
server/src/test/java/org/eclipse/openvsx/admin/AdminAPITest.java Updates test wiring
server/src/test/java/org/eclipse/openvsx/adapter/VSCodeAPITest.java Updates test wiring
server/src/test/java/org/eclipse/openvsx/AbstractTimeseriesContainerTest.java Adds Timescale test base
server/src/test/java/org/eclipse/openvsx/AbstractPostgresContainerTest.java Makes image configurable
server/src/main/resources/db/migration-timeseries/V1__Download_Analytics.sql.conf Configures Flyway transaction mode
server/src/main/resources/db/migration-timeseries/V1__Download_Analytics.sql Creates analytics schema
server/src/main/java/org/eclipse/openvsx/storage/StorageUtilService.java Captures request-path events
server/src/main/java/org/eclipse/openvsx/storage/log/FastlyLogFileParser.java Removes legacy parser
server/src/main/java/org/eclipse/openvsx/storage/log/DownloadCountService.java Removes legacy scheduler
server/src/main/java/org/eclipse/openvsx/storage/log/DownloadCountProcessor.java Removes legacy processor
server/src/main/java/org/eclipse/openvsx/storage/log/CloudFrontLogFileParser.java Removes legacy parser
server/src/main/java/org/eclipse/openvsx/repositories/RepositoryService.java Renames ingestion queries
server/src/main/java/org/eclipse/openvsx/repositories/DownloadIngestionRepository.java Provides ingestion-state queries
server/src/main/java/org/eclipse/openvsx/mirror/aop/DownloadCountServiceAspect.java Removes obsolete mirror aspect
server/src/main/java/org/eclipse/openvsx/LocalRegistryService.java Reports analytics availability
server/src/main/java/org/eclipse/openvsx/json/RegistryVersionJson.java Adds analytics feature flag
server/src/main/java/org/eclipse/openvsx/entities/DownloadIngestion.java Renames ingestion entity
server/src/main/java/org/eclipse/openvsx/analytics/timescale/TimeseriesDatabaseConfiguration.java Configures TimescaleDB
server/src/main/java/org/eclipse/openvsx/analytics/timescale/TimescaleDownloadAnalyticsRepository.java Implements Timescale repository
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/RawDownloadRecord.java Defines raw record
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/jobs/LogIngestionJob.java Schedules ingestion jobs
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/jobs/IngestionJobRequest.java Defines job request
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/DownloadRecordSource.java Defines source contract
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/DownloadIngestionRunner.java Orchestrates ingestion
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/DownloadIngestionMetrics.java Adds operational metrics
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/CountryCodes.java Normalizes countries
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/aws/LogFileParser.java Defines parser contract
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/aws/FastlyLogFileParser.java Parses Fastly logs
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/aws/CloudFrontLogFileParser.java Parses CloudFront logs
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/aws/AwsDownloadRecordSource.java Reads S3 access logs
server/src/main/java/org/eclipse/openvsx/analytics/ingestion/aws/AccessLogRecord.java Converts access records
server/src/main/java/org/eclipse/openvsx/analytics/DownloadSeriesRow.java Defines repository row
server/src/main/java/org/eclipse/openvsx/analytics/DownloadSeriesRequest.java Defines series query
server/src/main/java/org/eclipse/openvsx/analytics/DownloadSeriesPoint.java Defines series point
server/src/main/java/org/eclipse/openvsx/analytics/DownloadSeriesJson.java Defines API payload
server/src/main/java/org/eclipse/openvsx/analytics/DownloadSeriesInterval.java Defines bucket intervals
server/src/main/java/org/eclipse/openvsx/analytics/DownloadSeriesGroupBy.java Defines grouping dimensions
server/src/main/java/org/eclipse/openvsx/analytics/DownloadEvent.java Defines analytics event
server/src/main/java/org/eclipse/openvsx/analytics/DownloadAnalyticsService.java Builds and caches series
server/src/main/java/org/eclipse/openvsx/analytics/DownloadAnalyticsRepository.java Defines storage interface
server/src/main/java/org/eclipse/openvsx/analytics/DownloadAnalyticsConfiguration.java Wires analytics services
server/src/main/java/org/eclipse/openvsx/analytics/DownloadAnalyticsAPI.java Exposes public endpoint
server/src/dev/resources/application.yml Enables development analytics
server/scripts/generate-properties.sh Generates Timescale properties
server/build.gradle Supports test image overrides
docker-compose.yml Adds TimescaleDB service
doc/development.md Documents development databases
deploy/openshift/application.yml Documents OpenShift settings
deploy/kubernetes/configmap.yaml Documents Kubernetes settings
deploy/docker/configuration/application.yml Documents Docker settings

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread webui/src/pages/admin-dashboard/admin-dashboard.tsx
Comment thread server/src/main/java/org/eclipse/openvsx/analytics/DownloadAnalyticsAPI.java Outdated
gnugomez and others added 4 commits August 31, 2026 10:36
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Jordi Gómez Hidalgo <31970428+gnugomez@users.noreply.github.com>
Signed-off-by: Thomas Neidhart <thomas.neidhart@eclipse-foundation.org>
@netomi

netomi commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

This PR changes too many things at the same time. It is not clear to me what happens when ovsx.analytics.enabled is set to false. Does then the original download log processing still work?

Need to split this PR into smaller pieces to make the change manageable without introducing a regression for the existing functionality or when ovsx.analytics.enabled is set to false. Will create separate PRs based on that one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants