Extend telemetry demo to include logs and traces

.github/workflows/go-ci.yml

@@ -14,7 +14,7 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
-          fetch-depth: 0  # Get tags to allow build script to get build version
+          fetch-depth: 0  # Get tags to allow the build script to get a build version
 
       - name: Set up Go
         uses: actions/setup-go@v5

.github/workflows/golangci-lint.yml

@@ -14,7 +14,7 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
-          fetch-depth: 0  # Get tags to allow build script to get build version
+          fetch-depth: 0  # Get tags to allow the build script to get a build version
 
       - name: Set up Go
         uses: actions/setup-go@v5

.github/workflows/release.yml

@@ -12,7 +12,8 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
-          fetch-depth: 0  # Get tags to allow build script to get build version
+          ref: ${{ github.ref }}  # Checkout the specified tag
+          fetch-depth: 0  # Get tags to allow the build script to get a build version
 
       - name: Set up Go
         uses: actions/setup-go@v5

telemetry-demo/README.md

@@ -1,14 +1,17 @@
 # SCANOSS API Telemetry Demo
 
-This demo provides a complete, pre-configured telemetry stack to visualize SCANOSS API metrics using OpenTelemetry, Prometheus, and Grafana.
+This demo provides a complete, pre-configured telemetry stack to visualize SCANOSS API metrics using
+OpenTelemetry, Promtail, Prometheus, Loki, Tempo, and Grafana.
 
 ## What's Included
-
-✅ **SCANOSS API** with telemetry pre-enabled  
-✅ **OpenTelemetry Collector** to receive and export metrics  
-✅ **Prometheus** to store time-series data  
-✅ **Grafana** for advanced dashboards  
-✅ **Test SCANOSS engine** (no real scanner needed)
+- ✅ **SCANOSS API** with telemetry pre-enabled 
+- ✅ **Test SCANOSS engine** (no real scanner needed)
+- ✅ **OpenTelemetry Collector** to receive and export metrics/traces
+- ✅ **Promtail Collector** to receive and export logs
+- ✅ **Prometheus** to store time-series data
+- ✅ **Tempo** to store spans/traces data
+- ✅ **Loki** to store log data 
+- ✅ **Grafana** for advanced dashboards
 
 ## Quick Start
 
@@ -21,57 +24,103 @@ This demo provides a complete, pre-configured telemetry stack to visualize SCANO
 2. **Wait for services to start** (about 30-60 seconds)
 
 3. **Generate metrics by making API requests**:
-   
+Run a continuous scan (every 2 seconds):
    ```bash
-   # Scan every 0.5 seconds - let it run to see metrics accumulate
-   watch -n 0.5 "echo 'file=056f9b95f439d915bd3d81ceee9ccf9a,1234,test.js' | \
+   # Scan every 2 seconds - let it run to see metrics accumulate
+   watch -n 2.0 "echo 'file=056f9b95f439d915bd3d81ceee9ccf9a,1234,test.js' | \
      curl -s -X POST 'http://localhost:5443/scan/direct' \
      -F 'file=@-;filename=test.wfp'"
    ```
+Or run a single one-off scan:
+```bash
+echo 'file=056f9b95f439d915bd3d81ceee9ccf9a,1234,test.js' | curl -s -X POST 'http://localhost:5443/scan/direct' -F 'file=@-;filename=test.wfp' 
+```
 
-
-## Where to See Metrics
-
-### Option 1: Prometheus UI (Built-in, Simple)
+## Visualisation
+The complete observability stack is configured to export metrics, logs, and traces. This can all be visualised in Grafana.
+To login to Grafana, please browse the following URL:
+- http://localhost:3000
+- No username/password required
+
+From there, it is possible to explore the metrics, logs, and traces.
+
+### Where to See Logs
+To view logs, please browse the following URL:
+1. http://localhost:3000/a/grafana-lokiexplore-app/explore/job/scanoss-api/logs
+2. Select `job` from the "Labels" dropdown and either set equal to `scanoss-api` or leave it blank
+3. And select the logs from below 
+4. From here, it is possible to filter and explore the logs.
+
+### Where to See Traces
+To view traces, please browse the following URL:
+1. http://localhost:3000/explore
+2. Select "Tempo" from the "Outline" dropdown
+3. Select one of the following Query types:
+   3.1 Select `Query type` "Search" and the time window and the list of Traces below
+   3.2 Select "TraceQL" from the Query type and enter a `trace_id` to search for
+4. These trace IDs can then be queried in the Loki logs to see detailed information about the trace.
+
+### Where to See Metrics
+#### Option 1: Grafana (Advanced Dashboards)
+1. **URL**: http://localhost:3000/a/grafana-metricsdrilldown-app/drilldown
+2. Select "prometheus" from the "Data Source" dropdown
+3. Type `scanoss` into the "Search metric" field to list all SCANOSS metrics
+4. Select an appropriate time window
+
+#### Option 2: Prometheus UI (Built-in, Simple)
 - **URL**: http://localhost:9090
 - **Usage**: 
-  1. Click "Graph" tab
+  1. Click the "Graph" tab
   2. Try queries like:
      - `scanoss_api_scan_file_count_total` (total files scanned)
      - `rate(scanoss_api_scan_file_count_total[5m])` (requests per second)
 
-### Option 2: Grafana (Advanced Dashboards)
-- **URL**: http://localhost:3000
-- **Login**: admin / admin
-- **Setup**:
-  1. Connections -> Add New Connections → Prometheus
-  2. URL: `http://prometheus:9090`
-  3. Create dashboard with your metrics
-
-## Available Metrics to Query
+### Data Sources
+If any datasources are not automatically configured, please configure them manually:
 
-Try these queries in Prometheus:
+#### Prometheus
+- **Setup**:
+    1. Connections -> Add New Connections → Prometheus
+    2. URL: `http://prometheus:9090`
 
-```promql
-# Total API requests over time
-scanoss_api_scan_file_count_total
+- A list of possible queries can be found in [Available Metrics to Query](#available-metrics-to-query)
 
-# Request rate (requests per second)
-rate(scanoss_api_scan_file_count_total[5m])
+#### Loki
+- **Setup**:
+    1. Connections -> Add New Connections → Loki
+    2. URL: `http://loki:3100`
 
-# Files scanned in last hour  
-increase(scanoss_api_scan_file_count_total[1h])
+#### Tempo
+- **Setup**:
+    1. Connections -> Add New Connections → Tempo
+    2. URL: `http://tempo:3200`
 
-# Total bytes scanned
-scanoss_api_scan_file_size_total
 
-# License requests
-scanoss_api_license_req_count_total
-```
+## Available Metrics to Query
 
+Try these queries in Prometheus:
+- Total API requests over time
+  ```promql
+  scanoss_api_scan_file_count_total
+  ```
+-  Request rate (requests per second)
+  ```promql
+  rate(scanoss_api_scan_file_count_total[5m])
+  ```
+- Files scanned in last hour
+  ```promql
+  increase(scanoss_api_scan_file_count_total[1h])
+  ```
+- Total bytes scanned
+  ```promql
+    scanoss_api_scan_file_size_total
+  ```
+- License requests
+  ```promql
+  scanoss_api_license_req_count_total
+  ```
 
 ## Stop the Demo
-
 ```bash
 # Stop all services
 docker compose down
@@ -80,17 +129,19 @@ docker compose down
 docker compose down -v
 ```
 
-
 ## Next Steps
-
 After exploring this demo, configure telemetry for your production API using [TELEMETRY_CONFIG.md](./TELEMETRY_CONFIG.md).
 
 ## Files in This Demo
 
-| File | Purpose |
-|------|---------|
-| `docker-compose.yml` | Orchestrates the complete telemetry stack |
-| `otel-collector-config.yml` | Configures OpenTelemetry Collector pipelines |
-| `prometheus.yml` | Defines Prometheus scrape targets |
-| `config/app-config-demo.json` | API configuration with telemetry enabled |
-| `TELEMETRY_CONFIG.md` | Production telemetry configuration guide |
+| File                          | Purpose                                      |
+|-------------------------------|----------------------------------------------|
+| `docker-compose.yml`          | Orchestrates the complete telemetry stack    |
+| `otel-collector-config.yml`   | Configures OpenTelemetry Collector pipelines |
+| `promtail-config.yml`         | Configures Promtail log Collector pipeline   |
+| `loki.yaml`                   | Defines Loki logging setup                   |
+| `prometheus.yml`              | Defines Prometheus scrape targets            |
+| `tempo.yml`                   | Defines Tempo traces setup                   |
+| `grafana-datasources.yaml`    | Data source configuration for Grafana        |
+| `config/app-config-demo.json` | API configuration with telemetry enabled     |
+| `TELEMETRY_CONFIG.md`         | Production telemetry configuration guide     |

telemetry-demo/TELEMETRY_CONFIG.md

@@ -22,14 +22,12 @@ Edit your service configuration file and change:
 ```json
 "Telemetry": {
   "Enabled": false,    // ← Change to true
-  "ExtraMetrics": false, // ← Change to true  
   "OltpExporter": "localhost:4317"
 }
 ```
 ### Option 2: Environment Variables
 ```bash
 export OTEL_ENABLED=true
-export OTEL_EXTRA=true  
 export OTEL_EXPORTER_OLTP=localhost:4317
 ```
 
@@ -47,17 +45,17 @@ OTEL_EXPORTER_OLTP=localhost:4317
 
 The following metrics are exposed by the SCANOSS API (defined in [utils_service.go](https://github.com/scanoss/api.go/blob/main/pkg/service/utils_service.go)):
 
-| Metric Name | Type | Description 
-|------------|------|-------------
-| `scanoss-api.scan.file_count` | Counter | Files received per scan request 
-| `scanoss-api.scan.file_size` | Counter | Total bytes scanned 
-| `scanoss-api.contents.req_count` | Counter | File contents requests 
-| `scanoss-api.license.req_count` | Counter | License details requests 
-| `scanoss-api.attribution.req_count` | Counter | Attribution requests 
-| `scanoss-api.scan.req_time` | Histogram | Scan duration (ms) 
-| `scanoss-api.scan.file_time` | Histogram | Per-file scan time (ms) 
-| `scanoss-api.scan.req_time_sec` | Histogram | Scan duration (seconds) 
-| `scanoss-api.scan.file_time_sec` | Histogram | Per-file scan time (seconds) 
+| Metric Name                         | Type      | Description                     |
+|-------------------------------------|-----------|---------------------------------|
+| `scanoss-api.scan.file_count`       | Counter   | Files received per scan request |
+| `scanoss-api.scan.file_size`        | Counter   | Total bytes scanned             |
+| `scanoss-api.contents.req_count`    | Counter   | File contents requests          |
+| `scanoss-api.license.req_count`     | Counter   | License details requests        |
+| `scanoss-api.attribution.req_count` | Counter   | Attribution requests            |
+| `scanoss-api.scan.req_time`         | Histogram | Scan duration (ms)              |
+| `scanoss-api.scan.file_time`        | Histogram | Per-file scan time (ms)         |
+| `scanoss-api.scan.req_time_sec`     | Histogram | Scan duration (seconds)         |
+| `scanoss-api.scan.file_time_sec`    | Histogram | Per-file scan time (seconds)    |
 
 ## Metric Name Translation
 

telemetry-demo/config/app-config-demo.json

@@ -4,20 +4,20 @@
     "Port": "5443",
     "Addr": "",
     "Debug": true,
-    "Trace": true,
-    "Mode": "dev"
+    "Trace": false,
+    "Mode": "prod"
   },
   "Logging": {
-    "DynamicLogging": true,
+    "DynamicLogging": false,
     "DynamicPort": "localhost:60085",
     "OutputPaths": [
-      "stderr"
+      "stderr", "/var/log/scanoss/api/scanoss-api-prod.log"
     ],
     "ConfigFile": ""
   },
   "Telemetry": {
     "Enabled": true,
-    "ExtraMetrics": true,
+    "ExtraMetrics": false,
     "OltpExporter": "otel-collector:4317"
   },
   "Scanning": {