New Relic GitLab Exporters

Monitor GitLab with OpenTelemetry and New Relic quickstarts

Overview

Monitor your GitLab CI/CD pipelines with comprehensive observability using New Relic's GitLab exporters. This project provides two complementary exporters that collect different types of data from your GitLab instance and send it to New Relic using OpenTelemetry.

What You Can Monitor

• Pipeline Performance: Track pipeline duration, success rates, and failure patterns
• Job-Level Insights: Monitor individual job execution times and resource usage
• Distributed Tracing: Visualize complete pipeline executions as distributed traces with logs in context
• DORA Metrics: Track deployment frequency, lead time, and failure recovery (GitLab Ultimate required)
• Runner Metrics: Monitor GitLab runner performance and availability
• Bridge Processing: Full support for downstream pipeline triggers and multi-project pipelines

GitLab Dashboard

Architecture

This project consists of two main components:

1. New Relic Exporter

• Purpose: Exports individual pipeline execution data as distributed traces
• Data Type: Traces and logs for specific pipeline runs
• Use Case: Detailed analysis of individual pipeline executions
• Deployment: Typically runs as part of GitLab CI/CD pipelines

2. New Relic Metrics Exporter

• Purpose: Exports aggregated metrics and historical data
• Data Type: Metrics, project statistics, and runner information
• Use Case: High-level monitoring and alerting across multiple projects
• Deployment: Can run standalone or as scheduled GitLab pipeline
• Data Collection:
  • Project metadata and event data are filtered by GLAB_EXPORT_LAST_MINUTES (default: 60 minutes) to collect only recent activity
  • Set GLAB_EXPORT_ALL_PROJECTS=true to export all historical data regardless of activity window

Configuration System

This project uses a centralized configuration management system with type safety and validation. Configuration is loaded from environment variables and provides backward compatibility with the previous global variables approach.

Key Features

• Type-safe configuration with dataclasses and validation
• Automatic New Relic region detection (US/EU) based on API key
• Environment variable validation with clear error messages
• Health monitoring and configuration validation
• Backward compatibility with deprecation warnings for migration

New Relic Exporter

All tests should pass. There are no dummy tests included; all tests validate real functionality.

Variables	Description	Optional	Values	Default
OTEL_EXPORTER_OTEL_ENDPOINT	New Relic OTEL endpoint including port	True	String	Auto-detected: "https://otlp.nr-data.net:4318" (US) or "https://otlp.eu01.nr-data.net:4318" (EU)
GLAB_TOKEN	MASKED - Token to access gitlab API	False	String	None
NEW_RELIC_API_KEY	MASKED - New Relic License Key	False	String	None
GLAB_EXPORT_LOGS	Export job logs to New Relic	True	Boolean	True
GLAB_ENDPOINT	Gitlab API endpoint	True	String	"https://gitlab.com"
GLAB_LOW_DATA_MODE	export only bear minimum data (only recommended during testing)	True	Boolean	False
GLAB_CONVERT_TO_TIMESTAMP	converts datetime to timestamp	True	Boolean	False
GLAB_EXCLUDE_JOBS	Comma-separated list of job or bridge names or stages to exclude from export (e.g. "build,test,deploy,bridge-stage")	True	List*	None
GLAB_USE_NAMESPACE_SLUG	Use GitLab namespace slugs for service names instead of display names (e.g. "main-group/sub-group/project" vs "Main Group / Sub Group / Project")	True	Boolean	False
OTEL_EXPORTER_TYPE	OTEL exporter output target. otlp sends to New Relic, console prints to stderr, both does both (useful for debugging)	True	String	otlp
LOG_LEVEL	Logging level for structured logs	True	String	INFO

New Relic Metrics Exporter

Variables	Description	Optional	Values	Default
OTEL_EXPORTER_OTEL_ENDPOINT	New Relic OTEL endpoint including port	True	String	Auto-detected: "https://otlp.nr-data.net:4318" (US) or "https://otlp.eu01.nr-data.net:4318" (EU)
GLAB_ENDPOINT	Gitlab API endpoint	True	String	"https://gitlab.com"
GLAB_TOKEN	MASKED - Token to access gitlab API	False	String	None
NEW_RELIC_API_KEY	MASKED - New Relic License Key	False	String	None
GLAB_PROJECT_OWNERSHIP	Filter projects by ownership. Set to "true" to get only owned projects, "false" to get all accessible projects	True	Boolean	True
GLAB_PROJECT_VISIBILITIES	Project visibilities (public,private,internal)	False	List*	private
GLAB_DORA_METRICS	Export DORA metrics, requires Gitlab ULTIMATE	True	Bool	False
GLAB_EXPORT_PATHS	Project paths aka namespace full_path to obtain data from. Use this to filter by specific groups/namespaces (e.g., "my-group,another-group")	True	List*	None if running as standalone or CI_PROJECT_ROOT_NAMESPACE if running as pipeline schedule
GLAB_RUNNERS_INSTANCE	Obtain runners from gitlab instance instead of project only	True	String
GLAB_EXPORT_PROJECTS_REGEX	Regex to match project names against ".*" for all	False	Boolean	None
GLAB_EXPORT_PATHS_ALL	When True ignore GLAB_EXPORT_PATHS variable and export projects matching GLAB_EXPORT_PROJECTS_REGEX in any groups or subgroups	True	Boolean	False
GLAB_CONVERT_TO_TIMESTAMP	converts datetime to timestamp	True	Boolean	False
GLAB_EXPORT_LAST_MINUTES	Time window (in minutes) for collecting event data (pipelines, jobs, deployments, releases). When GLAB_EXPORT_ALL_PROJECTS=False, also applies to project metadata.	True	Integer	60
GLAB_EXPORT_ALL_PROJECTS	When True (default), export all historical data regardless of activity. When False, only export projects with activity within GLAB_EXPORT_LAST_MINUTES window.	True	Boolean	True
GLAB_ATTRIBUTES_DROP	Attribute keys to drop from all exported data (log records, spans, parsed entity attributes). Comma-separated. Applied at every export point including OTEL log records.	True	List*	None
GLAB_ATTRIBUTES_TO_KEEP	Attribute keys to promote to resource level (in addition to system defaults). Comma-separated. Resource-level attributes are always sent to New Relic. All other attributes are passed as log-record-level attributes, where OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT applies.	True	List*	None
GLAB_DIMENSION_METRICS	Extra dimensional metric attributes to add to each metric	True	List*	NONE Note the following attributes will always be set as dimensions regardless of this setting: status,stage,name
GLAB_RUNNERS_SCOPE	Get runners scope : all, active, paused, online, shared, specific (separated by comma)	True	List*	all
GLAB_STANDALONE	Set to True if not running as gitlab pipeline schedule	True	Boolean	False
GLAB_ENVS_DROP	Extra system environment variables to drop from span attributes	True	List*	Note the following environment variables will always be dropped regardless of this setting: NEW_RELIC_API_KEY,GLAB_TOKEN,CI_JOB_JWT,CI_JOB_JWT_V1,CI_JOB_JWT_V2,CI_JOB_TOKEN,CI_BUILD_TOKEN,CI_REGISTRY_PASSWORD,CI_DEPLOY_PASSWORD,CI_DEPENDENCY_PROXY_PASSWORD,CI_RUNNER_SHORT_TOKEN,CI_SERVER_TLS_CA_FILE,CI_SERVER_TLS_CERT_FILE,CI_SERVER_TLS_KEY_FILE,CI_RUNNER_TAGS,GIT_ASKPASS,CI_COMMIT_BEFORE_SHA,CI_BUILD_BEFORE_SHA,CI_BEFORE_SHA,GITLAB_FEATURES,OTEL_EXPORTER_OTEL_ENDPOINT,GLAB_EXPORT_PATHS,GLAB_EXPORT_PATHS_ALL,GLAB_EXPORT_PROJECTS_REGEX
GLAB_EXCLUDE_JOBS	Comma-separated list of job or bridge names or stages to exclude from export (e.g. "build,test,deploy,bridge-stage")	True	List*	None
GLAB_USE_NAMESPACE_SLUG	Use GitLab namespace slugs for service names instead of display names (e.g. "main-group/sub-group/project" vs "Main Group / Sub Group / Project")	True	Boolean	False
OTEL_EXPORTER_TYPE	OTEL exporter output target. otlp sends to New Relic, console prints to stderr, both does both (useful for debugging)	True	String	otlp
LOG_LEVEL	Logging level for structured logs	True	String	INFO
GLAB_API_WORKERS	Thread pool size for pipeline/deployment/environment/release list() API calls	True	Integer	50
GLAB_JOB_WORKERS	Thread pool size for per-pipeline job fetches (scales with pipeline volume)	True	Integer	20
GLAB_EXPORT_BATCH_SIZE	Number of projects processed per batch before draining the OTEL queue	True	Integer	100

*comma separated

Attribute Promotion: Attributes are split into two tiers. Resource-level attributes are always sent to New Relic. Log-record-level attributes are subject to OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT and may be dropped on high-volume exports.

The following are always promoted to resource level: id, project_id, pipeline_id, job_id, environment_id, deployment_id, release_id, service.name, gitlab.resource.type, gitlab.source, status, stage, online, failure_reason, entity.name, finished_at, description. Use GLAB_ATTRIBUTES_TO_KEEP to promote additional attributes to resource level.

Default configuration is based on using Gitlab runners with docker executor

If using Kubernetes executors instead, use the below configuration

image:
    name: docker.io/dpacheconr/gitlab-exporter:2.2.2
    entrypoint: [""]
  script:
    - python3 -u /app/main.py
    - echo "Done"

Development Setup

Requirements

The project uses pinned dependencies for reproducible builds:

• Dependencies: shared/requirements.txt

Install dependencies:

# Production dependencies
pip install -r shared/requirements.txt

Running Tests

The project includes comprehensive test coverage, covering:

• Configuration management and validation
• GitLab API integration
• New Relic integration
• Data transformation and processing
• Performance testing
• Error handling and edge cases
• Bridge and downstream pipeline processing
• OTEL attribute filtering and helpers

Run all tests:

python3 -m pytest tests/ -v

Run specific test modules:

# Configuration tests
python3 -m pytest tests/test_config_settings.py -v

# Main module tests
python3 -m pytest tests/test_main.py -v

# Integration tests
python3 -m pytest tests/test_gitlab_integration.py -v

# Processor tests
python3 -m pytest tests/test_*_processor.py -v

Run tests with coverage:

python3 -m pytest tests/ --cov=shared --cov=new_relic_exporter --cov=new_relic_metrics_exporter --cov-report=html

Quick Start

Prerequisites

1. GitLab Access Token: Create a GitLab access token with read_api scope
   • Personal Access Token (PAT): Create a personal access token with read_api scope
   • Group Access Token: If using a group access token (for non-user-based authentication), it must be created with the Owner role to access all data including Runner metrics. Note: Maintainer role will provide partial data but will not include Runner information.
2. New Relic License Key: Get your New Relic license key from your account settings

Option 1: Docker Deployment

New Relic Metrics Exporter (Standalone monitoring)

# Run standalone for continuous monitoring
docker run \
  -e GLAB_STANDALONE=True \
  -e GLAB_EXPORT_PATHS="your-namespace" \
  -e GLAB_EXPORT_PROJECTS_REGEX=".*" \
  -e GLAB_TOKEN="your_gitlab_token" \
  -e NEW_RELIC_API_KEY="your_newrelic_key" \
  docker.io/dpacheconr/gitlab-metrics-exporter:2.2.2

Option 2: GitLab CI/CD Integration

Add to your .gitlab-ci.yml:

# For pipeline tracing
new-relic-export:
  stage: .post
  image: docker.io/dpacheconr/gitlab-exporter:2.2.2
  script:
    - python3 -u /app/main.py
  variables:
    GLAB_TOKEN: $GITLAB_TOKEN
    NEW_RELIC_API_KEY: $NEW_RELIC_LICENSE_KEY
  when: always

# For metrics collection (scheduled pipeline)
new-relic-metrics:
  image: docker.io/dpacheconr/gitlab-metrics-exporter:2.2.2
  script:
    - python3 -u /app/main.py
  variables:
    GLAB_TOKEN: $GITLAB_TOKEN
    NEW_RELIC_API_KEY: $NEW_RELIC_LICENSE_KEY
    GLAB_EXPORT_PATHS: $CI_PROJECT_ROOT_NAMESPACE
  only:
    - schedules

Resources

• New Relic Quickstart: https://newrelic.com/instant-observability/gitlab
• Blog Tutorial: https://newrelic.com/blog/how-to-relic/monitor-gitlab-with-opentelemetry
• Docker Images:
  • docker.io/dpacheconr/gitlab-exporter:2.2.2
  • docker.io/dpacheconr/gitlab-metrics-exporter:2.2.2

Health Monitoring

Both exporters include comprehensive health monitoring and structured logging:

• Structured Logging: JSON-formatted logs with correlation IDs
• Performance Metrics: Built-in timing and performance tracking
• Error Handling: Graceful error handling with detailed error context
• Configuration Validation: Startup validation of all required settings
• Graceful OTEL Shutdown: All providers are explicitly shut down before exit, ensuring batched traces, logs, and metrics are fully exported and not lost
• OTEL Queue Monitoring: Active queue depth monitoring during export; warnings are emitted when the batch queue approaches capacity

Large-Scale Deployments

When exporting from GitLab instances with many projects (100+), the metrics exporter uses batched processing to prevent OTEL queue overflow:

• Projects are processed in batches of 100
• OTEL logs are flushed after each batch
• Queue depth is monitored between batches and logged

If you observe dropped telemetry at scale, set LOG_LEVEL=DEBUG to inspect queue stats and tune GLAB_EXPORT_LAST_MINUTES or GLAB_EXPORT_ALL_PROJECTS accordingly.

Security Considerations

• Token Security: All sensitive tokens are masked in logs
• Environment Variables: Sensitive environment variables are automatically filtered
• HTTPS: All New Relic endpoints use HTTPS by default
• Minimal Permissions: GitLab tokens require read_api scope
  • Personal Access Tokens (PAT): read_api scope is sufficient
  • Group Access Tokens: Require Owner role for full data access (including Runner metrics); Maintainer role provides limited access without Runner data

Troubleshooting

Common Issues

1. Authentication Errors (401 Unauthorized)

   • Problem: GitLabAPIError: Failed to retrieve projects for visibility private
   • Cause: Incorrect GLAB_PROJECT_OWNERSHIP configuration
   • Solution: Use GLAB_PROJECT_OWNERSHIP: "false" and filter by namespace using GLAB_EXPORT_PATHS: "your-group-name"
   • Verify GitLab token has read_api scope
   • Check New Relic license key format
   • Ensure correct GitLab endpoint URL

2. No Projects Found

   • Problem: "No projects found to export" in logs
   • Cause: Incorrect namespace filtering configuration
   • Solution: Check that GLAB_EXPORT_PATHS matches your GitLab group/namespace exactly
   • Use GLAB_EXPORT_PATHS_ALL: "true" to export all accessible projects for testing

3. No Data in New Relic

   • Verify OTEL endpoint is correct for your region
   • Check network connectivity to New Relic
   • Review structured logs for export errors

4. Performance Issues

   • Use GLAB_LOW_DATA_MODE=True for testing
   • Adjust GLAB_EXPORT_LAST_MINUTES for metrics exporter
   • Consider excluding jobs with GLAB_EXCLUDE_JOBS

5. Missing Runner Data

   • Problem: Runner metrics are not being exported
   • Cause: Insufficient token permissions when using Group Access Token
   • Solution: If using a Group Access Token, ensure it was created with the Owner role (not Maintainer). Personal Access Tokens with read_api scope do not have this limitation.

6. Missing Project Data (Standalone Deployment)

   • Problem: Projects without recent activity are not appearing on dashboards
   • Cause: GLAB_EXPORT_ALL_PROJECTS=false (default) only exports projects active within the GLAB_EXPORT_LAST_MINUTES window
   • Solution: Set GLAB_EXPORT_ALL_PROJECTS=true to export all project metadata regardless of activity, or increase GLAB_EXPORT_LAST_MINUTES to widen the time window.

Debug Mode

The project supports configurable logging levels via the LOG_LEVEL environment variable:

# Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
LOG_LEVEL=DEBUG

Contributing

We encourage your contributions to improve GitLab Exporters! Keep in mind when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project. If you have any questions, or to execute our corporate CLA, required if your contribution is on behalf of a company, please drop us an email at opensource@newrelic.com.

Development Guidelines

• All code must pass the comprehensive test suite (339 tests)
• Follow the existing code structure and patterns
• Add tests for new functionality
• Update documentation for configuration changes

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

License

GitLab Exporters are licensed under the Apache 2.0 License.

GitLab Exporters also use source code from third-party libraries. You can find full details on which libraries are used and the terms under which they are licensed in the third-party notices document.