Tail sampling Benchmark

This repository contains the files utilized to measure the footprint of the collector when applying Tail sampling decisions

this tutorial will also utilize the OpenTelemetry Operator with:

• the OpenTelemetry Demo
• the hipster-shop
• istio All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

GKE 1.Create a Google Cloud Platform Project

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

GKE 2.Create a GKE cluster

ZONE=europe-west3-a
NAME=collector-tailsampling-benchmark
gcloud container clusters create ${NAME} --zone=${ZONE} --machine-type=e2-standard-4 --num-nodes=2 --monitoring=NONE --logging=NONE

EKS 2.Create a eks cluster

First run the following cmd:

ZONE=eu-west-3
OWNER=henrik.rexed

• For all the tests that are not related to cilium ( istio, ambientmesh, traefikmesh, kuma, linkerd) :

sed -i  '' "s,CLUSTER_NAME_TO_REPLACE,$NAME,"  cluster/cluster.yaml
sed -i  '' "s,REGION_TO_REPLACE,$ZONE,"  cluster/cluster.yaml
sed -i  '' "s,OWNER_TO_REPLACE,$OWNER,"  cluster/cluster.yaml
eksctl create cluster -f cluster/cluster.yaml

Getting started

1. Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

2. Istio

1. Download Istioctl

curl -L https://istio.io/downloadIstio | sh -

This command download the latest version of istio ( in our case istio 1.18.2) compatible with our operating system. 2. Add istioctl to you PATH

cd istio-1.26.2

this directory contains samples with addons . We will refer to it later.

export PATH=$PWD/bin:$PATH

3.Clone Github repo

git clone https://github.com/henrikrexed/tailsampling
cd tailsampling

4.Deploy environment

chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" --agentype "${TYPE}"

1. Tests with no restriction and 2 policies( latency & error code)

1. cosntant load

The applications are already deployed with a constant load.
Let's wait 3h to collect enought data

2. Run load test

 kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo
 kbuectl apply -f hipstershop/loadtest_job.yaml -n hipser-shop

2. Tests with no restriction and 3 policies ( latency & advanced rule using and operator)

 kubectl delete -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl delete -f hipstershop/loadtest_job.yaml -n hipser-shop
 kubectl apply -f opentelemetry/collector/openTelemetry-manifest_statefulset_sampling_policy2.yaml

1. cosntant load

The applications are already deployed with a constant load.
Let's wait 3h to collect enought data

2. Run load test

 kubectl apply -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl apply -f hipstershop/loadtest_job.yaml -n hipser-shop

3. Tests with no restriction and 4 policy ( latency , advanced rule using and operator, error code, probabilistic)

 kubectl delete -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl delete -f hipstershop/loadtest_job.yaml -n hipser-shop
 kubectl apply -f opentelemetry/collector/openTelemetry-manifest_statefulset_sampling_policy4.yaml

1. cosntant load

The applications are already deployed with a constant load.
Let's wait 3h to collect enought data

2. Run load test

 kubectl apply -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl apply -f hipstershop/loadtest_job.yaml -n hipser-shop

4. Tests with no restriction and 6 policies ( latency , advanced rule using and operator, error code, probabilistic, httpcode and grpc code)

 kubectl delete -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl delete -f hipstershop/loadtest_job.yaml -n hipser-shop
 kubectl apply -f opentelemetry/collector/openTelemetry-manifest_statefulset_sampling_policy5.yaml

1. cosntant load

The applications are already deployed with a constant load.
Let's wait 3h to collect enought data

2. Run load test

 kubectl apply -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl apply -f hipstershop/loadtest_job.yaml -n hipser-shop

5. Tests with no restriction and 6 policies and cache configured

 kubectl delete -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl delete -f hipstershop/loadtest_job.yaml -n hipser-shop
 kubectl apply -f opentelemetry/collector/openTelemetry-manifest_statefulset_sampling_policy6.yaml

1. cosntant load

The applications are already deployed with a constant load.
Let's wait 3h to collect enought data

2. Run load test

 kubectl apply -f opentelemetry/collector/loadtest_job.yaml -n otel-demo
 kbuectl apply -f hipstershop/loadtest_job.yaml -n hipser-shop