OBE-8859: ip whitelist source only (PART 1)

[yoonlee-s1]

left out windows event forwarder for now, may need change in dataplane_private.

Sources implemented so far: tcp, splunk HEC, firehose, prometheus remote write, HTTP server (push), udp, datadog agent, OpenTelemetry

[yoonlee-s1]

./target/debug/vector --config test.toml

test.toml:

type = "socket"
mode = "udp"
address = "0.0.0.0:9000"
permit_origin = ["127.0.0.1/32", "127.0.0.22/32"]

[sinks.console]
type = "console"
inputs = ["my_udp"]
encoding.codec = "text"

echo "hello from udp" | nc -u 127.0.0.1 9000

also tried permit_origin ["127.0.0.1/32"] (message received), ["127.0.0.22/32"] (no messages received)

[yoonlee-s1]

comprehensive test.
test.toml

# ---- Socket: UDP ----
[sources.test_udp]
type = "socket"
mode = "udp"
address = "0.0.0.0:9000"
permit_origin = ["127.0.0.1/32"]

# ---- Socket: TCP ----
[sources.test_tcp]
type = "socket"
mode = "tcp"
address = "0.0.0.0:9001"
permit_origin = ["127.0.0.1/32"]

# ---- HTTP Server ----
[sources.test_http]
type = "http_server"
address = "0.0.0.0:9002"
permit_origin = ["127.0.0.1/32"]

# ---- Prometheus Remote Write ----
[sources.test_prom_rw]
type = "prometheus_remote_write"
address = "0.0.0.0:9003"
permit_origin = ["127.0.0.1/32"]

# ---- Prometheus Pushgateway ----
[sources.test_prom_pg]
type = "prometheus_pushgateway"
address = "0.0.0.0:9004"
permit_origin = ["127.0.0.1/32"]

# ---- Splunk HEC ----
[sources.test_splunk]
type = "splunk_hec"
address = "0.0.0.0:9005"
token = "test-token"
permit_origin = ["127.0.0.1/32"]

# ---- AWS Kinesis Firehose ----
[sources.test_firehose]
type = "aws_kinesis_firehose"
address = "0.0.0.0:9006"
store_access_key = false
permit_origin = ["127.0.0.1/32"]

# ---- Datadog Agent ----
[sources.test_datadog]
type = "datadog_agent"
address = "0.0.0.0:9007"
permit_origin = ["127.0.0.1/32"]

# ---- OpenTelemetry ----
[sources.test_otel]
type = "opentelemetry"
permit_origin = ["127.0.0.1/32"]

[sources.test_otel.grpc]
address = "0.0.0.0:9008"

[sources.test_otel.http]
address = "0.0.0.0:9009"

# ---- Heroku Logs ----
[sources.test_heroku]
type = "heroku_logs"
address = "0.0.0.0:9010"
permit_origin = ["127.0.0.1/32"]

# ---- Sink: all to console ----
[sinks.console]
type = "console"
inputs = ["test_*"]
encoding.codec = "json"

[yoonlee-s1]

send request from another terminal.

gRPC is skipped. need brew install grpcurl

grpcurl -plaintext -d '{"resource_logs":[{"scope_logs":[{"log_records":[{"body":{"string_value":"hello otel"}}]}]}]}' \ 127.0.0.1:9008 opentelemetry.proto.collector.logs.v1.LogsService/Export

test_requests.sh

#!/bin/bash
# Test commands for each source with permit_origin
# Run vector first: ./target/debug/vector --config test.toml

echo "=== UDP (port 9000) ==="
echo "hello udp" | nc -u -w1 127.0.0.1 9000

echo "=== TCP (port 9001) ==="
echo "hello tcp" | nc -w1 127.0.0.1 9001

echo "=== HTTP Server (port 9002) ==="
curl -s -X POST http://127.0.0.1:9002 -d "hello http"

echo "=== Prometheus Remote Write (port 9003) - connectivity test ==="
curl -s -X POST http://127.0.0.1:9003/api/v1/write -d "test"
echo " (any response = connection accepted, timeout = blocked)"

echo "=== Prometheus Pushgateway (port 9004) ==="
curl -s -X POST http://127.0.0.1:9004/metrics/job/test \
  -H "Content-Type: text/plain" \
  -d 'test_metric{label="value"} 42'

echo "=== Splunk HEC (port 9005) ==="
curl -s -X POST http://127.0.0.1:9005/services/collector \
  -H "Authorization: Splunk test-token" \
  -d '{"event":"hello splunk"}'

echo "=== AWS Kinesis Firehose (port 9006) ==="
curl -s -X POST http://127.0.0.1:9006 \
  -H "Content-Type: application/json" \
  -H "X-Amz-Firehose-Request-Id: test-123" \
  -H "X-Amz-Firehose-Protocol-Version: 1.0" \
  -H "X-Amz-Firehose-Source-Arn: arn:aws:firehose:us-east-1:123456789:deliverystream/test" \
  -d '{"requestId":"test-123","timestamp":1234567890,"records":[{"data":"aGVsbG8gZmlyZWhvc2U="}]}'

echo "=== Datadog Agent (port 9007) ==="
curl -s -X POST http://127.0.0.1:9007/api/v2/logs \
  -H "Content-Type: application/json" \
  -H "dd-api-key: test-key" \
  -d '[{"message":"hello datadog","status":"info"}]'

echo "=== OpenTelemetry HTTP (port 9009) - connectivity test ==="
curl -s -X POST http://127.0.0.1:9009/v1/logs -d "test"
echo " (any response = connection accepted, timeout = blocked)"

echo "=== Heroku Logs (port 9010) ==="
curl -s -X POST http://127.0.0.1:9010/events \
  -H "Content-Type: text/plain" \
  -d "hello heroku"

[janmejay-s1]

May be make one of these just the address (kill /32 suffix) just for completeness. Not everywhere, just one positive test and one negative test suffices.

[janmejay-s1]

Often it is possible to avoid so much duplication. One way I can think of is to create a helper that accepts permit-origin fragment (or in case of toml config-parsing tests, accepts the entire toml string) and returns the receive channel. Then the test can simply recv the message or assert timeout.

[janmejay-s1]

assert the message was received (don't ignore _recv).

[janmejay-s1]

try to dedup here and in other tests (should be a pretty quick thing with coding assistant)

[janmejay-s1]

Increment a metric (this can lead to a lot of confusion). Preferably also add a log-line.

[janmejay-s1]

In these cases toml can make the test really short.