BAM: An Asynchronous Microservices Load Balancer

Kafka

The Kafka config files are in k8s/clusters folder. The deployment should work as:

kubectl apply -f zookeeper.yaml
kubectl apply -f kafka.yaml
kubectl apply -f kafka-svc.yaml

NOTE: Keep an eye out for KRAFT - Kafka with Raft instead of zookeeper.

Consumer

This is in the java/consumer folder.

Producer

This is in the java/producer folder.

The data (order)

This is in the java/order folder.

Compiling

The docker images are built with the Google jib tool and compiled with:

mvn compile jib:build

The docker config file (~/.docker/config.json) has to be set up for this to work:

{
        "auths": {
                "https://index.docker.io/v1/": {
                        "auth": "YWRtaW46cGFzc3dvcmQ="
                }
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/20.10.12 (linux)"
        }
}

YWRtaW46cGFzc3dvcmQ= is the base64 encoding of the docker user id and password - echo -n admin:password | base64.

Maven

Installation

sudo apt install maven -y

Move maven repo

Edit ~/.m2/settings.xml:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      https://maven.apache.org/xsd/settings-1.0.0.xsd">
  <localRepository>/mydata/.m2</localRepository>
</settings>

Create a project

mvn archetype:generate -DgroupId=com.github.ratnadeepb.eda -DartifactId=producer -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

Execute a program

mvn exec:java -Dexec.mainClass=com.github.ratnadeepb.kafka.OrderProducer

Add order to the producer and consumer projects

In the order directory,

mvn install # install into local maven repo

In the pom.xml of producer and consumer:

<dependency>
    <groupId>com.github.ratnadeepb.kafka</groupId>
    <artifactId>order</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>

Performing a clean reinstall

mvn clean install # clean previous install and reinstall

Update the project

mvn update project # update the project

Generate sources

mvn generate-sources # generate source if using avro and some such

InfluxDB

cd k8s/db/influxdb
kubectl apply -f influxdb-k8.yaml 

Forwarding the service port for influxdb

kubectl port-forward svc/influxdb 8086:8086

Forwarding over SSH

ssh -L 8086:localhost:8086 node0

Async Sidecar LoadBalancer

Kafka

Kafka on Mac

brew install kafka

This installs Kafka in /usr/local/Cellar/kafka. Manually, start the services.

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
kafka-server-start /usr/local/etc/kafka/server.properties

Topics

List topics

kafka-topics --list --bootstrap-server localhost:9092

Describe topic

kafka-topics --describe --bootstrap-server localhost:9092 --topic first-topic

Delete topic

kafka-topics --delete --bootstrap-server localhost:9092 --topic first-topic

Create topic

kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 10 --topic first-topic

Listen on topic

kafka-console-consumer --bootstrap-server localhost:9092 --topic first-topic

Produce to topic

kafka-console-producer --broker-list localhost:9092 --topic first-topic

Confluent

Java 11

Find Java 11 with, /usr/libexec/java_home -v 11

confluent-kafka-go prerequisite

sudo apt-get install -y librdkafka-dev

Starting Kafka

JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home confluent local services start

Stopping the services

confluent local services stop

Kubernetes Kafka

kubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --list

kubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --describe --topic OrderTopic

kubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --delete --topic OrderTopic

kubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --create --replication-factor 2 --partitions 10 --topic OrderTopic

Building Producer and Consumer

sudo docker build -t ratnadeepb/kafka-producer .

Referecens

Kafka and Zookeeper yaml files
Producer-Consumer Example using Kafka
Producer-Consumer Example using Nats
Rust producer-consumer
Kafka Consumer Group
Kafka Operating Modes
Java Producer Consumer
Weighted Consumer Group Assignor - Java
Lag based assignor
Install Influx
InfluxDB Java Client
Java Main
Java Influx