Something that doesn’t get talked about much when you’re starting out with Kafka is monitoring. Most initial thoughts go into topic log timeouts and maximum message sizes.
JMX is your friend
Kafka emits a lot of JMX information, this gives us a very good indication of the internal measurements that are happening. To tap in to this gold mine of information we need to do some very basic settings (well, it’s one setting) to get this information.
Ladies and Gentlemen, start your brokers.
Let’s get this show on the road, it’s very easy.
First of all let’s startup Zookeeper.
$ $KAFKA_HOME/bin/zookeeper-server-start.sh config/zookeeper.properties
Open another terminal window tab. Before you start your server, export the JMX_PORTenvironment setting and use a free port, I’m going to use 9999.
$ export JMX_PORT=9999
Once this is set you can start the Kafka server.
$ bin/kafka-server-start.sh config/server.properties
Last thing to do is test that our JMX setup is working properly. I’m just using the plain old, but handy all the same, jconsolefrom the Java Development Kit.
$ jconsole localhost:9999
You will see the JConsole window appear, now if you click on the MBeans tab you’ll see all the Kafka JMX metrics. One you should really be keeping an eye on is kafka.server:type=ReplicaManager,name=UnderReplicatedPartitionsbut I’ll talk more on that another time.
It's All A Bet - MLOps, DevOps, AI and Synthetic Data Design 