Integrate Kafka with Google Cloud/Pub Sub

Set Up:

The set up of this lab is just like other labs. You will use the Google Cloud Shell in this lab. After opening the Google Console you have to activate your Google Cloud Shell following the instructions given.


Read though the intro section — I know, I know — this will help you in understanding the integration of Kafka with the Google Cloud Platform.

Setting up the environment:

The setup instructions can be a little confusing. Here is a tip: when you search for “Kafka” in the Marketplace you will see multiple similar results. You have to pick the right one in order to go further. You will see something like this:

Configure the Kafka VM instance

This section is all about running and understanding commands. You will configure your Kafka VM using SSH. Make sure you are entering your project ID wherever it is instructed. Find your project ID back in the lab manual.

Pub/Sub Topic and Subscription setup

In this section we will create a Pub/Sub Topic to-kafka from-kafka. ‘To-kafka’ will be communicating with from-kafka. In the next step we will create the subscription for to-kafka and also from-kafka. This step will ensure that communication happens between Kafka and GCP. We can say that Pub/Sub acts as the mediator between Kafka and GCP.

Start the Kafka VM application instance

This is a very crucial step in the lab: start the Kafka application! Follow the steps as they are given and you will be good to proceed in the lab. Please note that you have to perform the given steps in the SSH.

Data exchange between Kafka and Pub/Sub

In this section you will be working on a new SSH of Kafka instance. You will enter the Kafka console and add certain elements in the console. Once you have added the elements in the Kafka console you need to check if these elements are reflected successfully in the cloud shell. For this you will run the command gcloud pubsub subscriptions pull from-kafka — auto-ack — limit=10. Once you run this command it will take some time to sync with the Kafka console. You will get the results after running this command a couple of times. Once you see the output given in the lab instructions you are good to proceed further with testing Pub/sub to Kafka.

Pub/Sub to Kafka Testing

This is a very interesting task in which you will be verifying the data exchange between Pub/Sub to Kafka. You will run the commands in the Cloud Shell and see the output in the Kafka VM SSH. Your output will look like this

IOT Simulator — IoT core

In this section you will see how the IoT simulator works with Kafka. IRL (in real life) you can also use this for connecting to other devices. First you will make the set up ready in order to create a device registry. For this purpose you will be using git and cloning the repository which will help you to gain access to some specific lab tools. You will also create a cryptographic key which would allow IoT devices to connect to the Cloud Pub/Sub.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store