Enable SSL on Kafka

To install Kafka, follow the steps mentioned below:

1. Download Kafka binary .tar.gz version 2.12-3.6.1 from the below URL.

   Index of /dist/kafka

2. Extract the tar.gz using the below command:

   $ tar -xvf kafka\_2.12-3.6.1.tgz -C <<installationDir>>
   $ cd <<installationDir>>/<<extractedDir>>
   

   To enable SSL on Kafka, follow the steps mentioned below:

   Perform the following steps on each node in the cluster:

   Generating Node Certificates:

3. Create a certificate authority for your Kafka cluster. Substitute the <DOMAIN\_NAME> with your machine’s domain name on all nodes with the Keystore password and validity.

   👉

   Passwords should be same.

   $keytool -genkeypair -keystore kafka.keystore -keyalg RSA -alias <<Domain Name >> -dname "CN=$(hostname -f)" -storepass <<password>> -keypass <<password>> -validity 32767
   

4. On all the nodes, rename the keystore file to jks file.

   $mv kafka.keystore kafka.jks
   

5. Generate a self signed certificate on all the nodes.

   $keytool -export - alias <<Domain name of host>> -keystore kafka.jks -rfc -file selfsigned.cer
   

6. Rename selfsigned.cer to selfsigned.pem

   $mv selfsigned.cer selfsigned<hostname/ip>.pem
   

7. Copy the selfsigned.pem file from all the nodes to one of the Kafka servers where the trust store file will be generated.

   $scp selfsigned<hostip/name>.pem <<Ip\_address of Kafka server >>:/path\_of\_certificate
   

8. Import the self-signed certificate to truststore on node where truststore file will be generated.

   $keytool-keystore truststore.jks-import-alias<<Hostname\_of\_the\_node>> -file selfsigned<<hostname/ip>>.pem
   

9. Copy the truststore files from the server to all the other nodes in the same path.

   $scp truststore.jks <hostname/ip of kafka brokers>:/path\_of\_certificate
   

10. Place the kafka.jks in the same path as the certificate. Change the file permissions of Kafka.jks and truststore.jks on all nodes.

    $chmod 777 kafka.jks truststore.jks
    

Configure SSL on all nodes of the Kafka Cluster

1. Enable TLS and specify the information required to access the node’s certificate.

   Add the following information to <<installationDir>>/<<extractedDir>>/config/server.properties file on each node.

   listeners=SSL://<<hostname>>:9093
   advertised.listeners=SSL://<<hostname>>:9093
   ssl.keystore.location=<<kafka.jks file location>>
   ssl.keystore.password= <<keystore password>>
   ssl.key.password=<<key password>>
   ssl.truststore.location=<<truststore.jks file location>>
   ssl.truststore.password=<<truststore password>>
   security.inter.broker.protocol = SSL
   

2. Configure more properties in <<installationDir>>/<<extractedDir>>/config/server.properties file under the extracted folder.

   👉

   The brokerid should be different for each kafka broker.

   $broker.id=
   log.dirs=
   zookeeper.connect= <<Ip address of zookeeper>>:2181
   

To start the Kafka servers on all nodes

$ nohup bin/kafka-server-start.sh config/server.properties &

Additional Certificates on SSL Enabled Kafka #

If you want to use a Kafka Alert operator in the Workflow, It requires an additional CA File, a Cert File, and a Key File.

Follow the steps below to generate these files:

1. Get the keystore jks file that you generated while enabling SSL on Kafka in the earlier steps.

2. Run the command prompt and get the alias name by running the below command:

   keytool -list -rfc -keystore keystore.jk1
   

3. Extract the client certificate.

   keytool -exportcert -alias <alias\_name> -keystore keystore.jks -rfc -file certificate.pem
   

4. Extract the client key.

   keytool -v -importkeystore -srckeystore keystore.jks -srcalias <alias\_name> -destkeystore cert\_and\_key.p12 -deststoretype PKCS12
   

5. Following command will print the key. Copy and paste the output in key.pem file.

   openssl pkcs12 -in cert\_and\_key.p12 -nocerts –nodes
   

6. At last, extract the CARoot certificate.

   keytool -exportcert -alias alias\_name -keystore keystore.jks -rfc -file CARoot.pem
   

These steps will generate the following files:

• CARoot.pem

• certificate.pem

• key.pem

You can place these files on the machine where Airflow is deployed and provide their path while creating Kafka Alert Operator.