Mostly while working with Kafka topics we create, drop, and recreate them but sometimes Kafka topics behave in an intended way. For example, after executing the drop command when we get the same “Gold Standard Message” that Topic is marked for deletion but when you check the topic is still present. This situation ideally doesn’t come when you have successfully deleted the topics in past but due to certain issues, this problem can still pop-up. The list below would give you the right sequence of steps to follow while deleting a topic from Kafka (0.10.0). How to delete a topic from Kafka:

1. Ensure that you have the topic deletion privileges enabled by setting:

delete.topic.enable=true

This Enables topic deletion using admin tools. If you don’t have this set to true, then deleting topics through the admin tools has no effect at all.

 

2. Disable auto topic creation by setting

auto.create.topics.enable=false

This setting enables the auto-creation of topics. If this is set to true, then any attempt to produce, consume, or fetch metadata for a non-existent topic would automatically create the topic with the default replication factor and number of partitions which is what we don’t want. Because if this is true and you try to get configuration detail of any specific topic or some other job (assuming being in the development environment) consuming the messages from or publishing the messages to the same topic then it would be recreated even if it was deleted earlier.

 

3. Finally execute the command to delete the topic as:

kafka-topics --zookeeper ${zookeeper_servers} --delete --topic ${kafka_topic}
Topic ${kafka_topic} is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

You should now be able to see the message stating that the topic is marked for deletion. If you have followed the above steps sequentially then the topic should have been deleted.

 

4. But sometimes due to network issues (or metafile corruption, metafile lock), the Kafka topic’s metadata is not deleted from zookeeper servers. The only option left here is to hard delete the leftover metadata information which can be easily done using the following commands:

kafka/bin/zookeeper-shell.sh ${zookeeper_servers} rmr /kafka/brokers/topics/${kafka_topic}
kafka/bin/zookeeper-shell.sh ${zookeeper_servers} rmr /kafka/config/topics/${kafka_topic}
kafka/bin/zookeeper-shell.sh ${zookeeper_servers} rmr /kafka/admin/delete_topics/${kafka_topic}

The diagram below depicts zookeeper znodes used/maintained by Kafka and you can easily identify which znodes hierarchy we want to remove to permanently hard delete a Kafka topic:

For more detailed information on Kafka znodes and their purpose watch this space for my upcoming blog.

Hopefully, you now know how to delete a topic from Kafka better and your issues with Kafka topic deletion should be resolved with this last step (if it was still a concern after normal deletion).

Do let me in case you still face any issue.