DC/OS Software Development Kit Status: Alpha

Kafka: Quick Start

Prerequisite

  1. Install a Kafka cluster. If you are using open source DC/OS, install a Kafka cluster with the following command from the DC/OS CLI. If you are using Enterprise DC/OS, you may need to follow additional instructions. See the Install and Customize section for more information.

    dcos package install beta-kafka

    Alternatively, you can install Kafka from the DC/OS web interface.

    Kafka will deploy with a default configuration. You can monitor deployment at the Services tab of the DC/OS web interface.

  2. Create a new topic.

     ```bash
 $ dcos beta-kafka topic create topic1
 ```

  3. Find Zookeeper and broker endpoint information.

     ```bash
 $ dcos beta-kafka endpoints zookeeper
 master.mesos:2181/dcos-service-kafka

 $ dcos beta-kafka endpoints broker
 {
   "address": [
     "10.0.3.226:1000",
     "10.0.3.98:1000",
     "10.0.0.120:1000"
   ],
   "dns": [
     "kafka-2-broker.kafka.autoip.dcos.thisdcos.directory:1000",
     "kafka-0-broker.kafka.autoip.dcos.thisdcos.directory:1000",
     "kafka-1-broker.kafka.autoip.dcos.thisdcos.directory:1000"
   ],
   "vip": "broker.kafka.l4lb.thisdcos.directory:9092"
 }
 ```

  4. Produce and consume data.

     ```bash
 # Create marathon app defintion
 $ cat <<'EOF' >> kafkaclient.json
 {
 "id": "/kafka-client",
 "instances": 1,
 "container": {
 "type": "MESOS",
 "docker": {
 "image": "wurstmeister/kafka:0.11.0.0"
 }
 },
 "cpus": 0.5,
 "mem": 256,
 "cmd": "sleep 100000"
 }
 EOF
    
 # Deploy marathon app definition
 $ dcos marathon app add kafkaclient.json
    
 # Produce single `Hello world` event
 $ dcos task exec kafka-client bash -c "export JAVA_HOME=/opt/jdk1.8.0_144/jre/; echo 'Hello, World.' | /opt/kafka_2.12-0.11.0.0/bin/kafka-console-producer.sh --broker-list broker.kafka.l4lb.thisdcos.directory:9092 --topic topic1"
    
 # Consume events from topic1
 $ dcos task exec kafka-client bash -c "export JAVA_HOME=/opt/jdk1.8.0_144/jre/; /opt/kafka_2.12-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic1 --from-beginning"
 Hello, World.
 ```

See also Connecting clients.