In this tutorial we’ll be walking through step-by-step instructions on getting a Kafka service up and running. This tutorial assumes you’ve already setup a local cluster using the Quick Start guide and that you’re in the VM environment.
Steps:
-
Change directory into
dcos-commons
and create a new service by running:cd /dcos-commons && ./new-framework.sh examples/kafka
. This will create a clone of thehello-world
service which you should then modify. -
Next change directory to where the Kafka service definition is:
cd examples/kafka/src/main/dist/
-
With an editor of your choice open
svc.yml
. Notice that some values, such asNODE_COUNT
andFRAMEWORK_CPUS
are mustached.[0] -
The name of a task can be anything but since Kafka has brokers, change the type of the task in the Kafka pod from
node
tobroker
. Also, notice thatcount
is specified as a property of thekafka
pod. This indicates there will becount
kafka pods, each of which will run abroker
. -
Next, add the following property to the
broker
task in order to open up the necessary port, and mark it asadvertise
d so that it appears in service endpoint information.
ports:
broker-port:
port: 9092
advertise: true
- Now, let’s modify the command that will start each
broker
. This is also a property of thebroker
task:
cmd: "env && exec $MESOS_SANDBOX/kafka_2.11-0.10.0.0/bin/kafka-server-start.sh $MESOS_SANDBOX/kafka_2.11-0.10.0.0/config/server.properties"
- Adding this under
kafka
for eachbroker
will fetch the Kafka binary in order to execute the above command:
uris:
- "https://downloads.mesosphere.com/kafka/assets/kafka_2.11-0.10.0.0.tgz"
- The following property allows for custom configuration of each
broker
:
configs:
server-properties:
template: server.properties.mustache
dest: kafka_2.11-0.10.0.0/config/server.properties
- Save and exit the
svc.yml
file. It should look similar to the following:
name: {{FRAMEWORK_NAME}}
scheduler:
service_account: {{FRAMEWORK_PRINCIPAL}}
user: {{FRAMEWORK_USER}}
pods:
kafka:
count: {{NODE_COUNT}}
placement: '{{{NODE_PLACEMENT}}}'
uris:
- "https://downloads.mesosphere.com/kafka/assets/kafka_2.11-0.10.0.0.tgz"
tasks:
broker:
goal: RUNNING
cmd: "env && exec $MESOS_SANDBOX/kafka_2.11-0.10.0.0/bin/kafka-server-start.sh $MESOS_SANDBOX/kafka_2.11-0.10.0.0/config/server.properties"
cpus: {{NODE_CPUS}}
memory: {{NODE_MEM}}
ports:
broker-port:
port: 9092
advertise: true
volume:
path: "kafka-container-path"
type: {{NODE_DISK_TYPE}}
size: {{NODE_DISK}}
env:
SLEEP_DURATION: {{SLEEP_DURATION}}
configs:
server-properties:
template: server.properties.mustache
dest: kafka_2.11-0.10.0.0/config/server.properties
The server.properties.mustache
should live in the current directory (dcos-commons/examples/kafka/src/main/dist
).
- Download the
server.properties.mustache
file mentioned above:curl -O https://raw.githubusercontent.com/mesosphere/dcos-commons/29c38ea81948c8fe550a9d77974f78155c318815/frameworks/kafka/src/main/dist/server.properties.mustache > server.properties.mustache
- Next, create the symlink that’s required to get the unit tests to pass and confirm that it’s created:
cd ../resources && ln -s ../dist/server.properties.mustache && ls -l
- Run the following command from the root directory of the service:
cd /dcos-commons/examples/kafka && ./build.sh local
Depending on your implementation, you may need to configure
ln -s /usr/bin/sha1sum /usr/bin/shasum
- Now, install the service via
dcos package install kafka
and visit your dashboard to see Kafka running: http://172.17.0.2/#/services/%2Fkafka/
[0] The mustaching provides support for dynamically updating a service’s configuration at runtime.