Kafka is available in the Universe and can be installed by using either the web interface or the DC/OS CLI.

Prerequisites

Depending on your security mode in Enterprise DC/OS, you may need to provision a service account before installing Kafka. Only someone with superuser permission can create the service account.
- strict security mode requires a service account.
- permissive security mode a service account is optional.
- disabled security mode does not require a service account.
Your cluster must have at least three private nodes.

Default Installation

To start a basic test cluster with three brokers, run the following command on the DC/OS CLI. Enterprise DC/OS users must follow additional instructions. More information about installing Kafka on Enterprise DC/OS.

$ dcos package install beta-kafka

This command creates a new Kafka cluster with the default name kafka. Two clusters cannot share the same name, so installing additional clusters beyond the default cluster requires customizing the name at install time for each additional instance.

All dcos beta-kafka CLI commands have a --name argument allowing the user to specify which Kafka instance to query. If you do not specify a service name, the CLI assumes the default value, kafka. The default value for --name can be customized via the DC/OS CLI configuration:

$ dcos beta-kafka --name kafka-dev <cmd>

**Note:** Alternatively, you can [install Kafka from the DC/OS web interface](https://docs.mesosphere.com/1.9/deploying-services/install/). If you install Kafka from the web interface, you must install the Kafka DC/OS CLI subcommands separately. From the DC/OS CLI, enter:

```bash
dcos package install beta-kafka --cli

Minimal Installation

For development purposes, you may wish to install Kafka on a local DC/OS cluster. For this, you can use dcos-vagrant.

To start a minimal cluster with a single broker, create a JSON options file named sample-kafka-minimal.json:

{
  "brokers": {
    "count": 1,
    "mem": 512,
    "disk": 1000
  }
}

The command below creates a cluster using sample-kafka-minimal.json:

$ dcos package install --options=sample-kafka-minimal.json beta-kafka

Custom Installation

Customize the defaults by creating a JSON file. Then, pass it to dcos package install using the --options parameter.

Sample JSON options file named sample-kafka-custom.json:

{
  "service": {
    "name": "sample-kafka-custom",
    "placement_strategy": "NODE"
  },
  "brokers": {
    "count": 10,
    "kill_grace_period": 30
  },
  "kafka": {
    "delete_topic_enable": true,
    "log_retention_hours": 128
  }
}

The command below creates a cluster using sample-kafka.json:

$ dcos package install --options=sample-kafka-custom.json beta-kafka

Recommendation: Store your custom configuration in source control.

See Configuration Options for a list of fields that can be customized via an options JSON file when the Kafka cluster is created.

Alternatively, you can perform a custom installation from the DC/OS web interface. Choose ADVANCED INSTALLATION at install time.

Multiple Kafka cluster installation

Installing multiple Kafka clusters is identical to installing Kafka clusters with custom configurations as described above. The only requirement on the operator is that a unique name be specified for each installation. For example:

$ cat kafka1.json
{
  "service": {
  "name": "kafka1"
  }
}

$ dcos package install beta-kafka --options=kafka1.json

Integration with DC/OS access controls

In Enterprise DC/OS 1.10 and above, you can integrate your SDK-based service with DC/OS ACLs to grant users and groups access to only certain services. You do this by installing your service into a folder, and then restricting access to some number of folders. Folders also allow you to namespace services. For instance, staging/kafka and production/kafka.

Steps:

In the DC/OS GUI, create a group, then add a user to the group. Or, just create a user. Click Organization > Groups > + or Organization > Users > +. If you create a group, you must also create a user and add them to the group.
Give the user permissions for the folder where you will install your service. In this example, we are creating a user called developer, who will have access to the /testing folder.

Select the group or user you created. Select ADD PERMISSION and then toggle to INSERT PERMISSION STRING. Add each of the following permissions to your user or group, and then click ADD PERMISSIONS.
```
    ```
    dcos:adminrouter:service:marathon full
    dcos:service:marathon:marathon:services:/testing full
    dcos:adminrouter:ops:mesos full
    dcos:adminrouter:ops:slave full
    ```
```
Install your service into a folder called test. Go to Catalog, then search for beta-kafka.
Click CONFIGURE and change the service name to /testing/kafka, then deploy.

The slashes in your service name are interpreted as folders. You are deploying Kafka in the /testing folder. Any user with access to the /testing folder will have access to the service.

Important:

Services cannot be renamed. Because the location of the service is specified in the name, you cannot move services between folders.
DC/OS 1.9 and earlier does not accept slashes in service names. You may be able to create the service, but you will encounter unexpected problems.

Interacting with your foldered service

Interact with your foldered service via the DC/OS CLI with this flag: --name=/path/to/myservice.
To interact with your foldered service over the web directly, use http://<dcos-url>/service/path/to/myservice. E.g., http://<dcos-url>/service/testing/kafka/v1/endpoints.

Placement Constraints

Placement constraints allow you to customize where a service is deployed in the DC/OS cluster. Depending on the service, some or all components may be configurable using Marathon operators (reference). For example, [["hostname", "UNIQUE"]] ensures that at most one pod instance is deployed per agent.

A common task is to specify a list of whitelisted systems to deploy to. To achieve this, use the following syntax for the placement constraint:

[["hostname", "LIKE", "10.0.0.159|10.0.1.202|10.0.3.3"]]

You must include spare capacity in this list, so that if one of the whitelisted systems goes down, there is still enough room to repair your service (via pod replace) without requiring that system.

Regions and Zones

Placement constraints can be applied to zones by referring to the @zone key. For example, one could spread pods across a minimum of 3 different zones by specifying the constraint:

[["@zone", "GROUP_BY", "3"]]

When the region awareness feature is enabled (currently in beta), the @region key can also be referenced for defining placement constraints. Any placement constraints that do not reference the @region key are constrained to the local region.

Alternate ZooKeeper

By default, the Kafka services uses the ZooKeeper ensemble made available on the Mesos masters of a DC/OS cluster. You can configure an alternate ZooKeeper at install time. This enables you to increase Kafka’s capacity and removes the system ZooKeeper’s involvment in the service.

To configure it:

Create a file named options.json with the following contents.

Note: If you are using the DC/OS Apache ZooKeeper service, use the DNS addresses provided by the dcos kafka-zookeeper endpoints clientport command as the value of kafka_zookeeper_uri.

{
    "kafka": {
      "kafka_zookeeper_uri": "zookeeper-0-server.kafka-zookeeper.autoip.dcos.thisdcos.directory:1140,zookeeper-1-server.kafka-zookeeper.autoip.dcos.thisdcos.directory:1140,zookeeper-2-server.kafka-zookeeper.autoip.dcos.thisdcos.directory:1140"
    }
}

Install Kafka with the options file.

dcos package install kafka --options="options.json"

You can also update an already-running Kafka instance from the DC/OS CLI, in case you need to migrate your ZooKeeper data elsewhere.

Note: The ZooKeeper ensemble you point to must have the same data as the previous ZooKeeper ensemble.

dcos kafka --name=/kafka update start --options=options.json

Kafka: Install and Customize