What is the Confluent Platform?¶

The Confluent Platform is a streaming platform that enables you to organize and manage data from many different sources with one reliable, high performance system.

Stream Data Platform

A streaming platform provides not only the system to transport data, but all of the tools needed to connect data sources, applications, and data sinks to the platform.

What is included in the Confluent Platform?¶

Overview¶

The Confluent Platform makes it easy build real-time data pipelines and streaming applications. By integrating data from multiple sources and locations into a single, central streaming platform for your company, the Confluent Platform lets you focus on how to derive business value from your data rather than worrying about the underlying mechanics such as how data is being transported or massaged between various systems. Specifically, the Confluent Platform simplifies connecting data sources to Kafka, building applications with Kafka, as well as securing, monitoring, and managing your Kafka infrastructure.

Confluent Platform 3.1 Components

As its backbone, the Confluent Platform leverages Apache Kafka, the most popular open source distributed streaming platform. It has three key capabilities:

It lets you publish and subscribe to streams of records
It lets you store streams of records in a fault tolerant way
It lets you process streams of records

It is used by hundreds of companies for many different use cases including collecting user activity data, system logs, application metrics, stock ticker data, and device instrumentation signals.

The Apache Kafka open source project consist of two key components:

Kafka Brokers (open source). Kafka brokers that form the mesaging, data persistency and storage tier of Apache Kafka
Kafka Java Client APIs (open source)
- Producer API (open source). Java Client that allows an application to publish a stream records to one or more Kafka topics.
- Consumer API (open source). Java Client that allows an application to subscribe to one or more topics and process the stream of records produced to them.
- Streams API (open source). Allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. It has a very low barrier to entry, easy operationalization, and a high-level DSL for writing stream processing applications. As such it is the most convenient yet scalable option to process and analyze data that is backed by Kafka.
- Connect API (open source). A component to stream data between Kafka and other data systems in a scalable and reliable way. It makes it simple to configure connectors to move data into and out of Kafka. Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing. Connectors can also deliver data from Kafka topics into secondary indexes like Elasticsearch or into batch systems such as Hadoop for offline analysis.

Note

Confluent is committed to maintaining, enhancing, and supporting the open source Apache Kafka project. As this documentation discusses, the version of Kafka in the Confluent Platform may contain patches over the matching open source version but is fully compatible with the matching open source version.

Apache Kafka and all its sub-components are available as open source under the Apache License v2.0 license. Source code at https://github.com/apache/kafka.

Any new release of the Confluent Platform include the latest release of Apache Kafka. It also includes more tools and services that make it easier to build and manage a streaming platform. Confluent Platform is available in two flavors:

Confluent Open Source This is 100% open-source. In addition to the components included with Apache Kafka, Confluent open source includes services and tools that is frequently used with Kafka. This makes Confluent Open Source the best way to get started with setting up a Kafka-based streaming platform. Confluent Open Source includes clients for C, C++, Python and Go programming languages; connectors for JDBC, ElasticSearch, and HDFS; Schema Registry for managing metadata for Kafka topics; and REST Proxy for integrating with web applications.
Confluent Enterprise This takes it to the next level by addressing requirements of modern enterprise streaming applications. It includes Confluent Control Center for end-to-end monitoring of event streams, MDC Replication for managing multi-datacenter deployments, and Automatic Data Balancing for optimizing resource utilization and easy scalability of Kafka clusters. Collectively, the components of Confluent Enterprise give your team a simple and clear path towards establishing a consistent yet flexible approach for building an enterprise-wide streaming platform for a wide array of use cases.

The capabilities of these components are briefly summarized in the following sections and described in more detail throughout the other chapters of this documentation. We help you get started easily through guides and quickstarts, describe best practices both for the deployment and management of Kafka, and show you how to use the Confluent Platform to get the most out of your data infrastructure.

Confluent Open Source¶

As mentioned previously, Confluent Open Source includes the latest Apache Kafka version. In addition, it includes the following critical Confluent open source components that make it easy to set up a production ready stream platform -

Confluent Kafka Connectors¶

Kafka connectors leverage the Kafka Connect API to connect Apache Kafka to other data systems such as Apache Hadoop. We have built connectors for the most popular data sources and sinks, and include fully tested and supported versions of these connectors with the Confluent Platform.

Tip

Beyond the Kafka connectors listed below, which are part of the Confluent Platform, you can find further connectors at the Kafka Connector Hub.

Kafka Connect JDBC Connector¶

The Confluent JDBC connector provides both source and sink connector. The source connector allows you to import data from any JDBC-compatible relational database into Kafka. The sink connector allows you to export data from Kafka topics to any relational database. By using JDBC, this connector can support a wide variety of databases without requiring custom code for each one.

The JDBC connector is available as open source software under the Apache License v2.0 license. Source code at https://github.com/confluentinc/kafka-connect-jdbc.

Kafka Connect HDFS Connector¶

The Confluent Hadoop Connector allows you to export data from Kafka topics to HDFS in a variety of formats and integrates with Hive to make data immediately available for querying with Hive.

The HDFS connector is available as open source software under the Apache License v2.0 license. Source code at https://github.com/confluentinc/kafka-connect-hdfs.

Kafka Connect Elasticsearch Connector¶

The Confluent Elasticsearch Connector allows you to move data from Kafka to Elasticsearch. It writes data from a topic in Kafka to an index in Elasticsearch and all data for a topic have the same type.

The Elasticsearch connector is available as open source software under the Apache License v2.0 license. Source code at https://github.com/confluentinc/kafka-connect-elasticsearch.

Confluent Kafka Clients¶

C/C++ Client Library¶

The library librdkafka is the C/C++ implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery, reliability and high performance in mind. Current benchmarking figures exceed 800,000 messages/second for the producer and 3 million messages/second for the consumer. See Kafka Clients for details. This library includes support for many new features Kafka 0.10 including message security. It also integrates easily with libserdes, our C/C++ library for Avro data serialization (supporting the Schema Registry).

The C/C++ library is available as open source software under the BSD 2-clause license. Source code at https://github.com/edenhill/librdkafka.

Python Client Library¶

Confluent Open Source also bundles a full-featured, high-performance client for Python. See Kafka Clients for details.

The Python library is available as open source software under the Apache License v2.0 license. Source code at https://github.com/confluentinc/confluent-kafka-python.

Go Client Library¶

With CP 3.1, we are releasing a full-featured, high-performance client for Go. See Kafka Clients for details.

The Go library is available as open source software under the Apache License v2.0 license. Source code at https://github.com/confluentinc/confluent-kafka-go.

Confluent Schema Registry¶

One of the most difficult challenges with loosely coupled systems is ensuring compatibility of data and code as the system grows and evolves. With a messaging service like Kafka, services that interact with each other must agree on a common format, called a schema, for messages. In many systems, these formats are ad hoc, only implicitly defined by the code, and often are duplicated across each system that uses that message type.

As requirements change, it becomes necessary to evolve these formats. With only an ad-hoc definition, it is very difficult for developers to determine what the impact of their change might be.

The Confluent Schema Registry enables safe, zero downtime evolution of schemas by centralizing the management of schemas written for the Avro serialization system. It tracks all versions of schemas used for every topic in Kafka and only allows evolution of schemas according to user-defined compatibility settings. This gives developers confidence that they can safely modify schemas as necessary without worrying that doing so will break a different service they may not even be aware of.

The Schema Registry also includes plugins for Kafka clients that handle schema storage and retrieval for Kafka messages that are sent in the Avro format. This integration is seamless – if you are already using Kafka with Avro data, using the Schema Registry only requires including the serializers with your application and changing one setting.

The Confluent Schema Registry is available as open source software under the Apache License v2.0 license. Source code at https://github.com/confluentinc/schema-registry.

Confluent Kafka REST Proxy¶

Apache Kafka and Confluent provide native clients for Java, C, C++, and Python that make it fast and easy to produce and consume messages through Kafka. These clients are usually the easiest, fastest, and most secure way to communicate directly with Kafka.

But sometimes, it isn’t practical to write and maintain an application that uses the native clients. For example, an organization might want to connect a legacy application written in PHP to Kafka. Or suppose that a company is running point-of-sale software that runs on cash registers, written in C# and runing on Windows NT 4.0, maintained by contractors, and needs to post data across the public internet. To help with these cases, Confluent Platform includes a REST Proxy. The REST Proxy addresses these problems.

The Confluent Kafka REST Proxy makes it easy to work with Kafka from any language by providing a RESTful HTTP service for interacting with Kafka clusters. The REST Proxy supports all the core functionality: sending messages to Kafka, reading messages, both individually and as part of a consumer group, and inspecting cluster metadata, such as the list of topics and their settings. You can get the full benefits of the high quality, officially maintained Java clients from any language.

The REST Proxy also integrates with the Schema Registry. It can read and write Avro data, registering and looking up schemas in the Schema Registry. Because it automatically translates JSON data to and from Avro, you can get all the benefits of centralized schema management from any language using only HTTP and JSON.

The Confluent Kafka REST Proxy is available as open source software under the Apache License v2.0 license. Source code at https://github.com/confluentinc/kafka-rest.

Confluent Enterprise¶

Automatic Data Balancing¶

As clusters grow, topics and partitions grow at different rates, brokers are added and removed and over time this leads to unbalanced workload across datacenter resources. Some brokers are not doing much at all, while others are heavily taxed with large or many partitions, slowing down message delivery. When executed, automatic data balancing monitors your cluster for number of brokers, size of partitions, number of partitions and number of leaders within the cluster. It allows you to shift data to create an even workload across your cluster, while throttling rebalance traffic to minimize impact on production workloads while rebalancing.

Multi-Datacenter Replication¶

Confluent Enterprise now makes it easier than ever to maintain multiple Kafka clusters in multiple data centers. Managing replication of data and topic configuration between data centers enables use-cases such as:

Active-active geo-localized deployments: allows users to access a near-by data center to optimize their architecture for low latency and high performance
Centralized analytics: Aggregate data from multiple Kafka clusters into one location for organization-wide analytics
Cloud migration: Use Kafka to synchronize data between on-prem applications and cloud deployments

The new Multi-Datacenter Replication feature allows configuring and managing replication for all these scenarios from either Confluent Control Center or command-line tools.

Confluent Control Center¶

Confluent Control Center is a GUI-based system for managing and monitoring Apache Kafka. It allows you to easily manage Kafka Connect, to create, edit, and manage connections to other systems. It also allows you to monitor data streams from producer to consumer, assuring that every message is delivered, and measuring how long it takes to deliver messages. Using Control Center, you can build a production data pipeline based on Apache Kafka without writing a line of code. Control Center also has the capability to define alerts on the latency and completeness statistics of data streams, which can be delivered by email or queried from a centralized alerting system.

Confluent Proactive Support¶

Proactive Support is a component of the Confluent Platform that improves Confluent’s support for the platform by collecting and reporting support metrics (“Metrics”) to Confluent. Proactive Support is enabled by default in the Confluent Platform. We do this to provide proactive support to our customers, to help us build better products, to help customers comply with support contracts, and to help guide our marketing efforts.

Proactive Support is composed of three sub-modules. Two of them are available as open source software under the Apache License v2.0 license. Source code for these components is available at https://github.com/confluentinc/support-metrics-common and https://github.com/confluentinc/support-metrics-client. There is also an optional metrics collection component available as part of the Support Metrics package. Support Metrics is a proprietary component of the Confluent Platform. You may use the Confluent Platform with (recommended) or without Support Metrics installed or enabled.

Migrating an existing Kafka deployment¶

Confluent is committed to supporting and developing the open source Apache Kafka project. The version of Kafka in the Confluent Platform is fully compatible with the matching open source version and only contains additional patches for critical bugs when the Confluent Platform and Apache Kafka release schedules do not align. An existing cluster can typically be upgraded easily by performing a rolling restart of Kafka brokers, though we do provide detailed instructions for each CP release.

Migrating developer tools¶

The rest of the Confluent Platform can be added incrementally. Start by adding the Schema Registry and updating your applications to use the Avro serializer. Next, add the REST Proxy to support applications that may not have access to good Kafka clients or Avro libraries. Run the HDFS connector to load data from Kafka into HDFS continuously.

Migrating to Control Center¶

You can also migrate an existing cluster to take advantage of Control Center. Simply deploy Control Center on a server, configure the server to communicate with your cluster, add interceptors to your Connect instance, producers, and consumers, and you’re ready to go.