Kafka Streams

This section describes Kafka Streams, a component of open source Apache Kafka. Kafka Streams is a powerful, easy-to-use library for building highly scalable, fault-tolerant, distributed stream processing applications on top of Apache Kafka.

Contents:

• Introduction
  • The Kafka Streams API in a Nutshell
  • Use Case Examples
  • A Closer Look
• Requirements
  • Apache Kafka
  • Confluent
• Quickstart
  • Goal of this quickstart
  • What we want to do
  • Start the Kafka cluster
  • Prepare the topics and the input data
  • Process the input data with Kafka Streams
  • Inspect the output data
  • Stop the Kafka cluster
  • Where to go from here
• Concepts
  • Kafka 101
  • Stream
  • Stream Processing Application
  • Processor Topology
  • Stream Processor
  • Stateful Stream Processing
  • Duality of Streams and Tables
  • KStream
  • KTable
  • GlobalKTable
  • Time
  • Aggregations
  • Joins
  • Windowing
• Architecture
  • Processor Topology
  • Parallelism Model
    • Stream Partitions and Tasks
    • Threading Model
    • Example
  • State
  • Memory management
    • Record caches
  • Fault Tolerance
  • Processing Guarantees
  • Flow Control with Timestamps
  • Backpressure
• Developer Guide
  • Code examples
    • Application examples for Kafka Streams in Apache Kafka
    • Application examples for Kafka Streams provided by Confluent
  • Configuring a Kafka Streams application
    • Overview
    • Required configuration parameters
    • Optional configuration parameters
    • Non-Streams configuration parameters
    • Recommended configuration parameters for resiliency
  • Writing a Kafka Streams application
    • Overview
    • Libraries and maven artifacts
    • Using Kafka Streams within your application code
    • Kafka Streams DSL
    • Processor API
  • Interactive Queries
    • Overview
    • Demo applications
    • Your application and interactive queries
    • Querying local state stores (for an application instance)
    • Querying remote state stores (for the entire application)
  • Memory management
    • Record caches in the DSL
    • State store caches in the Processor API
    • Other memory usage
  • Running a Kafka Streams application
    • Starting a Kafka Streams application
    • Elastic scaling of your application
    • Application Reset Tool
  • Managing topics of a Kafka Streams application
    • Overview
    • User topics
    • Internal topics
  • Data types and serialization
    • Overview
    • Configuring default serializers/deserializers (serdes)
    • Overriding default serializers/deserializers (serdes)
    • Available serializers/deserializers (serdes)
    • Implementing custom serializers/deserializers (serdes)
  • Security
    • Overview
    • Notable ACL settings
    • Security example
• Operations
  • Capacity planning and sizing
    • Background and context
    • Stateless vs. stateful
    • Examples
    • Troubleshooting
  • Monitoring your application
    • Metrics
    • Run-time Status Information
    • Integration with Confluent Control Center
• Upgrade Guide
  • Upgrading from CP 3.1.x (Kafka 0.10.1.x-cp2) to CP 3.2.x (Kafka 0.10.2.x-cp1)
    • Compatibility
    • Upgrading your Kafka Streams applications to CP 3.2.x
    • API changes
    • Full upgrade workflow
    • Known issues and workarounds
• FAQ
  • General
    • Is Kafka Streams a project separate from Apache Kafka?
    • Is Kafka Streams a proprietary library of Confluent?
    • Do Kafka Streams applications run inside the Kafka brokers?
    • What are the system dependencies of Kafka Streams?
    • How do I migrate my Kafka Streams applications from CP 3.0.x and CP 3.1.x applications to CP 3.2.x
    • Which versions of Kafka clusters are supported by Kafka Streams?
    • What programming languages are supported?
  • Scalability
    • Maximum parallelism of my application? Maximum number of app instances I can run?
  • Processing
    • Accessing record metadata such as topic, partition, and offset information?
    • How can I replace RocksDB with a different store?
  • Failure and exception handling
    • Handling corrupted records and deserialization errors (“poison pill messages”)?
    • Sending corrupt records to a quarantine topic or dead letter queue?
  • Interactive Queries
    • Handling InvalidStateStoreException: “the state store may have migrated to another instance”?
  • Security
    • Application fails when running against a secured Kafka cluster?
  • Troubleshooting and debugging
    • Easier to interpret Java stacktraces?
    • Visualizing topologies?
    • Inspecting streams and tables?
    • Invalid Timestamp Exception
    • Why do I get an IllegalStateException when accessing record metadata?
    • Why is punctuate() not called?
    • Scala: compile error “no type parameter”, “Java-defined trait is invariant in type T”
    • How can I convert a KStream to a KTable without an aggregation step?
    • RocksDB behavior in 1-core environments
• Javadocs