Confluent
  • Product
    • KSQLLearn about KSQL, Streaming SQL for Apache Kafka and what it can do.
      Play
    • Confluent Platform
    • Open Source
      • Connectors
    • Enterprise
      • Confluent Control Center
      • Auto Data Balancer
      • Replicator
      • Connectors
    • Cloud
    • KSQL
    • Subscription
    • Services
  • Solutions
    • NewGet an overview of an Apache Kafka-centric microservices architecture.
      Play
    • Industry Solutions
    • Microservices
    • Internet of Things
  • Developers
    • Kafka SummitGet early-bird pricing for Kafka Summit London, April 23-24, 2018.Register Now
    • What is Kafka?
    • Resources
    • Online Talks
    • Community
    • Training
    • Kafka Summit
  • Blog
  • Docs
  • Download
  • Getting Started
    • What is the Confluent Platform?
      • What is included in the Confluent Platform?
        • Overview
        • Confluent Open Source
        • Confluent Enterprise
        • Confluent Proactive Support
      • Migrating an existing Kafka deployment
        • Migrating developer tools
        • Migrating to Control Center
    • Confluent Platform Quick Start
    • Installing and Upgrading
      • Install Confluent Platform
        • System Requirements
        • Installation Formats
      • Packages
        • Platform Packages
        • Component Packages
      • Installing using Docker
        • Docker Quick Start
        • Docker Configuration
        • Docker Advanced Tutorials
      • Clients
        • Maven repository for JARs
        • C/C++
        • Python
        • Go
        • .NET
      • Migrating from Open Source to Enterprise
      • Upgrade
        • Preparation
        • Step-by-step Guide
      • Supported Versions and Interoperability
        • Confluent Platform Versions
        • Operating Systems
        • Confluent Control Center
        • Java
        • Scala
        • Component Security
        • Clients
        • Connectors
        • Cross-Component Compatibility
        • Docker and Orchestration Tools
    • Command Line Interface
      • Installing and Configuring the CLI
        • Installing the CLI
        • Configuring the CLI
      • Command Reference
        • confluent acl
        • confluent config
        • confluent current
        • confluent destroy
        • confluent help
        • confluent list
        • confluent load
        • confluent log
        • confluent start
        • confluent status
        • confluent stop
        • confluent top
        • confluent unload
        • Available Commands
        • Environment Variables
    • Confluent Platform 4.0.0 Release Notes
      • Highlights
        • Enterprise Features
        • Open Source Features
      • Previous releases
        • Confluent Platform 3.3.1 Release Notes
        • Confluent Platform 3.3.0 Release Notes
        • Confluent Platform 3.2.2 Release Notes
        • Confluent Platform 3.2.1 Release Notes
        • Confluent Platform 3.2.0 Release Notes
        • Confluent Platform 3.1.2 Release Notes
        • Confluent Platform 3.1.1 Release Notes
        • Confluent Platform 3.1.0 Release Notes
        • Confluent Platform 3.0.1 Release Notes
        • Confluent Platform 3.0.0 Release Notes
        • Confluent Platform 2.0.1 Release Notes
        • Confluent Platform 2.0.0 Release Notes
      • How to Download
      • Questions?
  • Operations
    • Kafka Operations
      • Production Deployment
        • Hardware
        • JVM
        • Important Configuration Options
        • File descriptors and mmap
        • Multi-node Configuration
      • Post Deployment
        • Changing configs dynamically
        • Logging
        • Admin operations
        • Rolling Restart
        • Performance Tips
        • Software Updates
        • Backup and Restoration
      • Auto Data Balancing
        • Requirements
        • Quickstart
        • Licensing
        • Configuration Options
      • Monitoring Kafka
        • Server Metrics
        • Producer Metrics
        • New Consumer Metrics
        • Old Consumer Metrics
      • Confluent Metrics Reporter
        • Installation
        • Configuration
        • Configuration Options
        • Verification
    • Confluent Control Center
      • Control Center Quick Start
        • Configure Kafka and Kafka Connect
        • Configure and start Control Center
        • Setup stream monitoring
      • Control Center Concepts
        • Motivation
        • Time windows and metrics
        • Latency
      • Installing and Upgrading Control Center
        • Installing Control Center
        • Control Center System Requirements
        • Configuring Control Center
        • Installing Control Center Interceptors
        • Installing Control Center on Apache Kafka
        • Troubleshooting Control Center
        • Upgrading Control Center
      • Control Center User Interface
        • System Health
        • Stream Monitoring
        • Kafka Connect
        • Alerts
        • Clusters
        • Topic Management
      • Security
        • Configuring SSL
        • Configuring SASL
        • UI Authentication
        • UI HTTPS
        • Authorization with Kafka ACLS
      • Changelog
        • Version 4.0.0
        • Version 3.3.1
        • Version 3.3.0
        • Version 3.2.2
        • Version 3.2.1
        • Version 3.2.0
        • Version 3.1.0
        • Version 3.0.1
    • Multi Data-Center Deployment
      • Replicator Quick Start
        • Start the destination cluster
        • Start the origin cluster
        • Create a topic
        • Configure and run Replicator
      • Installing and Configuring Replicator
        • Install and Configure Kafka Connect Cluster for Replicator
        • Configure and run a Confluent Replicator on the Connect Cluster
      • Tuning and Monitoring Replicator
        • Sizing Replicator Cluster
        • Getting More Throughput From Replicator Tasks
        • Monitoring Replicator
      • Configuration Options
        • Confluent Platform
        • Source Topics
        • Source Data Conversion
        • Source Zookeeper
        • Source Kafka
        • Source Kafka: Security
        • Source Kafka: Consumer
        • Destination Topics
        • Destination Zookeeper
        • Destination Data Conversion
      • Apache Kafka’s Mirror Maker
        • Comparing Mirror Maker to Confluent Replicator
        • Example of Mirror Maker Use
    • Schema Registry Operations
      • Production Deployment
        • Hardware
        • Memory
        • CPUs
        • Disks
        • Network
        • JVM
        • Important Configuration Options
        • Don’t Touch These Settings!
        • Kafka & ZooKeeper
        • Migration from Zookeeper master election to Kafka master election
        • Backup and Restore
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Kafka REST Proxy Operations
      • Production Deployment
        • Hardware
        • JVM
        • Deployment
        • Important Configuration Options
        • Don’t Touch These Settings!
        • Post Deployment
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Docker Operations
      • Monitoring
        • Using JMX
      • Configuring logging
        • log4j Log Levels
        • Component Names
      • Mounting External Volumes
        • Data Volumes for Kafka & ZooKeeper
        • Security: Data Volumes for Configuring Secrets
        • Configuring Connect with External jars
    • Security
      • Prefixes for Configuring Security
      • Encryption with SSL
        • SSL Overview
        • Creating SSL Keys and Certificates
        • Brokers
        • Clients
        • ZooKeeper
        • Kafka Connect
        • Confluent Replicator
        • Confluent Control Center
        • Confluent Metrics Reporter
        • Confluent Monitoring Interceptors
        • Schema Registry
        • REST Proxy
        • SSL Logging
      • Encryption and Authentication with SSL
        • SSL Overview
        • Creating SSL Keys and Certificates
        • Brokers
        • Clients
        • ZooKeeper
        • Kafka Connect
        • Confluent Replicator
        • Confluent Control Center
        • Confluent Metrics Reporter
        • Confluent Monitoring Interceptors
        • Schema Registry
        • REST Proxy
        • SSL Logging
      • Authentication with SASL
        • SASL Overview
        • GSSAPI
        • SCRAM
        • PLAIN
        • JAAS configurations
        • Enabling multiple SASL mechanisms
        • Modifying SASL mechanisms in a Running Cluster
      • Authorization and ACLs
        • Overview
        • Command Line Interface
      • Adding Security to a Running Cluster
        • Migrating Brokers and Clients
        • Migrating ZooKeeper
        • Migrating the ZooKeeper ensemble
      • Kafka Security & the Confluent Platform
        • Other Observations
      • Streams Security
        • Required ACL setting for secure Kafka clusters
        • Security example
      • Confluent Security Plugins
        • Installation
      • Docker Security
    • Kafka Connect
      • Introduction
      • Requirements
      • Kafka Connect Quick Start
        • Goal
        • What we will do
        • Start the services
        • Read File Data with Connect
        • Write File Data with Connect
      • Concepts
        • Connectors
        • Tasks
        • Workers
        • Converters
        • Transforms
      • Installing and Configuring Kafka Connect
        • Getting Started
        • Planning for Installation
        • Installing Plugins
        • Running Workers
        • Configuring Workers
        • Upgrading Kafka Connect Workers
      • Managing Connectors
        • Using Bundled Connectors
        • Configuring Connectors
        • Managing Running Connectors
        • Using Community Connectors
        • Upgrading a Connector Plugin
      • Bundled Connectors
        • Confluent JDBC Connector
        • Confluent HDFS Connector
        • Confluent S3 Connector
        • Confluent Elasticsearch Connector
        • Confluent Kafka Replicator
        • Kafka FileStream Connectors
      • Security
        • Encryption
        • Authentication
        • Separate principals
        • ACL Considerations
      • Architecture & Internals
        • Motivation
        • Architecture
      • Connector Developer Guide
        • Core Concepts and APIs
        • Developing a Simple Connector
        • Dynamic Input/Output Partitions
        • Configuration Validation
        • Working with Schemas
        • Schema Evolution
        • Testing
        • Packaging
      • FAQ
        • How do I change the output data format of a SinkConnector?
        • Why does a connector configuration update trigger a task rebalance?
        • Why should I use distributed mode instead of standalone?
        • Do I need to write custom code to use Kafka Connect?
        • Is the Schema Registry a required service to run Kafka Connect?
        • How can I use plain JSON data with Connect?
        • Does source connector X support output format Y?
        • Why is CPU usage high for my Connect worker when no connectors have been deployed?
        • Can connect sink connectors read data written by other clients, e.g. a custom client?
        • After testing a connector in standalone mode, restarting it doesn’t write the data again?
        • Can I use a newer version of connect with older brokers?
      • Reference
        • Javadocs
        • REST Interface
        • All Worker Configs
    • Camus Operations
      • Important Configuration Options
    • ZooKeeper Operations
      • ZooKeeper Production Deployment
        • Stable version
        • Hardware
        • JVM
        • Important Configuration Options
        • Monitoring
        • Multi-node Setup
      • Post Deployment
  • Development
    • Kafka Streams
      • Introduction
        • The Kafka Streams API in a Nutshell
        • Use Case Examples
        • A Closer Look
      • Requirements
        • Apache Kafka
        • Confluent
      • Kafka Streams Quick Start
        • Purpose
        • Start the Kafka cluster
        • Prepare the topics and the input data
        • Process the input data with Kafka Streams
        • Inspect the output data
        • Stop the Kafka cluster
        • Next steps
      • Kafka Streams Demo Application
        • Requirements
        • Running the Kafka Music demo application
        • Running further Confluent demo applications for the Kafka Streams API
        • Appendix
      • Concepts
        • Kafka 101
        • Stream
        • Stream Processing Application
        • Processor Topology
        • Stream Processor
        • Stateful Stream Processing
        • Duality of Streams and Tables
        • KStream
        • KTable
        • GlobalKTable
        • Time
        • Aggregations
        • Joins
        • Windowing
        • Interactive Queries
      • Architecture
        • Processor Topology
        • Parallelism Model
        • State
        • Memory management
        • Fault Tolerance
        • Processing Guarantees
        • Flow Control with Timestamps
        • Backpressure
      • Code Examples
        • Getting started examples
        • Security examples
        • Interactive queries examples
        • End-to-end application examples
      • Developer Guide
        • Writing a Streams Application
        • Configuring a Streams Application
        • Streams DSL
        • Processor API
        • Data Types and Serialization
        • Interactive Queries
        • Memory Management
        • Running Streams Applications
        • Managing Streams Application Topics
        • Streams Security
        • Application Reset Tool
      • Operations
        • Capacity planning and sizing
        • Monitoring your application
      • Upgrade Guide
        • Upgrading from CP 3.3.x (Kafka 0.11.0.x-cp1) to CP 4.0.0 (Kafka 1.0.0-cp1)
        • Upgrading older Kafka Streams applications to CP 4.0.0
      • FAQ
        • General
        • Scalability
        • Processing
        • Failure and exception handling
        • Interactive Queries
        • Security
        • Troubleshooting and debugging
      • Javadocs
    • KSQL
      • Download and Learn More
    • Kafka Clients
      • Kafka Consumers
        • Concepts
        • Configuration
        • Initialization
        • Basic Usage
        • Detailed Examples
        • Administration
      • Kafka Producers
        • Concepts
        • Configuration
        • Examples
      • Confluent’s JMS Client for Apache Kafka
        • Overview
        • Examples
        • Requirements
        • Installation
        • JMS 1.1 Compatibility
        • Features
        • Unsupported Features
        • Developing JMS Clients
        • Threading
      • API Docs
    • Connector Developer Guide
      • Core Concepts and APIs
        • Connectors and Tasks
        • Partitions and Records
        • Dynamic Connectors
      • Developing a Simple Connector
        • Connector Example
        • Task Example - Source Task
        • Sink Tasks
        • Resuming from Previous Offsets
      • Dynamic Input/Output Partitions
      • Configuration Validation
      • Working with Schemas
      • Schema Evolution
      • Testing
      • Packaging
        • Creating an Archive
        • Creating an Uber JAR
    • Schema Registry
      • Introduction
        • Quickstart
        • Installation
        • Deployment
        • Development
        • Requirements
        • Contribute
        • License
      • Changelog
        • Version 3.4.0
        • Version 3.3.0
        • Version 3.2.1
        • Version 3.2.0
        • Version 3.1.1
        • Version 3.1.0
        • Version 3.0.1
        • Version 3.0.0
        • Version 2.0.1
        • Version 2.0.0
      • API Reference
        • Overview
        • Schemas
        • Subjects
        • Compatibility
        • Config
      • Configuration Options
      • Design Overview
        • Schema ID Allocation
        • Kafka Backend
        • Single Master Architecture
      • Schema Registry Operations
        • Production Deployment
        • Monitoring
      • Multi-DC Setup
        • Overview
        • Kafka Election
        • Zookeeper Election
      • Security Overview
        • Kafka Store
        • ZooKeeper
        • Upgrade from HTTP to HTTPS for REST API
      • Serializer and Formatter
        • Serializer
        • Formatter
        • Wire Format
      • Schema Deletion Guidelines
      • Maven Plugin
        • schema-registry:download
        • schema-registry:test-compatibility
        • schema-registry:register
    • Docker Developer Guide
      • Image Design Overview
        • The Bootup Process
        • Configuration
        • Preflight Checks
        • Launching the Process
        • Development Guidelines
      • Setup
        • Building the Images
        • Running Tests
        • Make Targets
      • Extending the Docker Images
        • Prerequisites
        • Adding Connectors to the Kafka Connect Image
        • Examples
      • Utility Scripts
        • Docker Utility Belt (dub)
        • Confluent Platform Utility Belt (cub)
        • Client Properties
      • References
    • Data Serialization and Evolution
      • Introduction
      • Avro
        • Defining an Avro Schema
        • Schema Evolution
        • Backward Compatibility
        • Forward Compatibility
        • Full Compatibility
        • Confluent Schema Registry
    • Kafka REST Proxy
      • Kafka REST Proxy
        • Quickstart
        • Features
        • Installation
        • Deployment
        • Development
        • Requirements
        • Contribute
        • License
      • Changelog
        • Version 3.3.0
        • Version 3.2.2
        • Version 3.2.1
        • Version 3.2.0
        • Version 3.1.2
        • Version 3.1.1
        • Version 3.1.0
        • Version 3.0.0
        • Version 2.0.1
        • Version 2.0.0
      • API Reference
        • Overview
        • API v2
        • API v1
      • Configuration Options
        • Security Configuration Options
        • Interceptor Configuration Options
      • Kafka REST Operations
        • Production Deployment
        • Monitoring
      • Security Overview
    • Camus
      • Camus
        • Key Features
        • Quickstart
        • Installation
        • Deployment
        • Development
        • Requirements
        • Contribute
        • License
      • Changelog
        • Version 4.0.0
        • Version 3.3.1
        • Version 3.3.0
        • Version 3.2.2
        • Version 3.2.1
        • Version 3.2.0
        • Version 3.1.2
        • Version 3.1.1
        • Version 3.1.0
        • Version 3.0.0
        • Version 2.0.1
        • Version 2.0.0
      • Configuration Options
        • Schema Registry Configuration
        • Camus Job Configuration
        • Kafka Configuration
        • Example Configuration
      • Design
      • Operations
        • Important Configuration Options
    • Application Development
      • Native Clients with Serializers
        • Java
        • C/C++
      • REST Proxy
    • Confluent Proactive Support
      • What is Proactive Support?
      • How it works
      • Which metadata is collected?
        • Version Collector (default)
        • Confluent Support Metrics (add-on package)
      • Which metadata and data is not being collected?
      • Installing Support Metrics
        • DEB Packages via apt
        • RPM Packages via yum
      • Enabling or disabling the Metrics feature
      • Recommended Proactive Support configuration settings for licensed Confluent customers
      • Proactive Support configuration settings
      • Network ports used by Proactive Support
      • Sharing Proactive Support Metadata with Confluent manually
      • Privacy Policy
Version 4.0.0
  • Docs »
  • Operations »
  • Kafka Connect »
  • Bundled Connectors »
  • Confluent HDFS Connector
  • View page source

Confluent HDFS ConnectorΒΆ

Contents:

  • HDFS Connector
    • Quickstart
    • Features
    • Configuration
      • Example
      • Format and Partitioner
      • Hive Integration
      • Secure HDFS and Hive Metastore
    • Schema Evolution
  • Configuration Options
    • HDFS
    • Security
    • Connector
    • Storage
    • Partitioner
    • Hive
    • Schema
  • Changelog
    • Version 4.0.0
    • Version 3.3.1
    • Version 3.3.0
    • Version 3.2.2
    • Version 3.2.1
    • Version 3.2.0
    • Version 3.1.1
    • Version 3.1.0
    • Version 3.0.1
      • HDFS Connector
    • Version 3.0.0
      • HDFS Connector
Next Previous

© Copyright 2018, Confluent, Inc. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Built with Sphinx using a theme provided by Read the Docs.

Please report any inaccuracies on this page or suggest an edit.

v.--