Skippr Docs

Appearance

How to Configure Kafka as a Source Connector ​

May 2026

Kafka source setup gets easier once you treat broker reachability, consumer identity, and security settings as one configuration problem.

Short Answer ​

Configure Kafka as a source by setting source.kind to kafka, then supplying brokers, topic, and optionally a stable group_id. Add auto_offset_reset when you need explicit startup behavior such as earliest, and set security_protocol, sasl_mechanism, sasl_username, and sasl_password if the cluster uses SASL.

For a clean first setup, pick one topic, one consumer group, and decide whether the connector should run in stream or batch mode. A topic like events, brokers like broker1:9092,broker2:9092, and auto_offset_reset: earliest make it easy to confirm that Skippr can join the group, consume historical messages, and keep offsets moving in a predictable way.

Why Teams Struggle with This ​

Kafka connector problems are usually coordination problems. The topic name may be right, but the consumer still cannot read if the broker addresses are wrong, the SASL settings do not match the cluster, or ACLs allow topic access but block consumer-group commits.

  • The runner must reach the bootstrap brokers from the network where Skippr executes.
  • Broker ACLs need to allow both topic consumption and consumer-group commit behavior.
  • If the cluster uses SASL, the security_protocol, sasl_mechanism, username, and password all have to line up.
  • CDC-style Kafka use cases depend on Debezium envelope parsing, so topic contents matter as much as connector reachability.

How Skippr Handles It ​

Skippr gives Kafka enough structure to be useful without hiding the cluster details that operators actually care about. You configure the bootstrap brokers, topic, consumer identity, and optional security fields directly, which means the first working config is usually very close to what your Kafka platform team already documents.

That also makes the first-run behavior easier to explain. You can say which topic was consumed, which group ID claimed the offsets, whether the connector started from earliest or latest, and whether the same source should later move into Debezium-based CDC handling.

  • Topic-based consumption with explicit broker and group settings.
  • Support for stream or batch mode depending on the ingestion pattern.
  • SASL and secure-cluster fields exposed directly in connector config.
  • A clear on-ramp to Debezium envelope parsing for CDC-oriented topics.

What the First Useful Version Looks Like ​

Start with one non-ambiguous topic and a dedicated test consumer group so you can inspect offsets without stepping on another pipeline.

If the cluster is secured, validate SASL and ACL behavior from the same runtime environment that will host Skippr, not only from a workstation that already has broader network access.