Protocol Kafka

Apache Kafka is an open-source distributed event-streaming platform used to publish, store and consume streams of records. Producers write messages to topics, the broker stores them in partitioned, append-only logs, and consumers read them back in order, either individually or as part of a coordinated consumer group.

Kafka uses a binary, application-layer protocol over TCP. Each topic is split into one or more partitions, every record in a partition is identified by a monotonically increasing offset, and the broker retains the records so that consumers can read at their own pace and replay history when needed. This makes Kafka well suited to high-throughput messaging, event sourcing, log aggregation and stream processing.

Features

The sgcWebSockets Kafka client talks to the broker directly over the native Kafka protocol, with no REST proxy or external library required.

• Publish (produce) records to any topic and partition, with key, value and configurable acknowledgements.
• Consume records with a managed consumer group or without a group (read all partitions directly).
• Consumer group coordination: coordinator discovery, join/sync, partition assignment and offset commit/fetch.
• Offset management: query the earliest, latest and committed offsets.
• Topic administration: create and delete topics, request cluster metadata and list consumer groups.
• Reads and writes the v2 record batch format used by current Kafka brokers.

Supported Versions

The client uses the v2 record batch format introduced in Apache Kafka 0.11 and has been tested against Apache Kafka 3.x. The default broker port is 9092.

Components

TsgcWSPClient_Kafka: it's the client component that implements the Apache Kafka protocol.

Most common uses

• Connection
  • Client Kafka Connect
  • Client Kafka Disconnect

• Commands
  • Kafka Produce Messages
  • Kafka Consume Messages
  • Kafka Consumer Groups
  • Kafka Topics
  • Kafka Offsets