> For the complete documentation index, see [llms.txt](https://mariadb.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mariadb.com/docs/gridgain/gridgain-8/integrations/kafka/quick-start.md).

# Quick Start

Learn how to install, configure, and run the GridGain Kafka Connector, and how to manage connectors through the Kafka Connect REST API.

## Kafka Connect Ecosystem

There are different types of nodes in a distributed Kafka Connect ecosystem. This Kafka documentation uses the following terminology to refer to specific type of a cluster node:

* Kafka cluster nodes are called **Kafka Brokers**
* Kafka Connect cluster nodes are called **Kafka Connect Workers**
* GridGain cluster nodes are called **GridGain Servers**

![Kafka Overview](https://3529983645-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FArQLGGUSG0VzKl1t2Y53%2Fuploads%2Fgit-blob-be3e48ed24d3c03bbe472e2192381393fec6dc07%2Fgg8-kafka-qs.png?alt=media)

## GridGain Kafka Connector Installation

Kafka Connector installation require 3 steps:

1. Prepare Connector Package
2. Register Connector with Kafka
3. Register Connector with GridGain

### Step 1: Prepare Connector Package

The connector ships as the [`gridgain/gridgain-kafka-connect`](https://www.confluent.io/hub/gridgain/gridgain-kafka-connect) Confluent Hub component. It is not part of the GridGain server package. Use the component version that matches your GridGain cluster version.

Download the component archive from Confluent Hub, unzip it, and rename the extracted directory to `gridgain-kafka-connect`. The component's `lib` directory already contains the GridGain JARs the connector needs.

If your cluster runs GridGain Ultimate Edition, also copy `libs/gridgain-ultimate/*.jar` from the GridGain server package into the component's `lib` directory. The component contains the Enterprise Edition JARs only.

{% hint style="info" %}
You can also install the component on each Kafka Connect worker with the Confluent Hub client: `confluent-hub install gridgain/gridgain-kafka-connect:{gg-version}`. On Ultimate Edition clusters, copy the Ultimate Edition JARs into the installed component's `lib` directory.
{% endhint %}

Alternatively, download the component archive from the [Confluent Hub listing](https://www.confluent.io/hub/gridgain/gridgain-kafka-connect) and unzip it under a directory on your Kafka Connect workers' `plugin.path`, then restart the workers.

If your cluster runs GridGain Ultimate, the component carries only the Enterprise jars. Copy the `libs/gridgain-ultimate/*.jar` files from the GridGain server package into the component's `lib/` directory on every Kafka Connect worker.

### Step 2: Register GridGain Connector with Kafka

For every Kafka Connect Worker:

1. Copy the `gridgain-kafka-connect` directory you prepared in Step 1 to where you want Kafka Connectors to be located.
2. Edit Kafka Connect Worker configuration (`$KAFKA_HOME/config/connect-standalone.properties` for single-worker Kafka Connect cluster or `$KAFKA_HOME/config/connect-distributed.properties` for multiple node Kafka Connect cluster) to register the connector on the plugin path (replace `CONNECTORS_PATH` with directory where you copied the connector package):

   <pre data-title="connect-standalone.properties"><code>plugin.path=CONNECTORS_PATH
   </code></pre>

### Step 3: Register GridGain Connector with GridGain

This step is needed only for a source connector with `failoverPolicy=BACKLOG`. That connector deploys its backlog service as a cluster singleton, so the connector classes must be on the classpath of every GridGain server node. Other failover policies need nothing on the server nodes.

* `gridgain-kafka-connect-{gg-version}.jar` from the component's `lib` directory
* `connect-api-{kafka-version}.jar` and `kafka-clients-{kafka-version}.jar` from the `$KAFKA_HOME/libs` directory on a Kafka Connect worker (or a later version, if you choose to use it). The component does not include them.
* Copy the `ignite-slf4j` module from `libs/optional` to `libs`
* If Kafka Connect runs on JDK 17 or later, the following JVM properties must be added to your Kafka Connect configuration:
* `gridgain-kafka-connect-{gg-version}.jar` (located in the `lib` directory of the connector component)
* `connect-api-{kafka-version}.jar` and `kafka-clients-{kafka-version}.jar`, located on Kafka Connect workers in the `$KAFKA_HOME/libs` directory (or a later version, if you choose to use it). The component does not carry them, because the Connect runtime provides them on the workers.
* The `ignite-slf4j` module, copied from `libs/optional` to `libs`

If Kafka Connect runs on JDK 17 or later, the following JVM properties must be added to your Kafka Connect configuration:

```bash
  --add-opens=java.base/java.nio=ALL-UNNAMED
  --add-opens=java.base/java.util=ALL-UNNAMED
```

## GridGain Kafka Connector Configuration

### SSL Configuration

The GridGain connector is agnostic of the underlying communication channels. To have a secure communication between the GridGain connector and other components, implement SSL using one of the following options:

* SSL between the GridGain cluster and GridGain connector - follow the [SSL/TLS guide](/docs/gridgain/gridgain-8/security/encryption-and-transport/ssl-tls.md)
* SSL between Kafka and GridGain connector - use the [TLS authentication](https://docs.confluent.io/platform/current/security/authentication/mutual-tls/overview.html#kconnect-long)

### Source Properties

The only GridGain Source connector mandatory properties are the connector's name, class, and path to Ignite configuration describing how to connect to the source GridGain cluster. Here's what a minimal source connector configuration named "gridgain-kafka-connect-source" might look like:

{% code title="gridgain-kafka-connect-source.properties" %}

```
name=gridgain-kafka-connect-source
connector.class=org.gridgain.kafka.source.IgniteSourceConnector
igniteCfg=IGNITE_CONFIG_PATH/ignite-server-source.xml
```

{% endcode %}

See [Source Connector Configuration](/docs/gridgain/gridgain-8/integrations/kafka/configuration.md#source-connector) for the full properties list.

### Sink Properties

The only GridGain Sink connector mandatory properties are the connector's name, class, list of topics to stream data from and a path to Ignite configuration describing how to connect to the sink GridGain cluster. Here's what a minimal source connector configuration named "gridgain-kafka-connect-sink" might look like:

{% code title="gridgain-kafka-connect-sink.properties" %}

```
name=gridgain-kafka-connect-sink
topics=topic1,topic2,topic3
connector.class=org.gridgain.kafka.sink.IgniteSinkConnector
igniteCfg=IGNITE_CONFIG_PATH/ignite-server-sink.xml
```

{% endcode %}

See [Sink Connector Configuration](/docs/gridgain/gridgain-8/integrations/kafka/configuration.md#sink-connector) for the full properties list.

## Running Kafka Connect Ecosystem

See [Installing and Configuring Kafka Connect](https://docs.confluent.io/current/connect/userguide.html#installing-and-configuring-kconnect-long) for detailed documentation. As a summary, you need to:

1. Configure and Install Kafka Connectors
2. Configure and start Zookeeper
3. Configure and start Kafka brokers
4. Configure and start Kafka Connect workers

We already reviewed how to configure and install Kafka connectors. Below are shell commands to run Kafka Connect ecosystem on the same host using default zookeeper, broker, and connect worker configuration files (normally you would run each node on a separate host):

```bash
$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
$KAFKA_HOME/bin/connect-standalone.sh \
	$KAFKA_HOME/config/connect-standalone.properties \
	gridgain-kafka-connect-source.properties \
	gridgain-kafka-connect-sink.properties
```

## Managing Kafka Connectors

Each Kafka worker exposes REST API to manage Kafka Connectors (available on port `8083` by default). See [Kafka Connect REST Interface](https://docs.confluent.io/current/connect/references/restapi.html) for information on how to create, remove, pause, and resume connectors as well as see the status of the connectors and tasks.

## See Also

* [How to run the GridGain Kafka Connector on Amazon MSK](/docs/gridgain/gridgain-8/gridgain8-get-started/how-to-guides/example-running-on-amazon-msk.md)