DCR from GridGain 8 Guide
Near-zero downtime migration from GridGain 8 to GridGain 9 using one-way data center replication: prepare tables, replicate, monitor, cut over, and roll back.
In certain environments, downtime associated with migrating data between GridGain versions is not acceptable. For these scenarios, there is a one-way data migration tool based on GridGain 8 Data Center Replication. From the GridGain 8 source cluster's perspective, the GridGain 9 cluster appears as another GridGain 8 data center.
After replication is complete, a brief pause is required to stop writes, flush the replication backlog, and switch traffic to GridGain 9. As such, this is a "near-zero downtime" migration, not a seamless one. A brief maintenance window is required during the final cutover to ensure data consistency.
Migration Workflow Summary
Prepare target tables and indexes.
Configure the GridGain 8 DR sender and the GridGain 9 DR connector.
Start the DR connector.
Verify connectivity.
Start full state transfer.
Monitor replication backlog.
Pause writes on GridGain 8.
Wait for the backlog to drain.
Repoint clients to GridGain 9.
Stop replication and clean up replication artifacts.
Prerequisites
Before starting replication, complete the following:
Create an empty target GridGain 9 table for each cache to be replicated.
By default, fields are matched to columns with the same name. You can manually configure field-to-column mappings if necessary. For caches with a single key or value field, use the
_keyor_valfield name. See Mapping a plain-type key cache and Mapping a plain-type value cache for details, and Cache-to-Table Mapping Rules for default naming.Add a nullable
drVersioncolumn of typeVARBINARYto each target GridGain 9 table. This column stores data center replication metadata and can be safely deleted after migration. Replication cannot start without it.Recreate indexes from GridGain 8 on the target GridGain 9 tables. Indexes are not transferred automatically and must be created manually to maintain query performance. See Recreate indexes for details.
Initialize the GridGain 9 cluster if you have not done so already:
Contact MariaDB support to enable data center replication for any existing caches you plan to replicate.
Type Mapping
GridGain 8 and GridGain 9 differ in their support for data types. Make sure data is mapped to columns of equivalent types. The table below provides a reference for commonly used types:
java.lang.Boolean
BOOLEAN
java.lang.Byte
TINYINT
java.lang.Short
SMALLINT
java.lang.Integer
INT
java.lang.Long
BIGINT
java.lang.Float
REAL
java.lang.Double
DOUBLE
java.math.BigDecimal
DECIMAL(p,s)
Specify precision and scale
java.lang.String
VARCHAR
Specify length if needed
byte[]
VARBINARY
Specify length if needed
java.util.Date
TIMESTAMP WITH LOCAL TIME ZONE
Consider timezone settings
java.sql.Time
TIME
Consider timezone settings
java.sql.Date
DATE
Consider timezone settings
java.sql.Timestamp
TIMESTAMP
Consider timezone settings
java.util.UUID
UUID
Cache-to-Table Mapping Rules
The DR connector determines the target GridGain 9 table from the cache name, not from QueryEntity.tableName.
When no cacheMapping is configured for a cache, the DR connector:
Uses the cache name verbatim as the target table name.
Always targets the
PUBLICschema.Does not strip the
SQL_PUBLIC_prefix that GridGain 8 adds to caches backing SQL-created tables.Does not support configuring a different default schema.
Examples:
Cache
SQL_PUBLIC_PERSONmaps to tablePUBLIC.SQL_PUBLIC_PERSONCache
PersonCachemaps to tablePUBLIC.PersonCache
If the target table name or schema differs from the source cache name, configure an explicit cacheMapping entry in the DR connector configuration.
Examples of cases that require explicit cacheMapping:
Replicating cache
SQL_PUBLIC_PERSONto tablePUBLIC.PERSONReplicating cache
PersonCacheto tableCUSTOM.PERSONReplicating a cache created programmatically with a
QueryEntity.tableNamevalue different from the cache name
The DR connector does not read or use QueryEntity.tableName. If a cache was created through the programming API with a QueryEntity whose tableName differs from the cache name, the connector has no knowledge of that mapping and falls back to using the cache name directly.
For additional configuration examples, see DR Connector.
Row Replication Algorithm
Inserts/updates/removes from the source GridGain 8 cluster are versioned. If a row or tombstone record in the target GridGain 9 cluster has version V1, and the source GridGain 8 cluster sends an insert, update, or remove operation with version V2 lower than V1, the DR connector ignores the operation. Otherwise, the DR connector applies the operation as follows:
insert/update entry
key does not exist
insert row
insert/update entry
key exists, drVersion column is not NULL
update row, overwrites all columns
insert/update entry
key exists, drVersion column is NULL
no action
remove entry
key exists, drVersion column is not NULL
create tombstone record in the tombstones table
remove entry
key exists, drVersion column is NULL
no action
remove entry
key does not exist
create tombstone record in the tombstones table
Prepare GridGain 9 Tables and Indexes
Here's an example of migrating a GridGain 8 cache schema to GridGain 9:
Original GridGain 8 cache:
Creating equivalent GridGain 9 table:
Recreate Indexes
Since indexes are not automatically transferred during DCR migration, you need to identify existing indexes in GridGain 8 and recreate them in GridGain 9.
To identify indexes in GridGain 8, query the system views:
Alternatively, you can use the control script to list caches and check which indexes are configured:
After identifying the indexes, create them in GridGain 9 before starting replication:
The indexes should be created after table creation but before starting the replication process for optimal performance during data transfer.
Configuring Replication
Configure GridGain 8 DR Sender and GridGain 9 DR Connector
Configuring replication to a GridGain 9 cluster on the GridGain 8 side is the same as configuring replication to a GridGain 8 cluster. On the GridGain 9 side, you need to start a connector tool that converts data from GridGain 8 and transfers it to GridGain 9.
Make sure to note the ID of the data center the source cluster is running in. This ID will be used in connector configuration to filter incoming connections.
The example below shows where in GridGain 8 configuration you can find data center ID.
The cache-to-table mapping (including whether the connector uses the default PUBLIC.<cache-name> mapping or an explicit cacheMapping) is configured separately on the DR connector side. See Cache-to-Table Mapping Rules and DR Connector for additional details and connector-side mapping examples.
For additional information on data center replication in GridGain 8 see Data Center Replication in GridGain 8.
Tombstones
Tombstone time-to-live, configured via the tombstoneTtl property, on the receiving side must be the same as or greater than on the sending side. By default, the same value is used in GridGain 8 and GridGain 9 (30 minutes).
If a sender in the source GridGain 8 cluster has larger tombstoneTtl than the DR connector, then the following warning is written to the DR connector logs:
Ensure That Time Zones Are in Sync
For additional information on configuring and running the DR connector tool, see DR Connector.
Capacity Planning
Data between a GridGain 8 sender and the DR connector is transferred in batches. After a sender sends a batch, it waits for an acknowledgment from the DR connector.
A single batch can contain multiple entries from one cache (max=2048), and have a 1 MB soft upper size limit for regular replication traffic and 8 MB for Full State Transfer. A batch is closed as soon as any of these limits is reached, or after a configurable time interval if neither limit is reached. A sender keeps sending batches until 100 unacknowledged batches accumulate for a given target data center (both regular and FST batches are counted, across all replicated caches).
The example below shows how to configure these limits in the sender configuration:
To account the worst case scenario (100 in-flight FST batches at maximum density, when the batch size has reached 8 MB and the number of entries is also close to its maximum value), you need to allocate approximately 4 times more memory on the connector:
Then, add another 25% for cluster needs:
To pass this value to the connector JVM parameters, you can use the EXTRA_JVM_ARGS Java variable:
If there are several senders, the connector must be ready to accept that many times the load and to have that many times more memory available. You can also start several instances of the connector itself, in which case the heap size for each one can be calculated with the following formula:
Keep in mind that during normal operation the number of unacknowledged (in-flight) batches is usually several times lower than the limit of 100, and therefore memory consumption should be significantly lower than the calculated limit of 4 GB. If you see that connector consumption is approaching the limits, it is a sign that either the amount of traffic from GridGain 8 is too high for a single connector instance, or the GridGain 9 cluster is overloaded (there may be significant concurrent write load from other sources).
The simplest way to reduce the load on the connector is to lower the maximum number of in-flight batches on the senders (see the maxQueueSize property in the snippet above).
Starting Replication
Start the DR Connector
Start the connector tool on GridGain 9 side:
For additional information on configuring DR connector, see DR Connector.
Verify That the DR Connector Is Running
Check the DR connector logs to ensure it is listening on the configured port. Look for the "Listening port" line in the log, signifying that the DR connector is running and ready to receive data on the specified port. See Logging Configuration.
Once the DR connector connects to the GridGain 8 cluster, you can find the connection information in a GridGain 8 sender node's logs:
Check the GridGain 8 Sender Node
Retrieve the ID of a sender node in the source GridGain 8 cluster:
Example output:
Verify that this sender node is configured to send data to the target GridGain 9 cluster:
Make sure that the DataCenterId matches the DR connector's address and ID.
For additional information on how to configure senders see Data Center Replication, Configure Connection Between Clusters.
Verify That Cache Replication Is Enabled
Verify that each source cache has replication enabled by running this command for every source cache:
Example output:
For additional information on how to configure replication for a cache see Data Center Replication, Configure Caches.
Initiate Full State Transfer from GridGain 8
Start full state transfer from the source GridGain 8 cluster:
The command prompts for confirmation before starting. To skip the prompt (for example, in scripts), add the --yes flag.
Alternatively, to start full state transfer for a subset of caches (comma-separated):
This will transfer data from the source GridGain 8 cluster to the target GridGain 9 cluster.
To list active full-state-transfer states, use the list command.
Example output while transfer is in progress:
Wait until the full state transfer completes, when that happens, list command returns an empty result:
Cancel Full State Transfer
To cancel a running full state transfer, use the cancel command.
Example output:
Pause and Resume Replication
To pause replication from the source GridGain 8 cluster, run the pause command.
To resume replication from the source GridGain 8 cluster, run the resume command.
Monitoring Replication
After full state transfer is started, monitor replication progress to determine when the GridGain 9 cluster is caught up and ready for cutover.
Check DR Status on GridGain 8
Use the control script to view the current replication status:
Example output:
Replication JMX Metrics
GridGain 8 Sender Node
Check the replication backlog for a cache via JMX MBean attributes at a sender node of the source GridGain 8 cluster:
DrBatchWaitingAcknowledgeCount
The number of batches successfully processed by the DR connector
DrQueuedKeysCount
The number of entries waiting to be sent. When this value is 0, all updates to the cache have been replicated to the target table.
DR Connector (Per-Cache)
Check JMX MBean attributes of the DR connector for a specific cache:
BatchesReceived
The number of batches received.
BatchesAcked
The number of batches successfully processed.
EntriesReceived
The number of entries received.
EntriesAcked
The number of entries successfully processed.
MessageQueueLength
The number of batches currently being processed. When this value is 0, all updates to this cache have been replicated to the target table.
DR Connector (All Caches)
Check JMX MBean attributes of the DR connector to check overall replication progress:
TotalBatchesAcked
The total number of batches successfully processed across all caches.
TotalEntriesReceived
The total number of entries received across all caches.
TotalEntriesAcked
The total number of entries successfully processed across all caches.
TotalMessageQueueLength
The total number of batches currently being processed across all caches. When this value is 0, all updates from all caches have been replicated to the target cluster.
Monitor DR Connector Logs
Errors or warnings in the connector log typically indicate schema mismatches or connectivity issues. See Troubleshooting for common problems and solutions. By default dr connector writes logs to <dr-connector-directory>/log/dr-receiver-0.log. See Logging Configuration.
Confirm Incremental Sync Is Caught Up
Once the replication backlog on the GridGain 8 side reaches zero:
The full state transfer is complete.
The connector is now applying incremental updates in near-real time.
Any new writes to GridGain 8 will be replicated with minimal delay.
At this point the cluster is ready for cutover. If you continue to write to GridGain 8 during this phase, the backlog will fluctuate but should remain small.
Cutover
The cutover procedure requires a brief maintenance window to ensure no data is generated during the switch.
Pause Writes on GridGain 8
Stop all applications that write to the GridGain 8 cluster. This ensures no new data is generated while you drain the replication backlog.
Wait for the Backlog to Drain
Monitor the replication backlog for each source cache until it reaches zero. The relevant JMX MBean attributes:
GridGain 8 sender node (per cache)
org.apache.ignite.<cache-name>."Data Center Replication".DrQueuedKeysCount
DR connector (overall)
org.apache.ignite.metrics."Data Center Replication".receiver.TotalMessageQueueLength
As a CLI alternative to JMX, use the control script to check the sender node metrics. The StoreSize field equals the number of entries waiting to be sent:
Wait until all of the following are 0:
On the GridGain 8 sender:
DrQueuedKeysCountfor every replicated cache.On the DR connector:
MessageQueueLengthper cache.On the DR connector:
TotalMessageQueueLengthoverall.
This confirms that all data written before the pause has been transferred to GridGain 9.
Verify Row Counts
Compare row counts between GridGain 8 and GridGain 9 to confirm data consistency:
Check the number of records in the target table by executing the following SQL query:
Repeat for each replicated table. The counts must match. Compare results against known values or against the GridGain 8 cluster (which still holds the source data after writes have been paused).
Repoint Client Applications to GridGain 9
Update your application configuration to connect to the GridGain 9 cluster and restart your applications. The specific steps depend on your deployment, but typically involve changing the connection string or endpoint addresses.
Confirm That GridGain 9 Is Serving Traffic
Verify that client applications are successfully connected to GridGain 9 and processing requests. Check application logs and run test queries to confirm normal operation.
Post-Cutover
Stop Replication
Stop the replication process to the target GridGain 9 cluster on the GridGain 8 source cluster.
For non-interactive use, see the --yes-flag note in Pause and Resume Replication.
Then stop the DR connector (see Stopping the Connector).
Clean Up Replication Artifacts
Drop the drVersion Column
The drVersion bookkeeping column is no longer needed after migration is complete. Remove it from each replicated table:
Drop the Tombstones Table
Each time DR connector starts, it automatically creates a table named _DR_TOMBSTONES in the PUBLIC schema to store cache entries removed from the source GridGain 8 cluster. When replication is complete, this table is no longer needed and can be dropped:
Rollback
If validation fails before client applications have been repointed to GridGain 9, you can roll back to GridGain 8.
Rollback Procedure
Ensure that client applications continue to use the GridGain 8 cluster.
Stop the DR connector.
Remove data from the target tables in the GridGain 9 cluster.
Drop the tombstones table in the GridGain 9 cluster.
Investigate and fix the issue that caused validation to fail (for example, schema mismatches or missing data).
Restart the migration from the beginning: a new full state transfer is required.
Troubleshooting
Problem: Table Does Not Exist (Explicit Mapping)
The cache mapping for the source cache refers to a target table that does not exist.
Actions:
Verify the cache mapping for the source cache and update the
tableproperty to refer to the correct table.If the cache mapping is correct, create the target table under the specified qualified name (see the type mapping table).
Problem: Table Does Not Exist (Default Mapping)
When no cache mapping is configured for a source cache, the DR connector creates a default mapping that expects a target table with the same name as the cache in the PUBLIC schema.
Action:
Verify that a cache mapping for the source cache exists. If no mapping exists, create one and restart the replication.
Problem: Missing Bookkeeping Column
The target table is missing the required drVersion bookkeeping column.
Action:
Add the bookkeeping column to the target table:
Then restart the replication.
Problem: Value Too Long
A source cache entry field contains a value that exceeds the column length in the target table.
Action:
Increase the column length in the target table:
Then restart the replication.
Problem: Type Mismatch
There is a type mismatch between the source cache field and the target table column.
Actions:
If the source cache contains entries with the same type for the problem field, this is a configuration issue: the target table column must use the correct type mapping (see the type mapping table).
If the source cache contains entries with different types in the problem field, configure a RowTransformer to handle entries for this cache.
In both cases, drop the target table and re-create it with the correct column types. Then restart the replication.
Problem: Column Does Not Allow NULLs
A source cache entry field contains null values, but the target table column has a NOT NULL constraint.
Actions:
If the column is not part of the primary key, drop and re-create the target table without the
NOT NULLconstraint on the affected column. Then restart the replication.If the column is part of the primary key, configure an
EntryTransformerto replace null values with placeholder values. Update the DR connector's cache mapping to use theEntryTransformer, restart the DR connector, and then restart the replication.
Problem: Unsupported User Type
A source cache entry field contains a user-defined type that GridGain 9 does not support.
Action:
Configure an EntryTransformer to convert the user type to one or more supported columns. Adjust the target table schema if needed. Then restart the replication.
Problem: Network Connectivity Issue Between DR Connector and GridGain 8 / GridGain 9 Cluster
DR connector loses network connectivity with the source GridGain 8 cluster or the target GridGain 9 cluster.
Action:
No action is needed. Network connection is reestablished automatically.
This page is: Copyright © 2026 MariaDB. All rights reserved.
Last updated
Was this helpful?

