DBRM Recovery and Journal Management
The procedure outlines steps for bringing ColumnStore back online when a failed rollback blocks startup, as well as managing timeout configurations for large extent maps. In general, rollback fragments usually exist within vss and vbbm files.
Resolving Stuck load_brm and Failed Rollbacks
If a system is stuck due to a failed transaction rollback, you must manually clear the problematic files. The procedures differ depending on your deployment type.
Local Storage Deployments
For local storage deployments, you can refer to columnstore_review.sh --clearrollback. To perform the recovery manually, follow these steps:
Verify Master State: On the master node, shut down the cluster and check the BRM saves.
mcsShutdown
cat /var/lib/columnstore/data1/systemFiles/dbrm/BRM_saves_currentThe output should be
"BRM_saves".If it is not, you will need to reload a backup extent map.
Terminate Leftover Processes: On each node, check for and kill any remaining ColumnStore processes (aside from
mariadbd).
ps -ef | grep ^mysql
ps -ef | grep -E '(PrimProc|ExeMgr|DMLProc|DDLProc|WriteEngineServer|StorageManager|controllernode|workernode)' | grep -v "grep" | awk '{print $2}' | xargs killVerify Configuration and Restart CMAPI: On each node, verify the configuration files match and restart the CMAPI service.
ll /etc/columnstore/Columnstore.xml
clearShm
systemctl restart mariadb-columnstore-cmapiBackup DBRM Files: On the master node, back up the system dbrm files before proceeding.
Clear VSS and VBBM files: On the master node, truncate the
vssandvbbmfiles. This sends a NULL value to clear them out, allowing a fresh copy upon restart.
Restart System: In a separate session, monitor the logs while starting the system.
Advanced Local Troubleshooting
If, during startup, /var/log/messages indicates a rollback is still trying to process after clearing BRM_saves_vss and BRM_saves_vbbm (e.g., showing DMLProc starts rollbackAll or DMLProc is rolling back transaction), follow these steps:
Delete version buffer and journal: Truncate both files.
Force clear table locks: If issues persist, clear any table locks in addition to the prior truncations.
Reload extent map: If the system is still unable to start, try reloading a backup extent map from prior to the lock or rollback occurrence.
S3 Deployments
For deployments utilizing S3, follow this modified procedure:
Verify Master State: Shut down the cluster and check
BRM_saves_current(should return "BRM_saves", otherwise reload a backup extent map).
Terminate Leftover Processes & Restart CMAPI: Kill hanging processes on each node, clear shared memory, and restart CMAPI.
Backup Metadata: On the master node, back up the storage manager metadata.
Clear Meta Files and Cache: Remove and recreate the
vssandvbbmmetadata files, then purge the storage manager cache.
Restart System: Tail the logs and start the system.
Managing Timeouts for Large Extent Maps
When processing extremely large extent maps (e.g., massive cpimports exceeding 5 billion records) or experiencing long shutdowns for brm_save, adjusting default timeouts may be necessary.
Systemd Service Timeouts
You can raise the standard systemd timeouts for worker and controller nodes:
Worker Nodes: Raise
TimeoutStopSecandTimeoutStartSecto1800.Controller Node: Raise
TimeoutStopSecto900for massive cpimports.After modifying these values, apply them via
systemctl daemon-reload.
Long Shutdowns for BRM Save
To adjust timeout variables for DML processing and DBRM loading:
DMLProc: Open
/usr/lib/systemd/system/mcs-dmlproc.serviceand setTimeoutStopSec=15minandTimeoutStartSec=15min.LoadBRM: Open
mcs-loadbrm.serviceand setTimeoutStopSec=1800andTimeoutStartSec=1800.mcsShutdown Alias: Update the timeout for the shutdown command in the alias script to
900.
CMAPI Timeouts
Occasionally, CMAPI may force a restart of all processes every few seconds if operations exceed its default thresholds. To increase the CMAPI timeout:
Stop the service and clear shared memory.
Edit the CMAPI helpers file.
On line 290, raise the timeout to a higher number (e.g., higher than 120 or 300).
Restart the service.
This page is: Copyright © 2026 MariaDB. All rights reserved.
Last updated
Was this helpful?

