> For the complete documentation index, see [llms.txt](https://mariadb.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mariadb.com/docs/analytics/mariadb-columnstore/management/deployment/upgrades/refined-multi-node-upgrade-sequencing-6.x-to-23.10.md). # Refined Multi-Node Upgrade Sequencing: 6.x to 23.10 These instructions detail the comprehensive upgrade path from MariaDB ColumnStore 6 to MariaDB ColumnStore 23.10 in a Multi-Node topology. To ensure a successful upgrade, it is highly critical to execute package updates simultaneously across all nodes and ensure strict CMAPI synchronization. ### 1. Pre-Upgrade Preparation Before making any modifications, prepare your cluster and secure backups of your current state. * Perform Backups: Consider taking a full production backup using `bash mcs_backup_manager.sh backup` (note that this will lock out writes during the backup). Determine the primary node by running `mcs cluster status` and take a DBRM backup using `bash mcs_backup_manager.sh dbrm_backup -bl -nb before-shutdown-before-upgrade`. * Save Configurations: On every node, create a timestamped directory and save backups of `Columnstore.xml`, `storagemanager.cnf`, `cmapi_server.conf`, and `server.cnf`. * Set Maintenance Mode: On the MaxScale node, set each replica server to maintenance mode to prevent traffic routing during the upgrade. You can do this using MaxCtrl (e.g., `maxctrl set server mcs2 maintenance`). Verify this by running `maxctrl list servers`; the state should reflect `Maintenance`. * Disable GTID Strict Mode: Temporarily disable the `gtid_strict_mode` system variable on each replica server to reduce the chance of replication issues. Comment out `gtid_strict_mode=1` in your configuration files (e.g., `/etc/my.cnf.d/server.cnf`). ### 2. Coordinated Service Shutdown Services must be completely shut down across the cluster prior to upgrading packages. 1. Stop Cluster: On the primary node, stop the ColumnStore cluster by executing `mcs cluster stop`. 2. Stop Node Services: On *every* node, stop the MariaDB Enterprise Server, MariaDB ColumnStore, and CMAPI services. ```bash sudo systemctl stop mariadb-columnstore-cmapi sudo systemctl stop mariadb-columnstore sudo systemctl stop mariadb ``` 3. Terminate Orphaned Processes: On every node, forcefully kill any remaining ColumnStore processes if needed. Check for processes like `PrimProc`, `ExeMgr`, `DMLProc`, `DDLProc`, `WriteEngineServer`, `StorageManager`, `controllernode`, `workernode`, `load_brm`, or `save_brm`, and execute `kill -9` on their PIDs. Stop all individual `mcs-*` systemctl services. 4. Final Pre-Upgrade Backup: Take one more DBRM backup on the primary node before proceeding with the installation. ### 3. Simultaneous Package Upgrades {% hint style="danger" %} To prevent version mismatch issues within the cluster configuration, package updates must be executed at the exact same time across all nodes. {% endhint %} 1. Configure Package Manager: On every node, configure the package repository using the MariaDB Enterprise Release Helper script. Pass the target version using the `--mariadb-server-version="11.4"` flag (as Enterprise ColumnStore 23.10 is included with MariaDB Enterprise Server 11.4). 2. Execute Update: On *every* node at the same time, execute the package update. * For YUM (RHEL/CentOS): `sudo yum update "MariaDB-*" "MariaDB-columnstore-engine" "MariaDB-columnstore-cmapi"`. * For APT (Debian/Ubuntu, version 2.0+): `sudo apt install --only-upgrade '?upgradable ?name(mariadb.*)'`. 3. Disable Standalone Service: After the packages are updated, the standalone `mariadb-columnstore` service must be disabled on each node, as it will now be controlled directly by CMAPI. ```bash sudo systemctl stop mariadb-columnstore sudo systemctl disable mariadb-columnstore ``` ### 4. CMAPI Synchronization and Service Startup With packages updated, services must be brought back online in a specific order to allow CMAPI to synchronize the cluster state. 1. Start Services: On every node, start and enable the MariaDB server and CMAPI services. ```bash sudo systemctl start mariadb-columnstore-cmapi sudo systemctl enable mariadb-columnstore-cmapi sudo systemctl start mariadb sudo systemctl enable mariadb ``` 2. Verify Status: On the primary node, check the status using `mariadb -e "show status like '%Columnstore%';"` and `mcs cluster status`. Ensure all nodes and dbroots are present. (If there are issues, you may need to restore the `Columnstore.xml`backed up earlier and run `clearShm` ). 3. Start Cluster: On the primary node, start the cluster using `mcs cluster start`. ### 5. Post-Upgrade Finalization 1. Write Binary Log: On the primary server, run `mariadb-upgrade --write-binlog` to upgrade the data directory with binary logging enabled to update system tables. 2. Verify Versions: On each node, query `SHOW GLOBAL STATUS LIKE 'Columnstore_version';` to confirm the version reads `23.10.0`, and `SHOW GLOBAL VARIABLES LIKE 'version';` to check the ES version. 3. Data Validation: Run data checks (e.g., `sudo bash table_checker.sh -c 6 -m 1`) and compare output counts from before the upgrade. Confirm replication users are connected via `show processlist`. 4. Restore Configurations: Re-enable the `gtid_strict_mode` system variable on each replica server. Restart MariaDB to apply the changes. 5. Clear Maintenance Mode: Finally, on the MaxScale node, clear the maintenance mode for each replica using MaxCtrl (e.g., `maxctrl clear server mcs2 maintenance`). Confirm the mode is cleared by checking that the state no longer says `Maintenance` in `maxctrl list servers`. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://mariadb.com/docs/analytics/mariadb-columnstore/management/deployment/upgrades/refined-multi-node-upgrade-sequencing-6.x-to-23.10.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.