Overview

MariaDB ColumnStore is a columnar storage engine. This is the second GA release in the ColumnStore 1.4 series. This release contains new features and fixes, compared to MariaDB ColumnStore 1.4.3.

This release of MariaDB ColumnStore is included with MariaDB Enterprise Server 10.4.13-7.

MariaDB ColumnStore 1.4.4 was released on 2020-06-08.

Notable Changes

columnstore_use_import_for_batchinsert system variable option ALWAYS added to use cpimport for LOAD DATA LOCAL INFILE and INSERT .. SELECT regardless of whether the query runs in a transaction. This gives the user the ability to use a faster import method, with the caveat that if a user issues a rollback of the transaction, it will have no effect as the data would have already been committed to actual database files by cpimport. ()

Issues Fixed

Can result in data loss

• can modify incorrect data due to incorrect WHERE clause handling. ()

Can result in a hang or crash

• A hang may occur in load_brm on dbroot1 failover. ()

• may be blocked by improper failover. ()

• Crash may occur when using cpimport () to process data containing multi-byte characters. ()

Can result in unexpected behavior

• over negative SEC_TO_TIME() results produce an incorrect order. ()

• Subqueries with ORDER and LIMIT may produce the wrong answer. ()

• clause in WHERE could be evaluated improperly. ()

Related to performance

• Subquery wrapping impacts performance. ()

• Unoptimized lines (CPU impact and unnecessary rounding error) in regr_sxx, regr-sxy, and regr_syy ()

Related to install and upgrade

• postConfigure uses invalid device nodes for dbroots. ()

• startsystem allows startup without correct DBROOT 1 assignment. ()

• Incorrect handling of unassigned IP addresses in configuration file. ()

Known Issues

• Built-in SQL functions do not work in ColumnStore when sql_mode=ORACLE ()

Interface Changes

• None.

Platforms

In alignment to the MariaDB Corporation Engineering Policy, MariaDB ColumnStore 1.4.4 is provided for:

• Red Hat Enterprise Linux 8

• Red Hat Enterprise Linux 7

• CentOS 8

• CentOS 7

Note: MariaDB Enterprise ColumnStore 1.4 is no longer supported. If you would like to deploy Enterprise ColumnStore, please use MariaDB Enterprise ColumnStore 5 or later. For installation and upgrade instructions, see "Deploy".

Overview

MariaDB ColumnStore is a columnar storage engine. This is the first GA release in the ColumnStore 1.4 series. This release contains new features and fixes, compared to MariaDB ColumnStore 1.4.2.

This release of MariaDB ColumnStore is included with MariaDB Enterprise Server 10.4.12-6.

MariaDB ColumnStore 1.4.3 was released on 2020-03-02.

Notable Changes

• MariaDB ColumnStore 1.4.3 is the first Generally Available (GA) release of the ColumnStore 1.4 release series. ()

• Support added for ALTER TABLE ... ENGINE=ColumnStore ()

• Support added for CREATE TABLE ... ENGINE=ColumnStore AS SELECT ... ()

Issues Fixed

Can result in data loss

• Potential data loss with S3 StorageManager when mcsadmin suspendDataWrites returns before S3 sync is complete. ()

Can result in a hang or crash

• Server aborts if Columnstore.xml is missing. (, )

• A User Module (UM) join memory overflow can cause session hang and fail to free memory. ()

• Improper StorageManager shutdown can result if process is killed by fatal error. ()

• Truncation during

Can result in unexpected behavior

• Incorrect comparison of padded string vs. non-padded column. ()

• Incorrect query validation for complex ORDER BY ()

• subquery does not return rows with NULL qualifying column values. ()

• GROUP BY

• StorageManager may log inconsistently. ()

• with DISTINCT and two columns may fail. ()

• If min/max are set, incorrect results may be provided after a cpimport due to incorrect extent elimination. ()

• Impossible

Related to performance

• Performance regression in and ()

• Slow DDL statement execution after a restart. ()

Related to install and upgrade

• postConfigure aborted due to missing /tmp/columnstore_tmp_files directory. ()

• postConfigure output 'unassigned' alias script name. ()

• columnstore-post-install isn't triggered during package install. (

Known Issues

• Performance of some queries, such as those containing UNION, may be worse than on ColumnStore 1.2.x.

• and queries currently fall back to a slower execution method.

Interface Changes

• None.

Platforms

In alignment to the MariaDB Corporation Engineering Policy, MariaDB ColumnStore 1.4.4 is provided for:

• Red Hat Enterprise Linux 8

• Red Hat Enterprise Linux 7

• CentOS 8

• CentOS 7

Note: MariaDB Enterprise ColumnStore 1.4 is no longer supported. If you would like to deploy Enterprise ColumnStore, please use MariaDB Enterprise ColumnStore 5 or later. For installation and upgrade instructions, see "Deploy".

Overview

MariaDB ColumnStore is a columnar storage engine. This is the first release in the ColumnStore 1.4 series. This release contains new features and fixes, compared to MariaDB ColumnStore 1.2.5.

This release of MariaDB ColumnStore is included with MariaDB Enterprise Server 10.4.11-5.

MariaDB ColumnStore 1.4.2 was released on 2020-01-06.

MariaDB Server Convergence

Until now, MariaDB ColumnStore has been maintained as a custom fork of MariaDB Server, to handle the unique way that queries are handled for distributed processing.

With this release, a joint project between the MariaDB Server and MariaDB ColumnStore engineering teams, ColumnStore now works as a pluggable storage engine on the standard MariaDB Enterprise Server 10.4 platform.

MariaDB Enterprise Server 10.4 includes distributed processing engine support features. These features are not present in the older 10.3 and 10.2 release series.

A standard MariaDB Server is now used for ColumnStore UM (User Module) nodes. ColumnStore users can now enjoy the benefits of MariaDB Server 10.4, and MariaDB Server 10.4 users are now able to deploy ColumnStore on top of their existing stack.

New Features

S3 Storage Manager

MariaDB ColumnStore now has the ability to use any object store that is Amazon S3 API compatible. The new Storage Manager uses a persistent disk cache for read/write operations so that it has minimal performance impact on ColumnStore. In some cases it will perform better than local disk operations.

Usage instructions:

1. Before running postConfigure, edit the storagemanager.cnf configuration file to specify the S3 connection parameters (as detailed in the S3 section of that file), and the local machine configuration (as detailed in the ObjectStorage and Cache sections). The configuration file is documented in-line. Enable the Storage Manager by setting ObjectStorage/Service to S3

2. Run postConfigure, and when promoted for type of storage, select the StorageManager option.

cpimport S3 Support

cpimport is a high-speed bulk data loading utility for ColumnStore. cpimport now includes command-line options for loading a CSV file from Amazon S3 (and compatible) buckets.

When these options are set, cpimport will use the path/filename provided to load an object from object storage instead of a local file. Current behavior is to download the entire file into memory before processing.

Expanded Data Type Support

• (with CURRENT_TIMESTAMP)

Please note that for cpimport the current system time of the PM node is used.

MODA() Mode Average UDAF

The MODA() UDAF (User-Defined Aggregate Function) determines the mode average. MODA() has tie-break behavior to use the closest to the average, and then the smallest absolute value.

Statement-Based Replication Support

Statement-based replication into ColumnStore tables is supported by setting columnstore_replication_slave=on on the UM that will apply the replication data. Row-based replication events on ColumnStore replica (slave) tables will currently fail, generating an error viewable with

Notable Changes

Performance Enhancements

• The performance of BRM (Block Resolution Manager) snapshots has been increased for improved performance when committing data to ColumnStore.

• To reduce SSD wear and increase write performance for large data sets containing many columns, ColumnStore now allocates disk as-needed, writing only real data and padding to fill the remainder of an 8KB block. ColumnStore previously wrote twice -- once to pre-allocate an empty file for each new extent (8 million item file for a column), and a second time to fill the file with real data.

• The outer "ORDER BY" of a query is now processed using ColumnStore's engine instead of MariaDB server. This uses a faster sorting algorithm for higher performance with larger result sets.

InfiniDB Alias Eliminated

ColumnStore 1.2 and earlier included the InfiniDB engine as an alias. This alias has now been removed. All ColumnStore tables must now be created with the engine name "columnstore". All MariaDB system variables prefixed with "infinidb_" have now been removed.

vtable Replaced by Query Execution Handlers

vtable has been replaced with a set of query execution handlers: Select Handler, Derived Handler, and table API mode.

The vtable mode switch (infinidb_vtable_mode system variable) has been eliminated. Two new session variables have been added: columnstore_select_handler, columnstore_derived_handler

The Select Handler is the replacement for a vtable, and is the default query execution handler. It is expected to provide the fastest execution path for the whole query.

Select Handler lacks support for some vtable features, including:

• Cartesian JOIN

• Non-Equi JOIN

• INSERT .. SELECT

• SELECT INTO OUTFILE

If the Select Handler fails to execute a query, an error is returned. If a query fails under the Select Handler, set columnstore_select_handler=off for the session. This will cause the Server to hand-off query execution to the Derived Handler. The query must be restarted after the session variable has been set.

If the Derived Handler fails to execute a query, an error is returned. If a query fails under both the Select Handler and Derived Handler, set columnstore_select_handler=off and columnstore_derived_handler=off for the session. This will cause table API execution, an equivalent to disabled vtable mode in ColumnStore 1.2.x and earlier. The query must be restarted after the session variables have been set.

Deployment Methods

ColumnStore 1.4 is included with MariaDB Enterprise Server 10.4 on select Platforms.

ColumnStore is available for deployment from package tarball and repository. ColumnStore is not available for deployment from binary tarballs.

"Distributed Install" Method Eliminated

The "distributed install" method which pushed packages onto other nodes during postConfigure has been removed. ColumnStore packages must now be installed on all nodes prior to startup.

Configuration Path Changes

ColumnStore XML configuration files have moved to /etc/columnstore

MariaDB Enterprise Server configuration options for ColumnStore have moved to /etc/my.cnf.d/columnstore.cnf and the default MariaDB Enterprise Server my.cnf will load this file.

Data Directory Path Change

The ColumnStore data directory has moved to /var/lib/columnstore and is separate from the MariaDB Server data directory at /var/lib/mysql

Executable Path Changes

ColumnStore binaries have moved to /usr/bin or /usr/sbin, and the libraries are in the OS standard /usr library path. Some ColumnStore binaries have been renamed to avoid conflict, including:

User Account for Cross-Engine Joins

Cross-engine joins depend on TCP connection from ExeMgr to the Server process. Since the database root user in MariaDB Enterprise Server 10.4 authenticates only by UNIX socket, a dedicated user must be created to support cross-engine joins. The cross engine section of Columnstore.xml should be edited accordingly.

Issues Fixed

Can result in data loss

• DBRM files could be deleted during a failover scenario. ()

Can result in a hang or crash

• Certain window function queries could crash the Server process. ()

Can result in unexpected behavior

• group_concat() with DISTINCT concatenates even non-distinct values. ()

• Wrong results could be returned for a complex query with subquery and window functions over decimal(12,4) column. ()

• Pipe operator (|) could return wrong results. ()

Related to install and upgrade

• Cosmetic typo in postConfigure output. Output upon success is now MariaDB ColumnStore Install Successfully Completed, System is Active ()

Known Issues

• Performance of some queries, such as those containing UNION, may be worse than on ColumnStore 1.2.x.

• and queries currently fall back to a slower execution method.

Interface Changes

• Columnstore_commit_hash status variable added

• Columnstore_version status variable added

• columnstore_compression_type system variable added

Platforms

In alignment to the MariaDB Corporation Engineering Policy, MariaDB ColumnStore 1.4.4 is provided for:

• Red Hat Enterprise Linux 8

• Red Hat Enterprise Linux 7

• CentOS 8

• CentOS 7

Note: MariaDB Enterprise ColumnStore 1.4 is no longer supported. If you would like to deploy Enterprise ColumnStore, please use MariaDB Enterprise ColumnStore 5 or later. For installation and upgrade instructions, see "Deploy".