Storage

When a table is created on MariaDB ColumnStore, the system creates at least one file per column for the table. So, for example, a table created with three columns would have at a minimum, three separately-addressable logical objects created on a SAN or a server’s disk drives.

• Each column is stored independently in 8M rows logical measure called an Extent. An extent for 1 byte datatypes consume 8MB; 2 byte datatypes consume 16MB; 4 byte datatypes consume 32MB; 8 byte datatypes consume 64MB; and variable size datatypes consume 64MB. Once an extent becomes full, a new extent is automatically created.
• An Extent is physically stored as collection of blocks.
• A block is 8K Bytes. Every database block in a database is uniquely identified by its Logical Block Identifier (LBID)
• String columns > 8 characters store indexes in the main column file and actual values in separate dictionary files.
• A segment file is a physical file on disk that holds a column’s data. Once a segment file reaches its maximum number of extents, a new segment file is automatically created
• Collectively, all the column's segment files for one ore more extents form a Partition
• Partitions are stored in a hierarchical structure organized by segments (i.e. folders)
• MariaDB ColumnStore meta-store maps file structure/location to DB schema as well as information used for partitioning
• By default the data is compressed

Extent Map

MariaDB ColumnStore utilizes a smart structure known as the Extent Map which provides logical range partitioning and removes the need for indexing, manual table partitioning, materialized views, summary tables, and other structures and objects that row-based databases must implement for query performance.

An extent is a logical block of space that exists within a physical segment file, and is anywhere from 8-64MB in size. Each extent supports the same number of rows, with smaller data types using less space on disk.

The Extent Map catalogs all extents and their corresponding blocks (LBID’s) along with minimum and maximum values for a column’s data within an extent.

A master copy of the Extent Map exists on the primary Performance Module. Upon system startup, the file is read into memory and then physically copied to all other participating User and Performance Modules for disaster recovery and failover purposes. All nodes keep the Extent Map resident in memory for quick access. As extents are modified, updates are broadcasted to all participating nodes as well.

How Extent Map Works

Extent Map provides the ability logical range partitioning and ability to only retrieve the blocks needed to satisfy a query. This is accomplished via what is called "extent elimination" in MariaDB ColumnStore. 'Extent Elimination' is process of eliminating those extents for the purpose of I/O operation that do not meet the join and filter condition of a query.

Extent elimination is first accomplished in MariaDB ColumnStore via only scanning columns in join and filter conditions. Then the logical horizontal partitioning information of each extents along with min and max value for the column is used for further elimination. To eliminate an extent when a column scan involves a filter, that filter value is compared to the minimum and maximum values stored in each extent for the column: If the filter value is outside of a extent's min-max value range, the extent is eliminated

This automatic extent elimination behavior is well suited for series, ordered, patterned, or time-based data where data is loaded frequently and often referenced by time. Any column with clustered values is a good candidate for extent elimination.

Example: