ColumnStore Bulk Write SDK

You are viewing an old version of this article. View the current version here.

Introduction

Starting with MariaDB ColumnStore 1.1 a C++ SDK is available which supports bulk write into ColumnStore. Conceptually this is an API version of cpimport. The SDK is intended to be integrated by custom code and adapters to enable easier publishing of data into ColumnStore.

The API is licensed under LGPLv3.

Getting Started

Prebuilt binary packages may be downloaded here or you can build from scratch from here. To build from scratch see the appropriate building document below in the Documentation section, latest version here.

Package Installation

RHEL / CentOS 7

The following libraries need to be installed on the system for the binary package install:

yum install epel-release
yum install libuv libxml2 snappy

The API rpm can be installed:

rpm -ivh mariadb-columnstore-api-*-centos7.rpm

Getting Started with C++

The documentation below is the best place to get started with building and developing against the C++ SDK. Some sample programs are installed to /usr/share/doc/mcsapi/ for review.

Getting Started with Java

The Java version of the SDK provides a very similar API to the C++ one so the pdf documentation can generally be transposed 1 for 1 to understand the API calls. Since the Java version is a wrapper on top of the C++ API the underlying library must be loaded using a static initializer once in your program.

    static {
        System.loadLibrary("_javamcsapi");
    }

The corresponding java jar must be also be included in the java classpath. The packaged install is built and tested with OpenJDK 8.

First a simple table is created with the mcsmysql client:

MariaDB [test]> create table t1(i int, c char(3)) engine=columnstore;

Next create a file MCSAPITest.java with the following contents:

import com.mariadb.columnstore.api.*;

public class MCSAPITest {
    // load java mcsapi native library.
    static {
        System.loadLibrary("_javamcsapi");
    }

    public static void main(String[] args) {
        ColumnStoreDriver d = new ColumnStoreDriver();
        ColumnStoreBulkInsert b = d.createBulkInsert("test", "t1", (short)0, 0);
        try {
            b.setColumn(0, 2);
            b.setColumn(1, "XYZ");
            b.writeRow();
            b.commit();
        }
        catch (Exception e) {
            b.rollback();
            e.printStackTrace();
        }
    }
}

Now compile and run the program, here this assumes the jar file is in the default install location of /usr/lib64:

[david@centos dev]$ javac -classpath ".:/usr/lib64/javamcsapi.jar" MCSAPITest.java
[david@centos dev]$ java -classpath ".:/usr/lib64/javamcsapi.jar" MCSAPITest

Now back in mcsmysql verify the data is written:

MariaDB [test]> select * from t1;
+------+------+
| i    | c    |
+------+------+
|    2 | XYZ  |
+------+------+

Getting Started with Python

The current package install supports Python 2.x only and once installed the library is available for immediate use on the system.

First a simple table is created with the mcsmysql client:

MariaDB [test]> create table t1(i int, c char(3)) engine=columnstore;

For this simple test the python CLI will be used by simply running the python program with no arguments and entering the following:

import pymcsapi
driver = pymcsapi.ColumnStoreDriver()
bulk = driver.createBulkInsert('test', 't1', 0, 0)
bulk.setColumn(0,1)
bulk.setColumn(1, 'ABC')
bulk.writeRow()
bulk.commit()

In interactive command line mode, the bulk.setColumn and bulk.writeRow methods return the bulk object to allow for more concise chained invocation. You may see something like the following as a result which is normal:

>>> bulk.setColumn(0,1)
<pymcsapi.ColumnStoreBulkInsert; proxy of <Swig Object of type 'mcsapi::ColumnStoreBulkInsert *' at 0x7f0d5295bcc0> >

Now back in mcsmysql verify the data is written:

MariaDB [test]> select * from t1;
+------+------+
| i    | c    |
+------+------+
|    1 | ABC  |
+------+------+

Documentation

The following documents provide SDK documentation:

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.