ColumnStore Bulk Write SDK

You are viewing an old version of this article. View the current version here.

Introduction

Starting with MariaDB ColumnStore 1.1 a C++ SDK is available which supports bulk write into ColumnStore. Conceptually this is an API version of cpimport. The SDK is intended to be integrated by custom code and adapters to enable easier publishing of data into ColumnStore.

The API is licensed under LGPLv3.

Getting Started

Prebuilt binary packages may be downloaded here or you can build from scratch from here. To build from scratch see the appropriate building document below in the Documentation section, latest version here.

Getting Started with C++

The documentation below is the best place to get started with building and developing against the C++ SDK. Some sample programs are installed to /usr/share/doc/mcsapi/ for review.

Getting Started with Java

The Java version of the SDK provides a very similar API to the C++ one so the pdf documentation can generally be transposed 1 for 1 to understand the API calls. Since the Java version is a wrapper on top of the C++ API the underlying library must be loaded using a static initializer once in your program.

    static {
        System.loadLibrary("_javamcsapi");
    }

The corresponding java jar must be also be included in the java classpath.

First a simple table is created with the mcsmysql client:

MariaDB [test]> create table t1(i int, c char(3)) engine=columnstore;

Next create a file MCSAPITest.java with the following contents:

import com.mariadb.columnstore.api.*;

public class MCSAPITest {
    // load java mcsapi native library.
    static {
        System.loadLibrary("_javamcsapi");
    }

    public static void main(String[] args) {
        ColumnStoreDriver d = new ColumnStoreDriver();
        ColumnStoreBulkInsert b = d.createBulkInsert("test", "t1", (short)0, 0);
        try {
            b.setColumn(0, 2);
            b.setColumn(1, "XYZ");
            b.writeRow();
            b.commit();
        }
        catch (Exception e) {
            b.rollback();
            e.printStackTrace();
        }
    }
}

Now compile and run the program, here this assumes the jar file is in the default install location of /usr/lib64:

[david@centos dev]$ javac -classpath ".:/usr/lib64/javamcsapi.jar" MCSAPITest.java
[david@centos dev]$ java -classpath ".:/usr/lib64/javamcsapi.jar" MCSAPITest

Now back in mcsmysql verify the data is written:

MariaDB [test]> select * from t1;
+------+------+
| i    | c    |
+------+------+
|    2 | XYZ  |
+------+------+

Getting Started with Python

With the package installed Python 2.x should be able to consume the API directly.

First a simple table is created with the mcsmysql client:

MariaDB [test]> create table t1(i int, c char(3)) engine=columnstore;

For this simple test the python CLI will be used:

[david@centos pdf]$ python
Python 2.7.5 (default, Aug  4 2017, 00:39:18)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymcsapi
>>> driver = pymcsapi.ColumnStoreDriver()
>>> bulk = driver.createBulkInsert('test', 't1', 0, 0)
>>> bulk.setColumn(0,1)
>>> bulk.setColumn(1, 'ABC')
>>> bulk.writeRow()
>>> bulk.commit()
>>>

Now back in mcsmysql verify the data is written:

MariaDB [test]> select * from t1;
+------+------+
| i    | c    |
+------+------+
|    1 | ABC  |
+------+------+

Documentation

The following documents provide SDK documentation:

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.