October 12, 2016

Leveraging MariaDB in NEOs Distributed Storage Architecture

By Guest

The following is a guest blog post from Sven Franck, marketing manager at Nexedi, one of the largest OSS publishers in Europe with most solutions based on prior research projects. Headquarters are located in Lille, France with Nexedi offices serving clients around the world in Germany, Japan and China.

In this blog post, we’ll show why we chose MariaDB as a core component of the NEO system architecture. NEO is our distributed, redundant and transactional storage system, designed to be an alternative to ZEO and FileStorage.

The volume of data we’re seeing is growing across the board, with IoT, big data and machine learning becoming more relevant, and projects such as Wendelin gaining industry traction. Because our services often range from network infrastructure to business applications, we’re increasingly putting greater emphasis on efficient data handling, which is making accessibility, availability, distribution and performance a high-priority.

NEO (short for Nexedi Enterprise Object database) was developed to replace all implementations of ZEO (Zope Enterprise Object database) being used at Nexedi as well as to switch to an architecture that could be scaled automatically, while also providing persistence and resiliency. During development, SQL and MariaDB proved to be a good choice because they met our immediate needs while also being flexible enough to reshape tables as the design of NEO grew. This was key because frequent schema changes in early project phases were expected. MariaDB allowed us to focus on specific features of NEO, while not having to call the low-level InnoDB BTree implementation directly, which reduced development time considerably.

Leveraging the Power of MariaDB

NEO uses MariaDB to store low level object, transaction and location data, and makes these accessible via the "medium" ZOPE layer, which our main software solutions, likeERP5, are running on. Using MariaDB, we can overcome the shortcomings of ZODBs traditional way of maintaining indices, and are now able to easily scale our application architectures automatically and depending on need. MariaDB also provides well-tested durability, which is especially important for the more advanced features of NEO, such as storing data in non-chronological order when lagging nodes are catching up or a node's workload is reduced by moving data to another node. Finally, since most of our implementations are running long-term, MariaDB also provides the needed stability in terms of source code availability - it's a proven open-source embeddable solution.

As part of NEO, MariaDB was included in all latest ERP5 implementations as well as our internal system and test infrastructure (30 users, 70 test nodes), currently handling indices of about 97GB (TokuDB) or 133GB (InnoDB backup), which without NEO compression would equate to a FileStorage equivalent between 250GB and 350GB.

Scaling MariaDB into the Future

For the past couple of years, we’ve been increasing our efforts to contribute back to MariaDB, because we believe there should be open-source industry-grade software solutions available, and in terms of database technologies, we consider MariaDB an essential solution. We are proud of what we have accomplished using MariaDB and will continue our support, which includes working on joint research projects and helping to improve the code of MariaDB itself.

Additional Resources