MariaDB Xpand Crunches Cockroach with Sysbench

This blog compares the performance of two distributed SQL databases, MariaDB Xpand and CockroachDB, using each vendor’s respective cloud offering.

Evaluating distributed SQL offerings can be challenging. Comparison of instance prices will not yield a correct assessment because you may find that one database performs more work with fewer, cheaper instances than another. Moreover, two databases at their default configuration may not be an equal comparison. With this blog, we hope to make this evaluation simpler with a comparison that provides the total throughput of MariaDB Xpand and CockroachDB using the same configuration. We will follow this blog with a more in-depth paper to assist software teams in evaluating MariaDB Xpand and CockroachDB on price, performance and scale.

MariaDB Xpand is used by large-scale applications with millions of users by global companies such as Samsung and Massive Media. Xpand excels at read and write scale data applications and provides absolute availability even with multiple zone failures. Additionally, Xpand offers columnar indexes to enable ad hoc, operational and personal analytics on transactional data. This capability precludes the need to copy data to an analytical database for intraday and real-time analytics. Xpand is compatible with MariaDB and MySQL and is available on-premises or via SkySQL, MariaDB’s Database-as-a-Service offering.

CockroachDB claims scalability, availability, and disaster recovery. It is available as an on-premises installation or via Cockroach’s Cloud. CockroachDB includes partial compatibility with PostgreSQL. It does not currently support columnar indexes or state support for operational analytics.

Just the Results

In each test, Xpand on SkySQL outperformed and outscaled CockroachDB on Cockroach Cloud with dedicated instances. Sysbench was used for testing. These are the findings:

  • OLTP Point Select – this is a query by id similar to: SELECT c FROM x WHERE id=?
    • Xpand scaled to 1024 threads and a throughput of 250977 operations. There were no errors.
    • CockroachDB scaled to 128 threads and a throughput of 59427 operations. There were no errors.

Sysbench oltp_point_select throughput graph

 

  • OLTP Read/Write – a mix of 90 percent reads and 10 percent writes
    • Xpand scaled to 1024 threads with a throughput of 19616 operations. There were no errors.
    • CockroachDB scaled to 128 threads and throughput of 3706 operations. There were errors during the test.

Sysbench oltp_read_write throughput graph

 

Interpretations

It appears that in a standard Xpand on SkySQL configuration and with a default CockroachDB on CockroachCloud configuration, Xpand greatly outscales and outperforms CockroachDB. It seems that the primary reason is that CockroachDB is CPU bound because the number of threads is limited to three times the number of vCPUs. We were unable to achieve results similar to what is claimed on their performance page (https://www.cockroachlabs.com/docs/v19.2/performance.html). Evaluators should note that Cockroach’s only published results were not produced using their database as a service. Except for changes required to ensure the same level of consistency for equal comparison, we left products to default configurations as this is what a customer would expect to work.

Price/Performance

While the two services are close in terms of price for the same configuration, given the throughput and latency advantage of Xpand, CockroachDB is significantly more expensive on a cost-per-operation basis. Software development teams will be able to use a smaller cluster to achieve higher performance with Xpand than they will with CockroachDB. We will expand and explore this topic in a webinar on Mar 24, 2022 entitled MariaDB Xpand vs CockroachDB and in this in-depth white paper, Evaluating MariaDB Xpand and CockroachDB with Sysbench.

Infrastructure Configuration

For this benchmark, we used as close to the default configuration for each cloud as possible. Modifications were required in order to ensure an apples to apples configuration.

  • Driver
    • 32 vCPU
    • 128 GB
    • sysbench 1.1.0-b73ae9a
    • CentOS Linux 7 (Core)
  • For both Xpand and CockroachDB
    • AWS us-west-2
    • Three 16×64 instances
    • 13500 IOPS
    • RAM 64 GiB
    • EBS Storage 900GiB
  • For CockroachDB
    • Version 21.2.5
    • Configured to three nodes instead of the default four
  • For Xpand
    • Version 6.0.2
    • Configured to three replicas instead of the default two
    • Two instances of MaxScale database proxy

Learn More

We are pleased to announce a webinar on Mar 24, 2022 entitled MariaDB Xpand vs CockroachDB. A more in-depth paper, “Evaluating MariaDB Xpand and CockroachDB with Sysbench,” provides commands and information to reproduce these results, pricing information, and tuning advice for both SkySQL and CockroachDB.

Try Xpand

Start your trial today or for the easiest and fastest way to get started use Xpand in the cloud. Help is available on the MariaDB Community Slack.