ScaleDB ONE: Let’s Get Started

ScaleDB 15.10 is out. Some users have downloaded it and tested it and we have received pretty positive feedback, but also some requests to have more info and help on how to start. I will try to condense here the basic steps to install and test ScaleDB ONE for the first time.

First of all, some terminology. We have two versions: ScaleDB ONE and ScaleDB Cluster. ScaleDB ONE is meant to be used on a single machine (ONE = One Node Edition), whether it is a VM, a cloud instance or a physical server, whilst ScaleDB Cluster is the full size, multi-node cluster that everybody expects to run for mission critical applications. This means that the typical use cases for ScaleDB ONE are testing and development, data marts and streaming data collection and analysis that are limited by a single server (although you can always replicate your data to another server using the standard MySQL Replication). ScaleDB Cluster instead, is highly available out of the box, with no single point of failure, and scaleable on demand (i.e. you do not need replication to set up availability and a more scalable environment).

From now on, in this post I will refer to ScaleDB as ScaleDB ONE.

Now, some prerequisites. ScaleDB has been tested on CentOS 6.7, CentOS 7.1 and on Ubuntu 14.04.3 Trusty Tahr. 1GB of memory and few GB of free disk space would be enough to test the product but, as for any other databases, the more cores, memory and storage you can add, the better. One interesting aspect is that, if you are planning to store a large amount of data, you will be very pleased with the performance you can get from ScaleDB using magnetic HDDs instead of SSD (but this is a topic for another post).

On CentOS 6.7, you must add nmap and nc, since you will need them later to interact with the ScaleDB daemons. A yum install nmap-ncat should do the trick. If you have done a minimal install of CentOS 7.1, I would also recommend to install nettools.

Another mandatory requirement is the installation of the AIO libraries (with yum install libaio). For Ubuntu 14.04.3, you will be required to install the AIO libraries, with sudo apt-get install libaio1.

The last two steps are not mandatory, but they will make your life easier: create a scaledb user that can be a sudoer and disable the firewall on your testing machine. From now on, I assume that you will log in as scaledb.

Downloading the software

If you have not downloaded ScaleDB ONE yet, you will have to hit few pages on the ScaleDB website, but the process is very straightforward. Just go to, click the Download button, scroll down to the bottom and click another Download button (this time it is grey). The next screen is used to select the type of download. You have two choices:

  • VirtualBox Image, which will allow you to download a OVA (a compressed VirtualBox image file), so everything is self contained in a CentOS 7 image and you do not need to install any software.
  • Tarball, which allows you to decompress and unarchive the ScaleDB product. We do not have Linux packages yet, they will be available soon.

Once you select your favourite download, all you have to do is to fill five fields, then you will receive am email with your personalised link to the downloads. You can keep this link, we will update your environment with new versions. Soon we will also add a yum and a apt repository.

You can use the unique URL to download ScaleDB ONE. You must add /scaledb-15.10/latest-release to the URL or you can simply browse the repository to search the release you need. In the latest-release folder, you will find 3 tarballs:

  1. The ScaleDB ONE UDE: UDE stands for Universal Data Engine. This is the ScaleDB engine. Right now (2015-11-17), the latest UDE tarball is scaledb-15.10.1-13199-ude.tgz.
  2. MariaDB: We use MariaDB as database server. ScaleDB can work on its own, i.e. you can access the engine by using the ScaleDB API, but for the MySQL and MariaDB users we have created a storage engine layer, so ScaleDB is fully accessible from MariaDB. The version available with 15.10 is MariaDB 10.0.14, soon we will release a version that works with the latest MariaDB 10.1. For Ubuntu you can use scaledb-15.10.1-mariadb-10.0.14-glibc2.14.tgz, for CentOS scaledb-15.10.1-mariadb-10.0.14.tgz.

You can donwload the tarballs on your own machine and then copy them to the testing machine or you can download them directly into the testing machine with a wget command.

Assuming that the tarballs are in the root directory of the scaledb user, you can now uncompress and copy the files on /usr/local with these commands:

sudo tar xzvf ~/scaledb-15.10.1-13199-ude.tgz -C /


sudo tar xzvf ~/scaledb-15.10.1-mariadb-10.0.14.tgz -C /

That’s it! ScaleDB is ready to be used.

In order to make things easier for anybody who wants to test ScaleDB ONE, we have assumed that the software will be installed in /usr/local and we have a predefined configuration. More specifically:

For MariaDB:

  • The base directory is /usr/local/mysql
  • The data directory is /usr/local/mysql/data
  • The configuration file is /usr/local/mysql/my.cnf
  • The admin user is root (no password)

For the ScaleDB engine:

  • The base directory is /usr/local/scaledb
  • The data directory is /usr/local/scaledb/data
  • There are three configuration files:
    • Storage Engine: /usr/local/mysql/scaledb.cnf
    • Storage Node: /usr/local/scaledb/cas.cnf
    • Lock Manager: /usr/local/scaledb/slm.cnf

At this point you can launch ScaleDB ONE with this script:

/usr/local/scaledb/scripts/scaledb_one start

If you add /usr/local/scaledb/scripts to your PATH you would see something like:

scaledb@ONE:~$ scaledb_one start
 Starting ScaleDB CAS server...
 ScaleDB CAS Server started.
 Starting ScaleDB SLM server...
 ScaleDB SLM server started.
 Starting MariaDB Server...
 MariaDB Server started.
 ScaleDB ONE started.

The script starts both MariaDB server and the ScaleDB engine. The ScaleDB engine is formed by two daemons, the CAS Server (Cache Accelerated Storage Server) and the SLM Server (ScaleDB Lock Manager Server). The same script must be used to stop the environment, by simply using scaledb_one stop.

Testing ScaleDB ONE

I will give you more information on how to properly test ScaleDB very soon. In the meantime, let’s just see if it works as expected.

The MariaDB interface has no difference, the only news is the ScaleDB storage engine:


Now it is time to test the engine. We can start by creating a table. The command to create a streaming table is a bit different from a standard InnoDB table. Here is an example:

MariaDB [(none)]> CREATE TABLE test.test (
 -> create_time timestamp NOT NULL,
 -> account char(8) NOT NULL,
 -> store int(10) UNSIGNED NOT NULL,
 -> amount decimal(8,2) NOT NULL,
 -> KEY create_time (create_time) RANGE_KEY=SYSTEM )
 Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]>

And here is the explanation line by line:

  • Streaming tables are special tables in ScaleDB that are used to store streaming data. They are really fast and work as a constant stream, i.e. you can INSERT the data, you can SELECT to run a query and you can DELETE the oldest data, but you cannot UPDATE any row or DELETE a row with a general condition.
  • The id column is the primary key: streaming tables always require a primary key.
  • The create_time is a timestamp associated to a range key, a special time-based key used in ScaleDB to query by a time interval.
  • The table attributes to add are the engine (ScaleDB) and the table type (STREAMING). At the moment we do not recommend to use any other type of table.

When you have received an OK, you have created your first ScaleDB table – congratulations!

And finally a simple INSERT and SELECT:

MariaDB [(none)]> INSERT INTO test.test VALUES (NULL, NULL, 'A', 1, 100);
Query OK, 1 row affected (1.54 sec)

MariaDB [(none)]> SELECT * FROM test.test;
 | id  | create_time         | account | store | amount |
 | 256 | 2015-11-16 06:30:24 | A       |     1 | 100.00 |
 1 row in set (0.00 sec)

MariaDB [(none)]>

One warning, if you run a SELECT query immediately after the INSERT and you do not see the that you have just inserted, it is a “normal behaviour”. By default, ScaleDB has a time window of 30 seconds that is used to flush a large number of rows in one go. This behaviour can be changed to a more usual OLTP behaviour, but the price is a lower load rate. In ScaleDB Cluster, the rows are safely stored on two servers and they are not lost in case of fault.

Welcome ScaleDB 15.10!

Time really flies. A bit less than 4 months ago, I wrote a post about my decision to join ScaleDB. Today, after 4 months and a lot of excitement working with a great team and genuinely good people, I am proud to announce that the first version of ScaleDB is available to the public.

ScaleDB 15.10 Ararat

We decided to number this version 15.10 and to name it Ararat. Indeed, we intend to follow the release cycle of other famous software projects, such as Ubuntu, OpenStack and recently CentOS. Our logo is a peak, we are all about scaling, as in our name and as the main objective of our products. To scale comes from the Latin word scandere, i.e. ‘to climb’. Mount Ararat is one of the most beautiful peaks in the whole world, yet hard to climb and full of significance and mystery for many. It looked natural for us to start our journey naming the product after this mountain.

ScaleDB 15.10 is the first public version of our product. So far, we’ve been using a private beta and we have been working with users, developers and DBAs to make the product available to the public for a more general use.

We have customers and community users who use ScaleDB in production- in the last year we have worked hard to fix all the S1 bugs known to us, but as with any software, we cannot guarantee that the quality of the product will be top notch right from its first public version, therefore we strongly recommend you thoroughly test ScaleDB 15.10 before deploying it in a production environment.

The software is available for download from our website as a tarball, and we are going to provide Red Hat and Ubuntu packages very soon. You can click here, fill in a quick form and receive information on how to download and use ScaleDB within minutes. We will setup an account for you, in which you will also find updates, patches and new releases.

Streaming data, time series and realtime analytics

The main objective of ScaleDB 15.10 is to provide a Big Data solution with MySQL, currently in the form of streaming data and realtime analytics (see some extra info here). From the perspective of a MySQL user, ScaleDB is a standard storage engine that can be plugged into MariaDB. Behind the scenes, we make extensive use of special handlers in MariaDB 10.1 that extend the condition pushdown to the storage engine (although in the very first version of 15.10 we still recommend MariaDB 10.0) and to a cluster of machines. We also call the ScaleDB Cluster IDC, Intelligent Data Cluster.

ScaleDB can handle millions of inserts per second loaded by parallel workers, whilst hundreds of concurrent users can analyse the very same data in realtime. We ran some basic tests and one of them is published here: it can give you an idea of the type of analysis and scalability you may expect from ScaleDB.

We use a different and innovative approach to storing, indexing and accessing the data. The best fit for ScaleDB is time series data, which probably represent a significant part of the data currently qualified as Big Data. That said, ScaleDB can also be used to store and analyse not only time series, but any kind of data, although we are not currently focused on rich data such as documents and multimedia.

ScaleDB ONE and ScaleDB Cluster

ScaleDB 15.10 comes in two flavours, ScaleDB ONE and ScaleDB Cluster.

ScaleDB ONE stands for One Node Edition. It is a single node version of the product. DBAs can install and use the product on a single node. Performance is great for many use cases, and ScaleDB ONE can already sustain hundreds of thousands of inserts per second and real time analysis on a single node. ScaleDB ONE is completely free, and support can be purchased on request.

ScaleDB Cluster is the fully loaded version of ScaleDB that can scale up to many Terabytes of data and hundreds of concurrent users. ScaleDB Cluster is available as a commercial license with technical support that can be purchased from ScaleDB Inc.

What’s next?

Well, this is just the start. We will talk more about ScaleDB in future posts, from its internal structures, to advanced indexing, scalability, roadmap and much more! As they often tell me as a frequent flyer: sit back, relax, and enjoy the journey with ScaleDB.