This time it is real…

A few months ago I updated my profile on LinkedIN, and adjusted my position as CTO and founder of Athoa Ltd, a British company currently active for translation services and events that in the past hosted a couple of interesting open source projects. I simply forgot to disable the email notification to my connections, set by default, and in 2-3 hours I received tens of messages from friends and ex-colleagues who were curious to hear about my new adventure.

Today, I changed my profile on LinkedIN again and have left the email notification set on purpose.

As of today, I join the team at ScaleDB. My role is to define the product and the strategy for the company, working closely with CEO Tom Arthur, CTO Moshe Shadmon, CMO Mike Hogan and the rest of the team.

Leaving Canonical

The last nine months at Canonical have been an outstanding and crazily intense journey. I learned as I never learned before about systems and network infrastructures, and I met an amazing team of core engineers. It has been a unique experience, one of those that only come along once in a lifetime – I really mean it – and I will never forget it.

The decision to leave Canonical came after a lot of thinking and many sleepless nights. I met so many great people that in many ways, are making history in IT. In my team under Dan Poler, I worked with experienced Cloud and Solutions Architects that can analyze problems, discuss architectures and suggest solutions from the high level view, down to the kernel of the operating system and even to the silicon of systems and devices. Chris Kenyon and John Zannos teams are called “Sales”, but they are really advisors for a growing ecosystem of providers and adopters of Ubuntu and OpenStack technologies.

I have been inspired by the dedication and leadership of Canonical CEO Jane Silber. Jane has the difficult job of leading a company that is moving at lightspeed in many different directions, so that the technology that powers clouds, networks, end users and small devices can share the same kernel and will eventually converge. Jane is in my opinion the leading force, making Canonical blossom like a plum tree in mid-winter, when the rest of the of the nature still sleeps under the snow.

My greatest experience at Canonical has been working with Mark Shuttleworth. Mark is an inspiration not only for the people of Canonical or for the users of Ubuntu, but for us all. Mark’s energy and passion are second only to his great vision for the future. I recommend everybody to follow Mark’s blog and watch or attend his talks. His attention to detail and search for perfection never shadows the core message and understanding of the big picture; for this reason, both experienced listeners and newbies will have takeaways from his talks.

Back in June last year, I decided to join Canonical because of Mark’s vision. His ideas were in sync with what I wanted to bring at SkySQL/MariaDB. At Canonical, I could see this vision materialize in the direction the products were going, only on larger scale. This experience has reinforced in me the belief that we have an amazing opportunity right in front of us. The world is changing dramatically and at a speed that is incomparable with the past, even when compared with the first 10 years of the new millennium. We must think out of the box and reconsider the models that companies have used so far to sustain their business, since some of them are already anachronistic and create artificial barriers that will eventually collapse.

This experience at Canonical will stay with me forever and I hope to make a good use of what I have learned so far and all that I will learn in the future from Mark.

Joining ScaleDB

The last Percona Live was a great event. It was great to see so many friends and ex-colleagues again, now working on different companies but gathering together once a year as in a school reunion. Percona has now become a mature company, but more importantly, it has reached its maturity growing organically. The results are outstanding and the new course to be a global player in the world of databases looks even more promising.

The list of people and companies I would like to mention is simply too long and it would be a subject for a post per se. I found the MySQL world more active than ever. In this Percona Live I found the perfect balance between solid and mature technologies that are constantly improving, and new and disruptive technologies that are coming out under the same MySQL roof.

I simply feel as I am part of this world, and it is part of me. I worked with databases in many different roles for all my life, first with Digital/Oracle RDB and Digital/HP Datatrieve, then with IBM/Informix, Oracle, Sybase and SQLServer, and last with MySQL. I am looking at this world with the eyes of someone who has been enriched by new experiences. I simply think I have more to offer to this market than to networks and systems infrastructures. I therefore decided to come back. I also feel I can offer more in designing and defining products than in running services.

ScaleDB seems to me the company where I can express myself and I can help more at this point of my working life. With my previous role as advisor for the company, working on products and strategies just feels natural to me. The position is also compatible with my intention to improve and extend my involvement in the MySQL ecosystem, not only as MariaDB Ambassador, but also and equally advocating for Oracle and Percona products.

I also believe that MySQL should not be an isolated world from the rest of the database market. I already expressed my interest in Hadoop and other DB technologies in the past, and I believe that there should be more integration and sharing of information and experiences among these products.

I’ve known and have been working with Moshe Shadmon, ScaleDB CTO, for many years. Back in 2007, we spent time together discussing the use, advantages and disadvantages of distributed databases. At the time, we were talking about the differences between Oracle RAC, MySQL/NDB and DB/2, their strong and weak points, what needed to be improved. That was the time when ScaleDB as a technology started taking the shape that it has today.

ScaleDB is an amazing technology. It is currently usable as a storage engine with MariaDB 10.0, it has been developed with the idea of a cluster database from the ground up. As for MySQL in 2005, when the goal was to provide performance, scalability and ease of use in a single product, ScaleDB today provides more performance and greater scalability, without compromising availability and the use of standard SQL/MySQL. The engineering team at ScaleDB has recently worked on an amazing extension of their technology to sustain fast inserts and real-time queries on commodity hardware, at a fraction of the cost of NoSQL alternatives. This addition makes ScaleDB the perfect solution for storing and retrieving time series data, which is the essence for stream analytics and Internet of Things.

I believe ScaleDB has the incredible potential to become a significant player in the DB world, not only in MySQL. I feel excited and honored to be given the opportunity to work on this new adventure. I will try my hardest to serve the MySQL ecosystem in the best possible way, contributing to its success and improving the collaboration of companies – providers, customers, developers and end users – in MySQL and in the world of databases.

Now hop onto the new ride, the future is already here…

VirtualBox extensions for MAAS

During the last season’s holidays, I spent some time cleaning up demos and code that I use for my daily activities at Canonical. It’s nothing really sophisticated, but for me, and I suspect for some others too, a small set of scripts makes a big difference.

In my daily job, I like to show live demos and I need to install a large set of machines, scale workloads, monitor and administer servers and data centres. Many people I meet don’t want to know only the theoretical details, they want to see the software in action, but as you can imagine, the process of spinning up 8 or 10 machines and install and run a full version of OpenStack in 10-15 minutes, while you also explain how the tools work and perhaps you even try to give suggestions on how to implement a specific solution, is not something you can handle easily without help. Yet, that is what CTOs and Chief Architects want to know in order to decide whether a technology is good or not for them.

At Canonical, workloads are orchestrated, provisioned, monitored and administered using MAAS, Juju and Landscape, around Ubuntu Cloud, which is the Canonical OpenStack offering. These are the products that can do the magic of what I described, but providing in minutes something that usually takes days to install, set up and run.

In addition to this long preface, I am an enthusiastic Mac user. I do love and prefer Ubuntu software and I am not entirely happy with many technical decisions around OS X, but I also found Mac laptops to be a fantastic hardware that simply fits my needs. Unfortunately, the KVM porting to OS X is not available yet, hence the easiest and most stable way to spin up Linux VMs in OS X is to use VMWare Fusion, Parallels or VirtualBox. Coming from Sun/Oracle and willing to use open source software as much as I can, VirtualBox is my favourite and natural choice.

Now, if you mix all the technologies mentioned above, you end up with a specific need: the integration of VirtualBox hosts, specifically running on OS X (but not only), with Ubuntu Server running MAAS. The current version of MAAS (1.5 GA in the Ubuntu archives and 1.7 RC in the maintainers branch), supports virsh for power management (i.e. you can use MAAS to power up, power check and power down your physical and virtual machines), but the VirtualBox integration with virsh is limited to socket communication, i.e. you cannot connect to a remote VirtualBox host, or in other words MAAS and VirtualBox must run in the same OS environment.

Connections to local and remote VirtualBox hosts
Connections to local and remote VirtualBox hosts

 

My first instinct was to solve the core issue, i.e. add support to remote VirtualBox hosts, but I simply did not have enough bandwidth to embark on such an adventure, and becoming accustomed to the virsh libraries would have taken a significant amount of time. So I opted for a smaller, quicker and dirtier approach: to emulate the most simple power management features in MAAS using scripts that would interact with VirtualBox.

MAAS – Metal As A Service, the open source product available from Canonical to provision bare metal and VMs in data centres, relies on the use of templates for power management. The templates cover all the hardware certified by Canonical and the majority of the hardware and virtualised solutions available today, but unfortunately they do not specifically cover VirtualBox. For my workaround, I modified the most basic power template provided for the Wake-On-LAN option. The template simply manages the power up of a VM, and leaves the power check and power down to other software components.

The scripts I have prepared are available on my GitHub account, and are licensed under GPL v2, so you are absolutely free to download it, study it, use it and, even more important, provide suggestions and ideas to improve them.

The README file in GitHub is quite extensive, so I am not going to replicate here what has been written already, but I am going to give a wider architectural overview, so you may better consider whether it makes sense to use the patches or not.

MAAS, VirtualBox and OS X

The testing scenario that I have prepared and used includes OS X (I am still on Mavericks as some of the software I need does not work well on Yosemite), VirtualBox and MAAS. What I need for my tests and demos is shown in the picture below. I can use one or more machines connected together, so I can distribute workloads on multiple physical machines. The use of a single machine makes things simpler, but of course it puts a big limitation to the scalability of the tests and demos.

A simplified testbed with MAAS set as VM that can control other VMs, all within a single OS X VirtualBox Host machine
A simplified testbed with MAAS set as VM that can control other VMs, all within a single OS X VirtualBox Host machine

The basic testbed I need to use is formed by a set of VMs prepared to be controlled by MAAS. The VMs are visible in this screenshot of the VirtualBox console.

VirtualBox Console
VirtualBox Console

Two aspects are extremely important here. First, the VMs must be connected using a network that allows direct communication between the MAAS node and the VMs. This can be achieved locally by using a host-only adapter where MAAS provides DNS and DHCP services and each VM has the Allow All option set in the communication mode combo.

VirtualBox Network

Secondly, VMs must have PXE boot set on. In VirtualBox, this is achievable by selecting the network boot option as the first option available in the system tab.

VirtualBox Boot Options

 

In this way, the VMs can start the very first time and can PXE Boot using a cloud image provided by MAAS. Once MAAS has the VM enlisted as a node, administrators can edit the node via the WEB UI, the CLI app or the RESTful API. Apart from changing the name, what is really important is the setting of the Power mode and the physical zone. The power mode must be set as Wake-On-LAN and the MAC address is the last part of the VM id in VirtualBox (with colons). The Physical zone must be associated to the VirtualBox Host machine.

MAAS Edit NodeMAAS Edit Node

In the picture above the Physical zone is set as izMBP13. The description of the Physical zone must contain the UID and the hostname or IP address of the host machine.

Physical Zone

Once the node has been set properly, it can be commissioned by simply clicking the Commission node button in the Node page. If the VM starts and loads the cloud image, then MAAS has been set correctly.

The MAAS instance interacts with the VirtualBox host via SSH and with responds to PXE Boot requests from the VMs
The MAAS instance interacts with the VirtualBox host via SSH and with responds to PXE Boot requests from the VMs

A quick look at the workflow

The workflow used to connect MAAS and VM is relatively simple. It is based on the components listed below.

A. MAAS control

Although I have already prepared scripts to Power Check and Power Off the VM, at the moment MAAS can only control the Power On. Power On is executed by many actions, such as Commission node or the explicit Start node in MAAS. You can always check the result of this action by checking the event log in the Node page.

09 MAAS Node

B. Power template

The Power On action is handled through a template, which in the case of Wake-On-LAN and of the patched version for VirtualBox is a shell script.

The small fragment of code used by the template is listed here and it is part of the file /etc/maas/templates/power/ether_wake.template:

...
if [ "${power_change}" != 'on' ]
then
...
elif [ -x ${home_dir}/VBox_extensions/power_on ]
then
    ${home_dir}/VBox_extensions/power_on \
    $mac_address
...
fi
...

C. MAAS script

The script ${home_dir}/VBox_extensions/power_on is called by the template. This is the fragment of code used to modify the MAC address and to execute a script on the VirtualBox Host machine:

...
vbox_host_credentials=${zone_description//\"}

# Check if there is the @ sign, typical of ssh
# user@address
if [[ ${vbox_host_credentials} == *"@"* ]]
then
  # Create the command string
  command_to_execute="ssh \
    ${vbox_host_credentials} \
    '~/VBox_host_extensions/startvm \
     ${vm_to_start}'"
  # Execute the command string
  eval "${command_to_execute}"
...

D. VirtualBox host script

The script in ~/VBox_host_extensions/startvm is called by the MAAS script and executes the stratvm command locally:

...
start_this_vm=`vboxmanage list vms \
| grep "${1}" \
| sort \
| head -1`
start_this_vm=${start_this_vm#*\{}
start_this_vm=${start_this_vm%\}*}
VBoxManage startvm ${start_this_vm} \
           --type headless
...

The final result will be a set of VMs that are then ready to be used for example by Juju to deploy Ubuntu OpenStack, as you can see in the image below.

MAAS Nodes (Ready)

 

Next Steps

I am not sure when I will have time to review the scripts, but they certainly have a lot of space for improvement. First of all, by adopting a richer power management option, MAAS will not only power on the VMs, but also power off and check their status. Another improvement regards the physical zones: right now, the scripts loop through all the available VirtualBox hosts. Finally, it would be ideal to use the standard virsh library to interact with VirtualBox. I can’t promise when, but I am going to look into it at some point this new year.