Dear MySQLers, the Cloud Is Good, But the Fog Is the Next Big Thing

And here we go again: here is another term. We are still debating about the real meaning of Cloud and if a database fits in IaaS, PaaS, SaaS or in a combination of all three, and now we experience another wave of change.

To be fair, the term Fog Computing has been around for quite some time, but it has never enjoyed the popularity of its buddy “the Cloud”.

Is Fog Computing another pointless renaming of well known technologies and IT infrastructures? Or is it a real new thing that Systems and Database Administrators should look at, study and embrace? My intent here is not to support one side or the other, but simply to instil a few thoughts that may turn handy in understanding where the IT market is going, and more specifically where MySQL can be a good fit in some of the new trends in technology.

IoT: Internet of Things? No! IT + OT

When we talk about IoT in general, everyone agrees that it is already changing the world we are living in. Furthermore, analysts predict trillions of dollars in business for IoT, and clearly all the big high-tech companies want a large slice of the pie. Things become interesting though when we ask analysts, decision makers and engineers what is IoT, or even better, what is the implementation of IoT. The thought goes immediately to our wearables, smart phones, or devices at home: smart fridges and smart kettles are embarrassing examples of something that looks like the new seasonal fashion trend. These devices are certainly a significant part of IoT, they make ordinary people aware of IoT, but they are not what developers and administrators should [only] look at. The multi-trillion$ business predicted by analysts is a mix of smart devices that can connect together cities and rural areas, homes and large buildings, offices and manufacturing plants, mines, farms, trains, ships, cars… but also goods and even animals and human beings. All these connected elements have one thing in common: they generate a massive amount of data. This data must be collected, stored, validated, moved, analyzed… and this is not a trivial job.

Many refer to IoT as Internet of Things, but also at IIoT as Industrial Internet of Things, i.e. to this part of IoT that is related to an industrial process. In industrial processes, we add more complexity to the equation: the environment is sometimes inhospitable, intermittently accessible and unattended by operators and users (or there are literally no users). All this may also be true for non IIoT environments, the difference is that if your Fitbit runs out of power you may be disappointed, but if a sensor on an oil platform or an actuator on a train does not have power, that may be a bigger deal.

To me, IoT is clearly all of the above, with IIoT being a subset of IoT. Personally, I have a particularly different approach to IoT. In almost my entire working life I have been involved in the IT (i.e. Information Technology) side of the business, recently with databases, but previously designing and building CRM and ERP products and solutions. In my mind IoT means IT meets OT (i.e. Operational Technology) and the two technologies cannot be treated separately: they are tightly related and any product in IoT has an IT and an OT aspect to consider. It also means that OT is no longer relegated to the industrial and manufacturing world of PLC and SCADA systems, and is now widely adopted in any environment and at any level, even in what we wear or implant.

The convergence of IT and OT into IoT makes IT physical, something that has been missing in many IT solutions. We, as IT people dealing with data, tend to manage data in an abstract manner. When we consider something physical, we refer to the performance we can squeeze from the hardware where our databases reside. With OT, we need to think broadly ofthe physical world where the data come from or go to and, even more importantly, of the journey of that data in every bit of the IT and OT infrastructure.

It’s a Database! No, It’s a Router! No, It’s Both!

The journey! That is the key point. MySQLers think about data as data stored into a database. When we think about the movement of data, we refer to it in terms of data extraction or data loading. The fact is, in IoT, data has value and it must be treated and considered when stored somewhere (data at rest) and when moving from one place to another (data in motion). Moving data in IoT terms means data streaming. There are a plethora of solutions for streaming, like Kafka, RabbitMQ and many other *MQ products, but their main focus is to store and forward data, not to use it while it is in motion. The problem is, infrastructures are so complicated, with multiple layers and with too many cases where data stops while in motion, that it becomes a priority to analyse and “use” the data even when it is transiting from one component to another.

This is a call to build the next generation database, optimised for IoT, with features that go beyond the ability to store and analyse data. Data streaming and analysis of streamed data must be part of a modern database, as also highlighted by a recent Gartner report. If you are a Database Administrator, you may consider it a database with all the features of a traditional database, but with routing and streaming capabilities. If you are a Network and Systems Administrator you may consider it a router or a streaming system with database capabilities. In a way or another, the database needed for IoT must incorporate the features of a traditional database and the ones of a traditional router. Furthermore, it must take into consideration all the security aspects of data moved and stored multiple times and, even more importantly, it must provide a safe data attestation (but let’s reserve this aspect for another post).

Welcome to Fog Computing

So, here it is: Fog Computing is all of the above. Take three layers:

  • The Edge: where things, animals and human beings live where data is collected, or where results from analysis go.
  • The Cloud: where a massive amount of data is stored and analysed, where systems are reliable and scalable, and are attended by operators and administrators.
  • The Fog: is everything in between. It is, in oriental terms, “where the ground meets the sky”. The Fog is the layer that is still closed to the Edge, but must provide features that are typically associated to the Cloud. It is also the layer where data is collected from a vast amount of things, and is consolidated and sent to the Cloud whenever it is possible.

The term Fog Computing is so vague that for some analysts it refers to everything from the Edge of sensors and devices to regional concentrators, routers and gateways. For other analysts, Fog Computing refers only to the layer above the Edge, ie. to the gateways and routers. Personally, I like to think that the former, i.e. Edge + middle layer, offers a more practical definition of Fog Computing.

In Fog Computing, we bring the capabilities of Cloud computing into a more complex, constrained and often technically inhospitable environment. We must collect and store a large amount of data on constrained devices the size of a wristwatch, where the processing power is mostly used to operate the system and the data management is a secondary aspect. Although the power of an Edge system is increasing exponentially, we no longer have the luxury of a stable, always-on environment. It is a bit like going back 20 years or more, when we started using personal computers to manage data. It is a fascinating challenge, certainly unwelcome by lazy administrators, which brings excitement to experienced developers.

Where Is MySQL in All This?

Here is the catch: Fog Computing desperately needs databases. Products that can handle data at rest and in motion, on constrained devices, with a small footprint, databases that can maximise the use of hardware resources, are reliable and can be installed in many flavours to be almost 100% available when needed. Many NoSQL solutions are good in theory (because of the the way they manage unstructured data), but they are often too resource-hungry to compete in this environment, or they lack features that MySQL has implemented more than a decade ago. Embedded databases are on the other side of the offer, but their features are often limited, making the solutions pretty incomplete.

Sounds familiar? Edge and Fog Computing are the perfect place for MySQL, or at least for solutions based on MySQL, where more features must be added. At the moment there are no real database and data management products for Fog Computing. The current solutions are mostly based on MySQL, but they are built ad hoc and their implementation is non replicable: a situation that slows the growth of this market, making the overall cost of a solution higher than it should be.

The opportunity is huge, but also challenging. The first implementation does not have to be a new fresh product, it can be something achievable, step by step. As for more examples and real, live projects, watch this space!


This time it is real…

A few months ago I updated my profile on LinkedIN, and adjusted my position as CTO and founder of Athoa Ltd, a British company currently active for translation services and events that in the past hosted a couple of interesting open source projects. I simply forgot to disable the email notification to my connections, set by default, and in 2-3 hours I received tens of messages from friends and ex-colleagues who were curious to hear about my new adventure.

Today, I changed my profile on LinkedIN again and have left the email notification set on purpose.

As of today, I join the team at ScaleDB. My role is to define the product and the strategy for the company, working closely with CEO Tom Arthur, CTO Moshe Shadmon, CMO Mike Hogan and the rest of the team.

Leaving Canonical

The last nine months at Canonical have been an outstanding and crazily intense journey. I learned as I never learned before about systems and network infrastructures, and I met an amazing team of core engineers. It has been a unique experience, one of those that only come along once in a lifetime – I really mean it – and I will never forget it.

The decision to leave Canonical came after a lot of thinking and many sleepless nights. I met so many great people that in many ways, are making history in IT. In my team under Dan Poler, I worked with experienced Cloud and Solutions Architects that can analyze problems, discuss architectures and suggest solutions from the high level view, down to the kernel of the operating system and even to the silicon of systems and devices. Chris Kenyon and John Zannos teams are called “Sales”, but they are really advisors for a growing ecosystem of providers and adopters of Ubuntu and OpenStack technologies.

I have been inspired by the dedication and leadership of Canonical CEO Jane Silber. Jane has the difficult job of leading a company that is moving at lightspeed in many different directions, so that the technology that powers clouds, networks, end users and small devices can share the same kernel and will eventually converge. Jane is in my opinion the leading force, making Canonical blossom like a plum tree in mid-winter, when the rest of the of the nature still sleeps under the snow.

My greatest experience at Canonical has been working with Mark Shuttleworth. Mark is an inspiration not only for the people of Canonical or for the users of Ubuntu, but for us all. Mark’s energy and passion are second only to his great vision for the future. I recommend everybody to follow Mark’s blog and watch or attend his talks. His attention to detail and search for perfection never shadows the core message and understanding of the big picture; for this reason, both experienced listeners and newbies will have takeaways from his talks.

Back in June last year, I decided to join Canonical because of Mark’s vision. His ideas were in sync with what I wanted to bring at SkySQL/MariaDB. At Canonical, I could see this vision materialize in the direction the products were going, only on larger scale. This experience has reinforced in me the belief that we have an amazing opportunity right in front of us. The world is changing dramatically and at a speed that is incomparable with the past, even when compared with the first 10 years of the new millennium. We must think out of the box and reconsider the models that companies have used so far to sustain their business, since some of them are already anachronistic and create artificial barriers that will eventually collapse.

This experience at Canonical will stay with me forever and I hope to make a good use of what I have learned so far and all that I will learn in the future from Mark.

Joining ScaleDB

The last Percona Live was a great event. It was great to see so many friends and ex-colleagues again, now working on different companies but gathering together once a year as in a school reunion. Percona has now become a mature company, but more importantly, it has reached its maturity growing organically. The results are outstanding and the new course to be a global player in the world of databases looks even more promising.

The list of people and companies I would like to mention is simply too long and it would be a subject for a post per se. I found the MySQL world more active than ever. In this Percona Live I found the perfect balance between solid and mature technologies that are constantly improving, and new and disruptive technologies that are coming out under the same MySQL roof.

I simply feel as I am part of this world, and it is part of me. I worked with databases in many different roles for all my life, first with Digital/Oracle RDB and Digital/HP Datatrieve, then with IBM/Informix, Oracle, Sybase and SQLServer, and last with MySQL. I am looking at this world with the eyes of someone who has been enriched by new experiences. I simply think I have more to offer to this market than to networks and systems infrastructures. I therefore decided to come back. I also feel I can offer more in designing and defining products than in running services.

ScaleDB seems to me the company where I can express myself and I can help more at this point of my working life. With my previous role as advisor for the company, working on products and strategies just feels natural to me. The position is also compatible with my intention to improve and extend my involvement in the MySQL ecosystem, not only as MariaDB Ambassador, but also and equally advocating for Oracle and Percona products.

I also believe that MySQL should not be an isolated world from the rest of the database market. I already expressed my interest in Hadoop and other DB technologies in the past, and I believe that there should be more integration and sharing of information and experiences among these products.

I’ve known and have been working with Moshe Shadmon, ScaleDB CTO, for many years. Back in 2007, we spent time together discussing the use, advantages and disadvantages of distributed databases. At the time, we were talking about the differences between Oracle RAC, MySQL/NDB and DB/2, their strong and weak points, what needed to be improved. That was the time when ScaleDB as a technology started taking the shape that it has today.

ScaleDB is an amazing technology. It is currently usable as a storage engine with MariaDB 10.0, it has been developed with the idea of a cluster database from the ground up. As for MySQL in 2005, when the goal was to provide performance, scalability and ease of use in a single product, ScaleDB today provides more performance and greater scalability, without compromising availability and the use of standard SQL/MySQL. The engineering team at ScaleDB has recently worked on an amazing extension of their technology to sustain fast inserts and real-time queries on commodity hardware, at a fraction of the cost of NoSQL alternatives. This addition makes ScaleDB the perfect solution for storing and retrieving time series data, which is the essence for stream analytics and Internet of Things.

I believe ScaleDB has the incredible potential to become a significant player in the DB world, not only in MySQL. I feel excited and honored to be given the opportunity to work on this new adventure. I will try my hardest to serve the MySQL ecosystem in the best possible way, contributing to its success and improving the collaboration of companies – providers, customers, developers and end users – in MySQL and in the world of databases.

Now hop onto the new ride, the future is already here…

Why we should care about wearable Linux

These thoughts have been inspired by the not-so-recent-anymore announcement of Apple Watch and by Jono Bacon’s post re Ubuntu for Smartwatches. They do not specifically refer to Ubuntu, and they reflect my personal opinion only.

Wearable devices are essentially another aspect of the Internet of Things. IoT is now expanded to include your clothes and tools, not to mention your body, or parts of it. In the next decade, people will experience the next generation of mobile applications, the ones that are now used to share your recent workout or your last marathon. Rest assured: there will be all sorts of new applications, and existing applications will be highly improved by the adoption of affordable wearable hardware.

Right now, the two main choices for wearable applications are IOS and Android platforms, understandably evolved to fit with the new hardware. A third and fourth option are based on Microsoft and Linux platforms, but at the moment there is very little to say about their ability to gather a good slice of the market.

Fact is, a market dominated by two companies who are already controlling 99% of the mobile applications, is not a good thing. Indeed, it is in their rights to take advantage from the huge investments they have made to build IOS and Android, but that does not mean that the world should not have other alternatives. Although it is not a walk in the park, Linux is the only platform that can rise as truly independent and have a chance to compete in this war.

Why wearable Linux is important to consumers

The majority of the applications we use are free. The cost of the platform, whether it is IOS or Android, is paid in other ways. One of them is the control over the applications available to the consumers. Apple and Google can decide what goes in and what stays out of your mobile devices, and the same will happen for your wearables. Any attempt to escape from this can be easily prevented by the owners of the technology. So, in simple terms, consumers have less choices and eventually less freedom.

Another aspect which is pretty concerning nowadays is data collection. This is and has always been the most controversial aspect of mobile applications. Terms and conditions may allow providers to take advantage from the the data collected through your smartphone or from your wearable in the way they want. It could even be “their data”, and for the consumers it would be just another way to pay for the “free application”. In the best case scenario, the data will not be used individually, i.e. it will not be connected to your name, but it will be aggregated together with many other users in order to analyse behaviours, define strategies, sell trends and results. In one way or another, rest assured that the data collected is a highly valuable asset for the technology providers.

Now, take these two aspects and consider them in connection to the availability and adoption of wearable Linux. A truly open platform, perhaps controlled in terms of technical choices but not in the way it will be used, will not prevent in any way the control of an app store or the data collection, but it will give you choice. Companies may be application providers or may even create their application store, more or less open than Google and Apple. Eventually, people will choose, and they will have more options. The choice is likely to be based on the applications available, but again it is up to the manufacturers and developers to offer an appealing product. This is not dissimilar from the choice of a Cloud infrastructure – for example comparing what you can do under the full control of a single provider in AWS or what goes with the adoption of OpenStack. To some extent, wearable Linux is just an important piece of a jigsaw that goes into the definition of an open infrastructure for IOT.

Why wearable Linux is important for developers and manufacturers

Take a look at the most prominent open source projects available today: the majority of them is backed by foundations and by companies working together, and then competing to provide the best solution on the market. This is not the case for Android, which is fully controlled by Google, not to mention IOS. Sure, Google is keeping Android open and is making it available to any developer and manufacturer. In a similar way, Apple takes pretty good care of its developers, but both companies  simply have their own agenda. A wearable stack which is part of a full platform developed within a foundation would have its internal huge politics, but also more contribution and independence. Put simply, there will be more players and more opportunities to do business in the biggest revolution in computers after the semiconductors.

From a more technical point of view, developers would immensely benefit from the use of the full Linux stack. If you look at the way Android and IOS have evolved, you will realise that the concept of lightweight has significantly changed since their early versions. On the other hand, some of the libraries available on Linux are not in Android, and this is a significant limitation for developers. Having a common platform where portability from the mainframe to a wearable, is a big, huge asset, and it is not inconceivable in modern scale-out infrastructures. Interoperability, robustness, rapid development and ease of deployment are not trivial aspects of a Linux wearable stack, something that would increase the amount of applications and competition in the market.

The essential takeaway is that wearable applications are a great opportunity to expand the current market of mobile infrastructures in order to introduce more players and competition, and to be ready for the next revolution of the IOT. Whether or not new and old companies will take this opportunity is still to be seen, but for us consumers it would only be positive in the long term.