Part 1: Image Classification with MariaDB Server and TensorFlow – an Overview

Cutting-edge companies are turning to artificial intelligence and machine learning to meet the challenges of the new digital business transformation era.

According to Gartner: “Eighty-seven percent of senior business leaders say digitalization is a company priority and 79% of corporate strategists say it is reinventing their business—creating new revenue streams in new ways“.

Source: Gartner, Speed Up Your Digital Business Transformation, January 15, 2019

But so far, digital change has been challenging. The complexity of the tools, architecture, and environment create barriers to using machine learning. Using SQL-based relational data management to store and perform data exploration of images reduces the barriers and unlocks the benefits of machine learning.

This blog post demonstrates using popular open source tools MariaDB Server, TensorFlow Python library, and Keras neural-network library to simplify the complexity of implementing machine learning. Using these technologies can help you accelerate your time-to-market by efficiently accessing, updating, inserting, manipulating and modifying data.

Machine Learning on Relational Databases

At the center of the digital business transformation enabled by machine learning are technologies such as chatbots, recommendation engines, personalized communications, intelligent advertisement targeting, and image classification.

Image classification has a wide variety of use cases—from law enforcement and the military, to retail and self driving cars. When implemented with machine learning, image classification can provide real-time business intelligence. The objective of image classification is to identify and portray, as a unique gray level (or color), the features occurring in an image. The most common tools for image classification are TensorFlow and Keras.

TensorFlow is a Python library for fast numerical computing created and released by Google. MariaDB Server is an open source relational database with a SQL interface for accessing and managing data. Keras is an open-source neural-network library written in Python.

In this post, you will discover how to test image classification by enabling interoperability between TensorFlow and MariaDB Server. This post uses the Fashion MNIST dataset which contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels).

Loading and preparing the data into MariaDB Server is outside the scope of this post. The following tables have been created and populated in advance with the Fashion MNIST dataset.

 

Images Image Role Image Label
CREATE TABLE tf_images (
 img_label tinyint(4),
 img_vector blob,
 img_idx    int(10) unsigned NOT NULL,
 img_blob   blob,
 img_use    tinyint(4)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
CREATE TABLE img_use (
 use_id tinyint(4) NOT NULL AUTO_INCREMENT,
 use_name varchar(10) NOT NULL,
 use_desc varchar(100) NOT NULL,
 PRIMARY KEY (use_id)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4
CREATE TABLE categories (
 class_idx tinyint(4) NOT NULL,
 class_name char(20) DEFAULT NULL,
 PRIMARY KEY (class_idx)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

Data Exploration

The following libraries are used to perform basic data exploration with MariaDB Server:

  1. The io module provides Python’s main facilities for dealing with various types of I/O.
  2. Matplotlib is a Python 2D plotting library to produce a variety of graphs across platforms.
  3. Pandas offers data structures and operations for manipulating numerical tables and time series.
  4. The pymysql package contains a pure-Python client library to access MariaDB Server.

Let’s start by connecting to the database server through Python:

import io
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import pandas as pd
import pymysql as MariaDB
%matplotlib inline

conn = MariaDB.connect( host = '127.0.0.1'
, port = 3306
, user = 'mdb'
, passwd = 'letmein'
, db = 'ml'
, charset = 'utf8')

cur = conn.cursor()

Once connected to the MariaDB server, the images in the database can be easily accessed and managed. All the images used for training and testing the model are stored in a single table (tf_images). How the image will be used is defined in the image use table (img_use). In this case, the table has only two tuples, training and testing:

sql="SELECT use_name AS 'Image Role'
          , use_desc AS 'Description'
     FROM img_use"
display( pd.read_sql(sql,conn) )

 

Image Role Description
Training The image is used for training the model
Testing The image is used for testing the model

Mapping target attributes to image objects in a dataset is called labeling. The label definition varies from application to application, and there is hardly a universal definition of what a “correct” label is for an image. Using a relational database simplifies the labeling process and provides a way of going from coarse to fine grain labels.

In this example, using the “categories” table, an image has only one label (coarse) as shown below:

sql="SELECT class_name AS 'Class Name' FROM categories"
display( pd.read_sql(sql,conn) )
Class Name
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot

The images table contains all the images to be used for training and testing. Each image has a unique identifier, a label, and whether it is used for training or testing the model. The images are stored in their original PNG format and as pre-processed floating point tensors. A simple inner join on this table can be executed to display the image representations (vector or png format), its label, and the intended usage.

sql="SELECT cn.class_name AS 'Class Name' 
          , iu.use_name AS 'Image Use' 
          , img_vector AS 'Vector Representation' 
          , img_blob AS 'Image PNG' 
     FROM tf_images AS ti 
          INNER JOIN categories AS cn ON ti.img_label = cn.class_idx
          INNER JOIN img_use AS iu ON ti.img_use  = iu.use_id
     LIMIT 5"
display( pd.read_sql(sql,conn) )

 

Class Name Image Use Vector Representation Image PNG
Ankle boot Training b’\x80\x02cnumpy.core.multiarray\n_reconstruct… b’\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\…
T-shirt/top Training b’\x80\x02cnumpy.core.multiarray\n_reconstruct… b”\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\…
T-shirt/top Training b’\x80\x02cnumpy.core.multiarray\n_reconstruct… b’\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\…
Dress Training b”\x80\x02cnumpy.core.multiarray\n_reconstruct… b’\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\…
T-shirt/top Training b’\x80\x02cnumpy.core.multiarray\n_reconstruct… b’\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\…

Using SQL statements makes the data exploration process easy. For example, the SQL statement below shows the image distribution by image label.

sql="SELECT class_name AS 'Image Label' \
     , COUNT(CASE WHEN img_use = 1 THEN img_label END) AS 'Training Images'\
     , COUNT(CASE WHEN img_use = 2 THEN img_label END) AS 'Testing Images'\
     FROM tf_images INNER JOIN categories ON class_idx = img_label \
     GROUP BY class_name"
df = pd.read_sql(sql,conn)
display (df)
ax = df.plot.bar(rot=0)

 

Image Label Training Images Testing Images
0 Ankle boot 6000 1000
1 Bag 6000 1000
2 Coat 6000 1000
3 Dress 6000 1000
4 Pullover 6000 1000
5 Sandal 6000 1000
6 Shirt 6000 1000
7 Sneaker 6000 1000
8 T-shirt/top 6000 1000
9 Trouser 6000 1000

There are 6,000 images for each label in the training set and 1,000 images for each label in the testing set. There are 60,000 total images in the training set and 10,000 total images in the testing set.

Individual articles of clothing are stored as low resolution images. Since the database can store those images efficiently as Binary Large OBjects (BLOBs) it is very easy to retrieve an image using SQL, as shown below:

sql="SELECT img_blob \
     FROM tf_images INNER JOIN img_use ON use_id = img_use \
     WHERE use_name = 'Testing' and img_idx = 0"

cur.execute(sql)
data = cur.fetchone()
file_like=io.BytesIO(data[0])
img = mpimg.imread(file_like)
plt.imshow(img)

above: image from fashion_mnist dataset

This first part of the blog series has demonstrated how a relational database can be used to store and perform data exploration of images using simple SQL statements. Part 2 will show how to format the data into the data structures needed by TensorFlow, and then how to train the model, perform predictions (i.e., identify images) and store those predictions back into the database for further analysis or usage.

MIT License

The Fashion MNIST (fashion_mnist) dataset leveraged by this blog is licensed under the MIT License.

The MIT License (MIT) Copyright © 2017 Zalando SE, https://tech.zalando.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

References

Convert own image to MNIST’s image
matplotlib: Image tutorial
5 ways AI is transforming customer experience
Digitalization is reinventing business
What is image classification?
Introduction to the Python Deep Learning Library TensorFlow