INTRODUCTION

Dator Modular Data Analysis Platform
Figure 1. Web UI Preview.

BACKGROUND

I have been exposed to a wide variety of sensors since I entered college in 2016. We do a lot of interesting things with these sensors, such as measuring the distance from obstacles with ultrasonic sensors, which enables cars to avoid obstacles automatically, and tracking people through infrared sensors to open doors or air conditioners. However, we only did a simple logical analysis of the data on the terminal, like operating in a blind box, and there was no place to host them, visualize them, and further mining them. As the amount of sensor data increases dramatically, it is necessary to store, visualize, and then mine information from large amounts of data.

MOTIVATION

In fact, there are many excellent data visualization and data processing platforms. For example, PowerBI and Tableau. However, it is not very user-friendly for custom function modules. Because of the variety of data from different sensors in different scenarios, we need to write special data processing methods for this and present the data the way we want it. We plan to create many microservices. (e.g. convolution, noise reduction, pooling, etc.) Data processing and visualization modules for developers to implement them.

METHOD

Figure 2. The system architecture consists of data sources, backends, and frontends.

Front End

At the front end, we are divided into four modules: user management module, device management module, data management module, and customized service module. The user management module implements functions such as user registration, deregistration, group, and authorization. Through the device management module, we can register data devices. After the registration is complete, the corresponding ID and key are allocated. After the registration is successful, different types of devices can use some basic data analysis microservices provided by we. It is worth mentioning that the devices can be hardware devices (such as cameras and sensors). It can also be a virtual device (for example, an HTTP request). The data management module can process data under the device, including data calculation and processing. In the service management module, users can write customized services based on their own data, which is our focus in the future. By the way, our front-end technology stack is: Vite4 + Vue3 + Bootstrap5 + SCSS.

Back End

Our backend project uses the classic MVC framework, including database services, data access services, data processing services, and authentication services.

- Database Services

Database Services corresponds to M in MVC. This service is used to perform operations on the database, such as adding, deleting, modifying, and querying data. Databases such as Postgrel SQL, MongoDB, and Redis are deployed. Postgrel SQL stores user, device, and service information, and MongoDB stores access data. Redis is used for data caching to improve data processing speed and protect databases.

- Data Access Services

Data Access Services corresponds to C in MVC and is used to receive data from devices (including hardware devices and virtual devices). We provide four data access protocols: MQTT, HTTP, and media stream. HTTP is the most widely used. Media streams are mainly used for video data access.

- Data Process Services

Data processing services correspond to V and C in MVC. After a user invokes the data processing microservice on the frontend, the frontend processes the data in the database and returns the processing result to the frontend. The frontend renders the result through the following plug-ins:

- Authentication Services

The authentication service includes data access authentication and user operation authentication. After a user logs in, the user obtains a JWT token. Each page and service accessed later must carry the token. The token contains the user information and expiration time, which is generally 15 minutes. After data registration, a device ID and key are returned. Each time a device accesses the data, the device needs to carry the ID and key for authentication. Otherwise, the access is rejected.

Data Source

Rich data is the basis for effective data analysis. Our platform supports the access of various data, such as various sensor data, video streams, audio streams, or file imports. Before data access, you need to register on the platform, and then obtain a device ID and key. Every time you send data, you need the ID and key corresponding to the carrier. The data is only allowed to be accessed after the platform verifies it. These data are stored in the MongoDB database.

KEYWORDS

Microservice, Modularization, Sensor, Visualization, Data Processing, Machine Learning, and Video Streaming.

EXPERIMENT

WEB UI

We have developed our UI pages based on an open source UI template. In addition to aesthetics, we are looking for system stability, security and responsiveness. We have completed the development of some basic pages to implement some basic services, such as user registration, login, device registration, deletion, modification and data display.

Figure 3. Login Page.
Figure 4. Home Page.
Figure 5. Add a new device.
Figure 6. Devices List.
Figure 7. Device Detail.
Figure 8. Data Analysis.