Anomaly Detection: Glimpse into the Future of IoT Data

Big data and the internet of things go hand in hand. With the continued proliferation of IoT devices — one prognosticator estimates there will be 64 billion IoT devices online by 2025 — enterprises are faced with extreme data volumes.

Managing these volumes, and yielding actionable insights from them, is key to transforming business for an era in which data is as vital as sunshine.

One area where the IoT data deluge can provide insights is through anomaly detection. Like traditional business applications and IT infrastructure, IoT can be monitored for issues in the physical devices to which they’re connected.

For example, when IoT devices are used to help modernize and automate processes in industrial or manufacturing settings, anomalies could be a sign that a particular piece of machinery needs maintenance. An early heads-up to a potential problem can help reduce unplanned downtime.

But corralling all that data can be a challenge as getting it moved from the device into a central compute platform can be problematic. Do you really need to send all the data from every device?

Edge computing can help overcome data overload and make IoT anomaly detection more effective. Exploring that practice in depth provides an illuminating look into emerging network architectures and how they will propel businesses in the future.

Common Detection Strategies

There are two prominent anomaly detection strategies integrated into IoT cases, each with advantages and drawbacks.

Static thresholds have the advantage of automating routine decision-making and alerting, giving it some advantage over traditional strategies. But to use static thresholds, administrators need domain knowledge thresholds at the right levels. And these levels don’t change according to conditions, making them difficult to maintain in ever-changing environments.

Methods based on machine learning (ML), either supervised or unsupervised, add intelligence to automated anomaly detection. By continuously learning time-series data behavior, the system becomes more adaptive to data changes and can better handle changing environments.

However, the volume of data generated by IoT devices will continue to increase over time, driving related increases in network bandwidth, storage volume and compute requirements. As a result, the use of centralized resources, even cloud, for real-time IoT data processing is becoming more and more expensive and lengthening the latency in data processing.

Edge computing provides a more effective way to leverage ML-based anomaly detection. By shifting critical data processing workloads closer to the data source (IoT devices), you can reduce workloads on the cloud, ensure zero-latency data processing, improve response time and decrease network load and cloud costs.

Edge computing can be also used to monitor machine health in real time to detect anomalies that might indicate a failure in a system. Zero-latency data processing makes it possible to report major incidents in near-real time and prevent system failures.

Edge Computing Architecture

Edge computing can be deployed with a number of architectures, including 100% edge computing and hybrid models that combine edge and cloud computing. For the purposes of illuminating edge architectures for anomaly deployment, the latter is shown here in Figure 1.

In this model, the IoT gateway collects and receives data from IoT devices using various IoT data protocols and pushes data to a messaging layer. Data processing components at the edge pull data from the messaging layer, process it and send insights to the cloud for centralized alerting and reporting. A monitoring and management system reports operational issues to the cloud and ensures that components are synchronized with the configuration data centrally managed from the cloud.

Edge computing components can be deployed together in a single-board computer like Raspberry Pi or across multiple devices in the same data center for greater scalability. The choice depends on scalability requirements.

IoT Anomaly Detection

Figure 2 shows an architecture for implementing edge computing for anomaly detection. This model adds intelligence to the generic architecture described previously.

In this model, an intelligent IoT gateway acts as a time-series data router. The messaging layer has separate queues for each anomaly detection method. Based on the anomaly detection methods being applied to specific time series, the gateway makes routing decisions and puts data into the respective queues. Dynamic routing decisions are driven by the configuration. Configuration changes are made at the cloud, and the monitoring and management system applies them.

Anomaly detection processors fetch data from the respective queues, detect anomalies and report them to the cloud. These processors also store data in a time-series database.

In this example, an offline training scheduled job periodically fetches bulk data from the time- series database, compresses data and sends it to the cloud to train ML models for supervised anomaly detection. The job also fetches ML models from the cloud and stores them in an ML models database. A supervised ML anomaly detector fetches ML models from the models database.

The monitoring and management system reports operational issues to the cloud and ensures that components are in sync with the configuration data centrally managed from the cloud.

Modernized IoT Implementation

For the foreseeable future, most enterprises will adopt a hybrid IT approach that includes both centralized and edge computing as part of an even more complex IT landscape.

The next step is to use edge computing to collect, apply analytics, and act. The days of centralized compute engines and lagging insights are numbered. Today, we are living in a world where we can collect, analyze and act on data where it’s generated in an instant.

This is where we can apply intelligent automation to find ways to orchestrate important edge data with central data pipelines for the required insights for automated action, instantaneously.

The data volumes and velocity of a modern IoT implementation call for scalable, efficient, real-time anomaly detection. Edge computing makes it possible to leverage machine learning for IoT anomaly detection while avoiding high cloud costs and processing latency. By using the architecture described above, you can detect and resolve IoT failures quickly to ensure optimal service for your organization.