When good IoT devices go bad
Smart devices and sensors are only as good as the data they collect, and the deck is stacked against them. Devices often exist in rugged environments. They wear out, or fail, need proper configuration to function properly, and are often targeted by hackers.
What happens when sensor data is wrong, or isn’t what was expected?
Consider the following:
- A study found that a glut of electronic noise from wireless networks and Internet of Things (IoT) devices “may lead to poor network performance or even application failures.” The study found examples of this at an advanced experimental digital factory and a large surface-mining site.
- Many industrial and enterprise IoT devices have sizable security holes that can be exploited by hackers, including the remote power managers used by thousands of companies. IoT medical devices are even more at risk, and there are reports of many being hacked.
- An explosion at an oil refinery in Texas City, Texas, which killed 15 people and injured 180, is believed to have been caused partly by devices that provided false readings because they were calibrated incorrectly.
- Ice crystals threw off airspeed sensors and disconnected the autopilot of an Air France flight in 2009. This was followed by a combination of events that led to its crash.
Below we examine the challenges involved in collecting accurate data from IoT devices. We asked a number of technology experts if inaccurate IoT data is an issue, and if so, how bad is it, what they see as the primary causes, and how they deal with the problem. We also offer advice from experts on how to improve the chances of gathering quality data.
Scope of the problem
Environmental factors and harsh surroundings can skew the data gathered by sensors. “From my experience, the data in any large and complicated IoT system can’t be completely trusted,” says Eddie Gotherman, sales engineer for IoT integrator Bright Wolf. “You’ve got hundreds of thousands of devices and sensors, and there’s going to be at least some bad data and noise in there.”
The sheer density of devices and sensors, combined with the proximity of wireless networks, can create so much interference and noise that it becomes difficult or impossible to accurately record data, according to "Troubleshooting Wireless Coexistence Problems in the Industrial Internet of Things," a paper presented at the 2016 IEEE/IFIP International Conference on Embedded and Ubiquitous Computing in Paris. The authors determined that problems may not surface for as long as two years.
Sensors and IoT devices also degrade over time. This is another major cause of questionable IoT data, according to Peter Marx, vice president of Advanced Concepts for Predix, a GE Digital platform for the Industrial Internet.
Sensors and devices are not always to blame. External factors can cause good sensors to record and transmit data that may seem bad, even though it’s perfectly good, says John Fruehe, senior analyst for networking and servers with consulting firm Moor Insights & Strategy.
“Let’s take a simple example: a business that uses IoT sensors to control their air conditioning system,” he says. “The sensors measure the temperature of the air, and then adjust the air conditioning accordingly. If someone moves a bookcase and covers up a sensor, that will cause the conference room to be too cold because the sensor will be warmer than the surrounding room and turn up the air conditioner. If you just looked at the incoming data, you would think the sensor had gone bad because its data didn’t match the overall temperature of the room. But if you examined the environment, you’d see the sensor and its data was good—its surrounding environment was causing the problem.”
Human error is the most frequent cause of bad IoT data, according to Dana Tamir, vice president of market strategy at Indegy, a cybersecurity firm specializing in protecting industrial control systems.
“We come across many incidents where people make changes to devices and do it incorrectly, which leads to bad data,” says Tamir. “Often, these devices are old, with no security built into them, and there’s no way to restrict people from making the changes and no way to trace exactly what they did. That makes it extremely difficult to know when data is going bad and what [caused the problem].”
Enterprises shouldn't take a one-size-fits-all approach when troubleshooting potential issues with IoT data. They should differentiate between data problems caused by device or sensor failure versus unusable data versus corrupted data versus malicious data. Only after they clearly identify the cause of the problems in this way can they solve them.
When hackers attack
The issue of IoT devices and data being hacked “comes up in virtually every industrial IoT meeting I have,” says Michael Tennefoss, vice president of strategic partnerships at Aruba, a Hewlett Packard Enterprise company. “People want to know, ‘What happens if my sensor is compromised as opposed to being broken?’”
Hackers frequently exploit built-in vulnerabilities in IoT devices and sensors. The engineers who design this equipment “are not the center of expertise for cybersecurity, and that’s why there are so many IoT security breaches," Tennefoss says. He adds that fundamental security features are missing and many older systems that run production lines have no security features whatsoever.
In many cases, companies never know their IoT devices have been compromised because they operate normally during working hours. Hackers are smart enough to infiltrate the network and steal production data during off hours.
Consequences of bad data
Industrial breaches can have serious and even life-threatening consequences. “When data or devices go bad in an industrial system, the damage is a lot greater than when an information system has been breached,” says Tamir. “With industrial systems, we’re not talking about data, numbers, and money, but physical systems that can cause anything from minor disruptions to major catastrophes. So a breach or an unauthorized change to a control system can result in the spill of hazardous materials or an explosion. People can be killed.”
Risk and reward
What can companies do to mitigate the risks posed by bad IoT data? First, they need to know that the bad data exists, and that is not always easy, says David Barnett, vice president of products and markets at RTI, an IoT connectivity company.
He offers the example of an accelerometer on a wind turbine measuring vibrations. If the accelerometer reports a sudden change in vibrations, that could signal one of two things: The accelerometer is failing, or the turbine is broken or about to fail. How can engineers determine which theory is correct?
It helps to correlate data from individual sensors or groups of sensors with as many other sensors and data source as possible. In the wind turbine example, if you correlate the data from the single accelerometer with other accelerometers on the turbine, and with other information such as the output of the turbine, you’ll be able to quickly determine whether the sensor data is bad or whether it’s an early sign the turbine is about to fail.
If there’s no change in power output and the accelerometer reports different information than the other sensors, the accelerometer and its data are bad. But if the accelerometer data is in line with others and there’s a drop in power output, the problem is the turbine, not the sensor.
Marx adds that in order to determine whether data is good or bad, companies should create baselines for "normal" IoT devices and data. That's more difficult than it sounds because standards of normality often vary with environmental changes and the time of year.
“Normal is a fuzzy term,” he says. “When jet engines are used in the winter, they act differently than in the summer.”
Companies should first establish a general baseline and then build normal variations into their models for IoT data.
Fruehe of Moor Insights & Strategy adds companies should correlate their sensor and IoT data with the widest range of information possible. Manufacturers, for example, should monitor their power supply closely, especially if they’re using robots. Fluctuations in power conditions can cause a millisecond or more delay in robots doing their jobs. That may not sound like much, but it’s enough to cause imperfect welds in automobiles.
Similarly, in a chip fabrication plant or factory that builds medical devices, air filters and air quality must be measured closely, because a single speck of dust can cause manufacturing malfunctions.
Another consideration is that IoT data is rarely clean and comes in many different formats, says Gotherman. That means data needs to be scrubbed and put into common formats that can be easily used. It's important to understand the provenance of all IoT data—where it comes from, what was done to clean it, and who did the cleaning. That's the only way to determine if data is good or bad.
Last, Tennefoss recommends that when enterprises troubleshoot potential issues with IoT data, they don’t take a one-size-fits-all approach. Organizations should differentiate between data problems caused by device or sensor failure versus unusable data versus corrupted data versus malicious data, he says. Only after they clearly identify the cause of the problems in this way can they solve them.
Why automation matters
It's impossible for humans to manually compile, compare, and analyze the tremendous amounts of data IoT networks generate. Increasingly, companies use machine learning to flag anomalous data and behavior.
“With machine learning, you can far more easily model what is normal data and what is anomalous data,” Tennefoss says. “Machine learning also does a great job when you need to analyze contextual data, such as the time and day of the week or location.”
IoT data can offer tremendous insights into improving quality, efficiency, and productivity. “All this new data changes the way that people think about running their factories,” Marx says. “In a way, we’re all becoming data scientists now, no matter our job titles. But the decisions you make are only as good as the data you have. So it’s more important than ever that people choose the right systems for making sure their IoT data can be trusted and doesn’t go bad.”
IoT devices and the data they collect: Lessons for leaders
- IoT data can yield valuable insights to help manufacturers improve quality, efficiency, and productivity.
- To identify problematic devices and data, companies must create baselines for "normal" IoT performance. This is easier said than done.
- IoT data is rarely clean and comes in many different formats. Companies must understand where it comes from, how it was cleaned, and by whom.