The State of IoT Testing

IoT is moving past the hype curve, past the trough of disillusionment, and into the slope of enlightenment—the model used by Gartner to estimate the traction any new technological innovation currently has. MachNation estimates hyper-growth for IoT in the coming decade.

Source: MachNation

With revenue from IoT platforms pegged at a mere $3B in 2018, it is set to explode to $65B by 2026. The reason for this growth is that IoT touches pretty much every industry vertical, and its applications are wide-ranging within each sector. And since there are many aspects to IoT— devices, development platforms, local and wireless networks, data integration, and even more to consider—There are plenty of difficult challenges to be solved, and many organizations are jumping on board, seeing the opportunity.

Building and maintaining an IoT system is not easy, however. In a survey conducted by Cisco, 60% of companies found it more challenging than expected to build IoT services.

Source: Cisco

Additionally, 75% of IoT projects done in-house failed to meet their objectives. So, despite the opportunity and healthy growth levels, IoT is still uncharted territory for most organizations.

This is evident in the kind of mishaps that we see in the news each month. Not too long ago, a huge DDoS attack took place when Dyn’s servers crashed due to traffic overload from infected IoT devices. These devices were DVRs with outdated firmware that hackers used to overload Dyn’s servers, along with some of the top websites that depend on Dyn. The dangers of IoT are as big as the opportunity they promise.

The key to building reliable, secure, and high-performing IoT systems is to ensure your testing strategy is IoT-ready. However, most organizations don’t have an IoT testing strategy.

Source: Capgemini

According to a survey by Capgemini and HPE, 68% of organizations don’t have an IoT testing strategy. This could be because these organizations are still exploring the right mix of devices, applications and components that make up their entire IoT stack, and they are yet to get to the finer details of building reliability, security, and high performance with testing and QA. That said, it is vital that organizations prioritize IoT testing—especially those which have IoT applications in production or IoT edge devices and sensors out in the field—before a major catastrophe hits.

There are many opportunities to drive results with IoT testing. From monitoring and diagnostics to APIs, stress and performance testing, and using device simulators, testing can solve a range of problems across the spectrum with IoT.

Source: Capgemini

Crafting an IoT testing strategy starts with understanding your priorities. There can be several, depending on the organization and business focus. Some of the main priorities are:

Once the priorities are in order, the next step is to take stock of the IoT system with all its complexity. There are typically four parts in an IoT system: devices, applications, networks, and the data backend. Devices can range from simple sensors to complex industrial robots. Applications can be mobile applications, web applications, or embedded native applications that are meant to work only on a particular device or platform. Networking for IoT can range from simple radio frequency to more standard options like bluetooth, WiFi, and mobile carrier networks. There are also IoT-specific lightweight communication protocols like ZigBee and Z-Wave. Finally, the data backend is critical to IoT. A large portion of development time can go into data integration and data wrangling. This is challenging because enterprise applications often require custom data integration plugins and scripts to be written that are optimized for integration with an IoT system. When looking to speed up development of IoT applications, data integration is the place to start.

Many types of IoT-specific testing are inherited from traditional testing approaches but have certain quirks that apply to IoT. For example, with security testing, the firmware version running on an IoT device is of particular importance. Outdated firmware can have security loopholes that compromise the entire system. Additionally, with many IoT devices in remote locations and intermittent connectivity, database testing and performance testing will need to take into account these suboptimal conditions, which are often the reality when it comes to IoT.

There are multiple IoT testing tools and generic IoT platforms that include testing features that should be leveraged according to the use case. JTAG is a consortium for testing of hardware like ICs and PCBs used in IoT devices. OWASP Zap is a penetration testing tool that helps secure IoT applications. BeEF is another penetration testing tool, but it focuses exclusively on vulnerabilities in a web browser. Apart from these niche testing tools, there are broader IoT development and management platforms with some testing functionality built-in or enabled via an integration. These include tools like Kaa, Thingsboard, Thinger, and Eclipse Kura.

In summary, IoT is coming of age and finding more and more real-world use cases. As IoT services reach customers and end-users, the expectations are higher than ever. Meeting these high expectations takes robust IoT testing. An IoT testing strategy starts by defining an organization’s objectives, taking stock of the various pieces of the system, and devising the right tactics and tools to perform the IoT testing.

Twain Taylor is a Fixate IO Contributor and began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.