IoT Testing: Challenges and Approach

IoT enables the connection of huge variety of devices to the Internet such as home appliances, lights, sensors, vehicles, and communication devices, paving the way for complex ecosystems like smart homes, smart buildings, and even smart cities. The innovation and telecommunications market are faced with a strong demand for value-added services and IoT-based services are one of the biggest innovation bets for the embedded software devices market, which boosts the qualification and delivery of test services with expertise in embedded systems. Throughout this article, the challenges related to software testing for IoT, and the testing processes of these devices will be addressed in order to ensure correct product validation, which is essential for the success of the product and its acceptance by the market.

Challenges

As a new technology, IoT opens a gap in the product validation industry where traditional software testing strategy may not be fully suitable for IoT validation. IoT software testing has several challenges that needs to be well addressed in order to ensure a robust software testing procedure. Among the challenges that may be faced by IoT software testing, it can be highlighted:

Test Environment

IoT technology allows for an ecosystem with the coexistence of devices with different characteristics, which makes the test environment very complex. In the IoT infrastructure, there may be interaction between devices with different versions of firmware, operating systems, and browsers, which makes it difficult to streamline the testing process.

Security Risks

Connecting devices to the Internet opens up a range of possibilities in terms of services, but on the other hand, numerous challenges related to security must be addressed. Security is a major challenge for the IoT as all tasks are performed using the network, which can result in attacks on previously unthinkable services and functions.

Stability

Stability is a major concern for the IoT since many applications using this technology requires high system availability. Connecting devices to the Internet is the main stability challenge, as connection issues can render the IoT device useless.

Diversity

IoT software testing is not just testing the embedded software of the IoT device. For a correct evaluation of the product, it is also necessary to test the application that controls the IoT in a variety of devices, which increases the complexity and test volume.

IoT Testing Approach

In order to ensure efficiency in the software testing process for IoT, a robust testing strategy is needed which considers the characteristics not only of the IoT device, but of the entire ecosystem that involves the IoT. The test approach for a IoT will vary depending on the characteristics of the product under test, but the following tests are common for most IoT devices and should be included in the test cycle considering the specifics of each product.

Usability Testing

In an IoT ecosystem, there is interaction with different devices with different shape, size and form factors. Usability testing is essential to assess the human perception when interacting with the product during data processing and display. During the usability testing, several items can be evaluated such as:

• Quantitative usability: measures the performance of the user under a given task. It focusses on obtaining metrics related to the software effectiveness, efficiency and satisfaction.
• Qualitative usability: identifies issues related to usability during interaction with software. Qualitative evaluation focus on the understanding of human needs and reason for the given human behavior during the execution of a task.
• Accessibility evaluation: addresses the interaction between a user with disabilities or limitations with the software. It will determine if the accessibility aspects were properly implemented as per standards or product requirements.

The usability testing can be applied to multiple items in the IoT ecosystem even to simple devices as sensors where the usability can be assessed during the sensor installations and configuration.

Compatibility Testing

In an IoT environment, it can coexist devices with different hardware versions, running different software versions or even using different browsers. Multiple combinations of hardware and software in the IoT ecosystem can lead to compatibility issues that may be the gate for the unsuccess of the product. During the compatibility testing it will be assessed the correct behavior of the software in the presence of multiple configurations simulating real conditions in the field. This test is directly linked with the success of the product and is a challenging in terms of resource consuming and volume of tests. In order to enable the compatibility testing, automation solutions should be applied to address the huge volume of tests linked to the multiple configuration environment. Automation solutions with cloud capabilities may be an interesting approach as it facilitates the test execution in different platforms which is the aim of the compatibility testing

Stability Testing

Software reliability testing involves testing the ability of the software to operate, under given environmental conditions, for a given period of time to uncover numerous problems in the design and functionality of the software. Automation can be an important support for the stability tests especially for the tests with long durations. Given the application of the IoT, stability can drive the adoption of the solution especially in high complexity environments such as smart cities. Stability related to the connectivity is the most critical aspect during stability assessment as the loss of connection can render a IoT device useless and even lead to massive malfunction in the hole IoT environment that relies on the embedded software distributed in different locations.

Security Testing

Connecting devices to the Internet brings numerous benefits and enables the creation and implementation of various services, however, it brings security risks due to the large number of users who access a vast amount of data. During security testing, many aspects will be inspected to validate the user though authentication and implementation of data privacy controls. Those tests will surround security areas such as login, password, security boot and device authentication.

Functional Testing

Functional tests consist of when the various functions implemented for the IoT will be tested. During this test phase, it is also possible to execute the various use cases provided for the product in order to guarantee the correct execution of the software functions. This test also ensures that each device performs all its functionality efficiently, ensuring that the entire IoT network can work collaboratively and perform as intended.

Conclusion

The Internet of Things has proven to be one of the biggest bets in the innovation market for value-added services. IoT enables countless applications ranging from a connected lamp to smart cities. With the strong growth of the IoT, there is a need to implement adequate techniques for testing software for IoT, since the qualification of the IoT presents several challenges related to its ecosystem. Proper coverage of IoT testing requires a testing strategy with full coverage of the IoT ecosystem. The definition of tests depends on the characteristics of the devices under test, but must include usability, security, stability, and compatibility tests as well as automation for a timely test execution. The success of IoT device is directly linked with a proper software testing process during product development. With 20 years of experience in software testing domain, onepoint plays a major role in streamlining customer’s software validation process. That includes from the test strategy as a quality milestone, where an efficient test approach is depicted, up to the product approval based on the KPIs generated by the manual and automated testing. onepoint follows up the customer on ensuring testing best practices based on methodological advice, sourcing teams as well as providing guidance on test automation. Moreover, onepoint transverse approach based on skilled communities liaise testing activities with complementary strategic areas, which represents a complete solution to ensure a successful product deployment.