Effective Test Strategy for IOT

Embedded software has been around for years, going back to the dawn of computers. Traditionally we tested these devices in isolation and did not worry about user interfaces (if there was one) or things such as internet connectivity. The connectivity of devices started not long after the internet’s arrival. However, in recent years the so called “internet of things” (IOT) has become of more importance and certainly more newsworthy as its use is growing rapidly.
IoT devices will share the development and test issues found in embedded software systems as well as more traditional IT/Web systems. With the growth in numbers of IoT devices and software projects the need for testers and approaches for these devices will likewise increase. Testers coming from these historic environments will face different testing challenges and bugs. This article outlines some starting points for those going into IoT testing and offers considerations for those already testing IoTs. Testing is a large subject with many books and thousands of articles, so readers should follow the links and resources to continue their learning. Remember, no one can know it all, but there are great reference materials available in many forms.
Examples of Product Test Challenges and Risks that IoT Testers Face
Testers face both new and old potential problems (errors) in IoT devices. These include:
  • Embedded functionality,
  • Web provided functionality,
  • Performance both of the network communication and internal computation,
  • Security including privacy, autonomy and control,
  • Smartness of the device and the user interface or of the software in some devices (may hide bugs),
  • Architecture of the hardware and of software, means more configurations must be tested, e.g., Android fragmentation [http://opensignal.com/reports/fragmentation-2013/ ],
  • Complexity of the software and system (means more bugs may be in the code hiding in the complexity),
  • The devices may have large amounts of code e.g., smart phones now have 10-to-20 million lines of code (where errors can hide),
  • Development time considerations, such as time to market pressure, which exists in IT and Mobile, will continue with IoT,
  • Resource considerations such as limitations in: memory, processing power, bandwidth, battery life, etc.
Unique environments the devices will be used in: hot, cold, wet, noise, at altitude, etc.
Many testers will be familiar with two or three of these issues but not the others. For example, many historic embedded software testers verified functionality and CPU timing issues yet did not worry about connectivity, performance, security, usability, or large amounts of code. Historic Web/IT testers worked these secondary items and did not worry about issues common in embedded systems such as: limited resources, unique hardware functionality, and high-risk, critical device control problems.
Additionally, I have heard project stories where historic embedded devices were “updated” with a network card or mobile connection. The embedded device was working so all the new testing focused only on the “new” connection. Could there be a problem with this line of thinking and how much would that cost the company? Consider the possible limitations of this simplistic initial testing and usage:
  • Security holes from the historic code may be missed.
  • Performance testing was CPU usage based and did not consider the impact of the connections, e.g., long waits (seconds versus milli- or micro seconds), loads, slow network, dropped connections, etc.
  • Viability and completeness of recorded data.
  • Usability of the system with the new connection.
  • Coupling impact from the new logic to existing functionality.
Certainly these challenges and risk are not the only ones IoT testers will face, but these are a start. And, once costs are examined with finding issues after a product is released, companies could lose a lot of profit.
A Start: IoT Strategy and Planning Impacts
A team should consider the implication of test strategies for both the new and re-hosted IoT device. I would start by obtaining and using IEEE1012 Verification and Validation standard [http://standards.ieee.org/findstds/standard/1012-2012.html]. Using this standard, I would assess device V&V test activities against my estimations of risk and determine an integrity level (defined in IEEE1012 and determine amounts and types of test activities). When dealing with a historic device, try analyzing white and black-box coverage levels (e.g., statement coverage, requirements coverage, performance analysis, etc.). When dealing with new devices, consider the product’s risks, quality characteristics and functionality. Finally, consider the strategy in light of allocated cost and schedule. The complete strategic information is reviewed with the stakeholders so that everyone agrees on the strategy before beginning test planning and design efforts.
Next Step: IoT Implication to test plans
Once there is agreement on test strategy, use it to guide the IoT software test plan. Here again, if you are new to IoT testing, continue with a refinement of the concepts from IEEE1012 to the next level of detail. Follow this planning with the test concepts, processes, and techniques from ISO29119. When using standards, tailor them to your local context, since a standard is only a basic beginning and not an end or best practice. A test organization that already has strong test practices and skilled testers might not need this standard since a skilled group can leverage their history and knowledge to start an IoT test plan. However, for test groups without much IoT history, I would analyze in more detail the testing that has been completed, look for error taxonomy information [ref me], determine what test concepts to include, and have a sound method for regression testing [http://www.logigear.com/magazine/issue/3350/ ].
Both new and historic IoT organizations should consider what test concepts and environment should be added to the test plan including:
  • Risk-based testing [29119];
  • Test attacks [Whittaker, How to Break Softare, Hagar, Software Test Attacks to Break Mobile and Embedded Devices] to find risky bugs;
  • Exploratory testing times and efforts;
  • Required (regulatory) scripted testing and documentation [ISO29119-3 ];
  • Test tools and automation needed;
  • Test lab(s) set up;
  • Test tours [Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test
  • Design, Whittaker ] to use; and
  • Test techniques to apply [ISO/IEC 29119-4].
  • An example IoT test lifecycle pattern for a test plan might look like:
  • Strategy
  • Plan
  • Design with regression considerations
  • Act (test)
  • Report and document [ISO/IEC 29119-3]
  • Repeat (within resource boundaries such as test team skill, cost, and schedule).

These activities might take day or hours, depending on the project context.
Finally, I find it is easy to forget things in test planning, so I like to use checklists to help me complete my planning, remembering that as soon as I start to execute my plan, the plan will change and I’ll have to refer back to my strategy, plans and checklist frequently. A sample beginning checklist is given in table 1.
Test Tools Needed to Support IoT
When a tester says “test tools” everyone typically thinks automated test execution tool, and while this is part of the story when I say the word “tool”, I mean anything that helps me to do better testing. Tools can be pieces of software, such as capture-playback tools, but a tool can also be a checklist, which supports manual testing.
I recommend white-box and black-box testing including analysis concepts such as static code analysis tools [Hagar, Software Test Attacks to Break Mobile and Embedded Devices] . These levels and approaches to testing allow testing, verification, and validation to be done throughout the lifecycle. Also, these combinations are complementary increasing the likelihood that errors will be found. Finally, many IoT embedded projects may benefit from the use of model and mathematical analysis which, in my experience, more progressive organizations will have the ability to use.
Classic test execution automation will support many of the issues in IoT such as testing device configurations and capture/playback. Support by vendors for embedded, mobile, IoT testing has been increasing in recent years. Teams working on IoT are advised to conduct tool trade studies and searches to find the best candidate tools for their project’s context.
The use of tools and automation does not mean that all testing should be automated. I find the advanced groups mix automated test execution and manual testing, particularly guided exploratory testing, with tours and attack patterns [Whittaker and me]. This follows the concept of having complementary approaches and activities to guide the testing. Complementary ideas would be reflected in test plans and designs.
Recommend Tester Knowledge and Skills
More than test processes or tools, skilled testers are needed for IoT. Software testing is practiced well when the knowledge and skill of the person doing the work determines the effectiveness of the effort. A project can have good practices and the right tools, but unless there are skilled people to drive these efforts, good testing may not be realized. A skilled test team would have knowledge in the following areas:
  • Web environments
  • Embedded environments
  • General test considerations (knowledge e.g., ISTQB and skills such as those outlined in the AST skills guide
  • Hardware understanding
  • Systems thinking
  • Network communication understanding
  • Performance test experience and
  • Experience with testing other quality characteristics associated with the IoT context.
Likely no single person will have all of these, so a diverse team of experienced and new testers will be optimal. Additionally, training, on-the-job learning and mentoring should be included as part of the project test plans.
Summary
IoT has been around for years, but lagged in usage behind the Web/PCs and smart phones. Now there are indicators that the sheer number of devices and software in IoT is growing rapidly day-by-day. This means that more testing and testers will be needed to minimize the bugs in the IoT devices being released to consumers. This article has introduced some of the problems IoT testers many face, and made some high-level recommendations for testers in the area of test strategies and planning. Like all test contexts, there is much more to this subject. More work on strategies, planning, error taxonomies and tools for IoT is needed.

Comments

Post a Comment

Popular posts from this blog

How to calculate maximum number of concurrent users using littile law formula

When load testing an application, the first set of tests should focus on measuring the maximum throughput. This is especially true of multi-user, interactive applications like web applications. The maximum throughput is best measured by running a few emulated users with zero think time. This means that each emulated user sends a request, receives a response and immediately loops back to send the next request. Although this is artificial, it is the best way to quickly determine the maximum performance of the server infrastructure. Littile Law: Once you have that throughput (say X), we can use  Little’s Law  to estimate the number of real simultaneous users that the application can support. In simple terms, Little’s Law states that : N = X  / λ where N is the number of concurrent users,  λ is the average arrival rate and X is the throughput. Note that the arrival rate is the inverse of the inter-arrival time i.e. the time between requests. To under...
Read more

How to develop a security test strategy

What do you need to write a security test plan? The folowing documents comprise the list of what I would expect as inputs to the creation of the individual security test plans. This is a good point to go and review your overall security delivery plan. Does it include these documents as deliverables? Does the supplier have any of these as standard off-the-shelf products? : System security requirements System threat model System risk model Secure coding standards Technical threat modelling Infrastructure secure configuration guides Security defect metrics What is in a security test plan? The security test strategy should as a minimum specify the following points as the core content for each of the security test plans: What are the inputs to the test plan? When in the process does this test occur? Who specifies the test plan contents? Who performs the test? How many instances of the test are expected? What are the required outputs of the test? What are the success...
Read more

Singleton Design Pattern using Page Factory

Image
n this tutorial, We will learn about the  Framework Design Pattern using singleton class  in Selenium. In the Previous tutorial we have seen, How can we run the Test using Page Factory. Now question is, Will we repeat the Browser invoke and Logger code again and again in each and every Test. No, To resolve this issue we will use the Singleton Pattern. We will design the Single  Test Base  class.  First, Let's see what is  Singleton Pattern .  With the Singleton design pattern you can: Ensure that only one instance of a class is created Provide a global point of access to the object In  TestBase  class, we will Design the Singleton Pattern, We will write the command code here and all Test Classes will inherit this  TestBase  class.  n previous Example, we have seen we were invoking  WebDriver   instance  in each @Test. Here, We will Implement the WebDriver initializat...
Read more