Embedded Software Testing: The Challenges of a 'Smart' World

Embedded systems can easily pass by unnoticed, but they are almost omnipresent in our day-to-day life. From smartphones and TV remote controllers, to cars and aeroplanes, embedded systems are essentially driving the modern world right under our noses. 

As with any other kind of system, embedded systems must be tested. The testing phase is the most important stage in the development lifecycle and it’s when requirements are analysed and ensured to be met. This phase begins with unit testing, which is then followed by integration testing, and in turn system testing.

Testing embedded systems comes with numerous challenges. To understand these challenges, we first need to understand the difference between embedded software and high-level applicational software.

In short, embedded software is less visible to the end user. Interfaces are quite limited; there’s usually a console-based application or a simple command-line interface through which the software is operated. The second main difference is hardware dependency, as the embedded software is developed to run on a particular hardware device and acts as the Operating System by itself.

Challenge #1 – Hardware Dependency

Because embedded software is strongly dependent on the hardware device it resides in, testing is naturally a serious challenge. Given that the hardware platform may not be available at the very beginning of the testing process, these initial activities are usually carried out with no access to the system hardware platform. Emulators and simulators are sometimes act as a workaround, however there’s always a significant risk that they don’t represent the real behaviour of the actual device and could give an inaccurate sense of system performance and usability, potentially leading to greater issues down the line.

Challenge #2 – Defects Can Be a Struggle

A high ratio of system defects can be identified during the testing process. This can become a struggle as the problem can either be with the software itself or the hardware it’s applied to, or even to a combination of events in both specific software and hardware. Debugging becomes unusually more complex and, therefore, having both software and hardware knowledge is of the upmost importance. Quite often, the developed embedded software might work as expected in one hardware variant but not in another, which adds yet another obstacle to both the testing process and development lifecycle.

Challenge #3 – It’s Not Easy to Reproduce Defects

Embedded systems make it harder to reproduce defects because the it implies reproduction of events in both software and hardware. This means that the testing procedures must consider every defect occurrence at a much higher level of analysis than in a standard case. Additionally, gathering this reproduction analysis data may sometimes require the deliberate change of the system so as to find the source of said defects.

Challenge #4 – Mid-life Upgrades

Embedded systems may require frequent updates like kernel upgrades, security fixes, different drivers, RTOS upgrades, or even microprocessor changes. All these changes have, of course, a direct impact on the testing activities and their complexity given the diversity of the potential areas of change. As a mitigation measure, special care must be given to the build, production and deployment procedures to ensure that the exact system environment can be discovered and replicated for testing if needed.

Challenge #5 – Test Automation

Embedded systems bring with them additional challenges in regards to test automation when compared to regular software testing. Given their previously mentioned hardware dependency, and the typical hardware interfaces with other systems / sub-systems, test automation of embedded systems usually requires the development of test rigs supporting not only software but hardware automation as well. The development of said test rigs enables automated test campaigns that are essential for successful regression testing.

There is no doubt that embedded systems testing has the potential to be more difficult than application software testing. The hard dependency on hardware may be the main cause of all these challenges, since testing teams do not always have access to the hardware environment in which the software will operate and frequently need to rely on simulators to proceed with these activities. 

Additionally, it can be hard to test embedded systems without custom tools, which in turn also require the developers’ attention to figure out exactly what the testing process will involve if they’re to find the correct tool and features to take it forward. Yet despite the challenges, and in light of its ubiquitous role in our society, testing has taken on an even more significant role in the world of embedded software.

Learn more about our work on embedded software development and testing here.