How to Test Hardware: Hardware in the Loop Testing (HITL)
This article was initially published in my newsletter. Subscribe here if you like this kind of content.
The secret superpower of engineering teams that want to move quickly and ship safe, high-quality software to complex, expensive, or dangerous products is something known as Hardware in the Loop Testing (HITL or HIL for short). If you ask around with any successful or high-functioning engineering teams you admire, you will find that many credits their HITL systems as the secret sauce which allows them to move fast without breaking expensive things.
What
A hardware-in-the-loop test system is designed to test software on real hardware in a safe environment before deploying the software to a real environment. This type of testing ensures that everything is functioning as intended in an environment where problems are acceptable.
Why
You might question whether it is worth it to build something like this. After all, it will likely be expensive as the system will require much of the same hardware as building an actual version of your product and will take up lots of space and time. However, the benefits are clear and easily justified.
Imagine you are building a new electric car. You could simply build several test units of the car and do all of the tests with that. However, there any many scenarios that are dangerous to test. For example, how would you test the behavior of the software when the driver shifts gears while traveling at high speeds? Some tests are also very difficult to do with the completed product: Example: How do you test the software’s behavior when a part of the product loses power during operation?
Hardware-in-the-loop testing allows engineers to run these tests on real hardware in a safe and repeatable way, both saving costs and improving the reliability of the product.
In addition, HITL testing can also help to ensure regulatory compliance. For example, in the automotive industry, there are strict regulations for vehicle safety and emissions. HIL testing can help automakers ensure that their vehicles meet these requirements before they are deployed in the market. Without HIL testing, it would be challenging to test the vehicle's performance under specific conditions to ensure regulatory compliance. In a past job, I was required to run 10,000 variations of a specific operating mode of a product and produce a log of the software’s reaction to prove the product’s safety to regulators. This testing would have taken years and millions of dollars with the completed product, but because of the robust HITL testbed we had created, this was achieved in only a few days.
In conclusion, hardware-in-the-loop testing is an essential step in the development process of any hardware product. It provides developers with a controlled environment to test the hardware product's performance, identify weaknesses in the design, and ensure regulatory compliance. HIL testing can also help to reduce development time and costs, improve the customer experience, and ensure that the hardware product performs as intended in the real world. As such, it should be a crucial part of any hardware product development process.
How to Build?
What follows is a practical guide for building hardware-in-the-loop test system. I hope that you find it helpful and will guide you along the way.
Theory
It’s important to understand the theory behind hardware-in-the-loop testing before beginning to build a system. There are lots of choices to be made and it is easy to create a system that does not actually achieve the goals of the testing even though it may be very large and impressive.
A hardware-in-the-loop system is designed to test the exact software that will be deployed to the real product. This implies several things:
The software for the product should be able to be packaged and deployed to more than one system in a repeatable way
The software loaded onto the units being tested should be modified minimally or ideally not at all when running in the testbed
This implies that the HITL system should include real or simulated versions of all hardware
Hardware-in-the-loop test system designers should remember those points when building the system. Poorly designed systems often become littered with “test only” conditionals or special builds of software. Each of these deviations represents a potential difference between the test environment and the real system, therefore, creating the probability for something to be missed.
Example Robot
For purposes of demonstration, let’s imagine a fairly simple product: a line-following robot. This is a common project built by engineering students everywhere and is quite simple as an example for this article. The robot’s goal is to follow a line drawn on the ground using a simple control loop and a minimal set of hardware.
It contains four primary components:
A battery that provides power to everything
Two motors to drive the robot
Two IR sensors to detect the edges of the line the robot is supposed to follow
A robot controller (microcontroller or computer) reads the input from the two sensors and commands the motors accordingly
The robot controller contains some software that we would like to test without having to constantly the batteries or set the robot loose in the world.
Now that we have an example system to test, we can construct a basic hardware-in-the-loop testbed for it.
Test Controller
The test controller is the central brain of the HITL testbed. It’s notable because it is the first main piece of equipment needed that is not part of your production product.
It has a few jobs:
Load the correct software onto the devices under test
Start and stop tests
Sequence events during a test
Control test hardware and simulations
Monitor the results of a test
Report the outcome
The hardware selection of the test controller is generally not terribly important as this piece of equipment will only be used in the comfy environment of the test lab. It will not need to experience environmental conditions or be subject to weight or power constraints. In the line follower example, perhaps a single-board computer like a raspberry pi could work nicely.
Power
One of the most important abilities of the HITL system is the ability of the test controller to control the power to parts of the product under test.
This allows the test to ensure that each test run starts with the unit under test is powered on, thus ensuring that the system is in a known state at the start of each test. It also frees the robot from the constraints of a battery which may often need to be removed and recharged.
It is also useful to separate the power control into sections that allow the test controller to control power to different parts of the system separately. In our line-follower, this would mean having independent power control of the IR sensors, Robot Controller, and Drive Motors.
Sensors
The behavior of any interesting piece of hardware changes based on input from its sensors. This can be from humans interacting with the device or the device observing the world around it.
Sensor Examples
Pressure Transducers
Electrical Voltage or Current Sensors
Temperature Sensors
Position Sensors
Buttons
Cameras
In order to test the software, these sensors must either be connected to the test system or simulated in some way.
Looking again at the line-following example, the robot has two IR line sensors. When integrating sensors into the hardware-in-the-loop system, it is important to note that the sensors will not be experiencing the normal input of the product. In this example, the IR sensors will not be rolling around on a robot following a line.
This reality creates an important choice that must be made for each sensor. The test system can either:
Use the actual sensors connected to a physical simulator
Pro: This method ensures that the control module software can always communicate with the real sensors
Con: This method requires some additional equipment that presents the real conditions to the sensors to match the intended test case (for example a temperature sensor must be placed inside of a thermal chamber). This can be complex, expensive, and large.
Going again to the line-follower example, we could add a servo with a wheel that can be rotated to expose either a light or dark material to each IR sensor.
The test controller or additional hardware connected to the test controller must simulate the readings from the real sensors.
Pro: This method is generally quite small and allows for infinite control of the simulated sensor readings as they are simulated and not bound by physics (for example temperature changes can happen instantly instead of waiting for the temperature chamber to actually change).
Con: An additional test must be constructed elsewhere to ensure the software can communicate with the real sensor.
Following this method with the line-follower robot results in the diagram below. You can see that the IR sensors are completely removed and the Test Controller is now directly connected to the Robot Controller.
Actuators
Actuators are the parts of the system that interact with the world in an active way. In the line-following robot example, the actuators are the two motors that drive the wheels.
Examples of Other Actuators
Motors
Linear Actuators
Hydraulic or Pneumatic Valves
Electrical Relays
Digital I/O Pins on a Circuit
Just as there are two choices with sensors, there are two options for integrating actuators into the hardware test system. The pros and cons of each are largely the same as the sensors, so I won’t reiterate them here.
Use the actual actuators connected to a physical simulator
Expanding on the line follower example with real sensors above, we end up with the following diagram:
There are two new controllable motor loads that are physically connected to the drive motors and are commanded by the test controller to simulate different loads on the motors for different test cases: driving up an incline will produce a higher load on the motor. This method is helpful in validating different control parameters such as gains or power supply limits which are often typically set during field testing of finished products.
The test controller or additional hardware connected to the test controller must simulate the interface and response of the actuator.
You can see above that the robot motors have been removed and the robot controller has now been connected directly to the test controller.
Cables or Harnessing
One important note is that each of the connective lines between the test equipment in the diagrams above represents physical harnesses or cables.
This is important because it allows the software on the product to operate exactly as it would in the real product. It is easy to be tempted to cut corners here and connect the robot controller to the test controller via a software or network interface instead of a physical cable, but that is a mistake. That represents a very large deviation from the goal of testing the software in an unmodified state and will certainly cause more headaches than help in my experience.
Simulator
Now that we have the sensors and actuators hooked up to the test controller, the next logical step is to create some sort of simulation that runs on the test controller which relates the output of the actuators to the sensor input.
This is an area of development that starts off simply but can grow in complexity and fidelity forever. Most people start with some sort of rough heuristic simulation (if this, then do that), but for complex projects, it can grow to a large multi-physics simulator that attempts to accurately model an entire environment in which the product operates.
Creating Failure Test Cases
One of the most interesting and useful things that can be done with the testbed is to create test cases that test the software during faults or unexpected conditions.
In the line-following robot example, here are some ideas of potentially interesting fault cases to test:
Loss of power to one or both motors - What happens if one of the motors loses power?
Loss of communication to the sensors - Verify that the software does the correct thing when the robot cannot communicate with its sensors
Nonsensical Data from Sensors - In reality, the line following the robot’s two sensors should never both detect the line at the same time. If this is detected if means that the robot has likely left the line entirely or one of the sensors is no longer functioning correctly.
Low Battery - Simulate a very low battery voltage. Does the software do the correct thing?
Continuous Integration
It is likely that test engineers will start out building the hardware-in-the-loop test system and starting tests manually at first while they gain confidence in the test system. Once the system is verified, additional reliability and speed can be granted to software teams by running the hardware tests as part of the Continuous Integration (CI) Software System.
Software teams are used to having automated CI systems run Unit Tests, Integration Tests, and Systems Tests before code is reviewed or merged into a larger set of the codebase. Why not the hardware tests?
It is relatively straightforward to integrate the test controller equipment into a CI system. For example, Golioth.io demonstrates a nice example here using GitHub Actions and Self-Hosted Runners.
Cost
The concepts and ideas described above are quite high-level and abstract from most actual hardware selection. Selecting the correct hardware choices for a particular project requires a specific study on the product and the problem domain you are working on. However, I did want to make it clear that all of the concepts above can be achieved with relatively low-cost hardware choices. That might not always be the correct choice for a particular project or the long-term future of a project, but it is always an option to start.
I make this point to ensure that teams don’t get stuck reading too many National Instruments blogs and start thinking they need to spend $2 million dollars immediately in order to start building any sort of hardware testing system.
Incremental Improvement
Obviously, the ideal state for any hardware testing system is to fully test the entire system. However, like many things in life, this is often not practical to do immediately and incremental progress is usually better than none.
Reading through the examples above, it becomes clear that it is possible to build hardware testing systems that only exercise a subset of the hardware initially. Generally, it is the correct choice to pursue building a system like that as a first step and then charting a roadmap towards a complete system from there.
Need Help?
Although the guide above is correct, any reasonable hardware in the loop system requires many specific features or design guidance. Keenan Johnson has extensive experience building hardware in the loop systems and is available to help a limited number of teams in a hands-on way.