Table of Contents
This article motivates developers to test code, explains the importance of testing, and details the testing performed in Autoware.Auto.
Quick reference
- colcon is the tool of choice for building and running tests
- ament_cmake is useful to specify tests in CMake
- Chris Hobbs' Embedded Software Development for Safety Critical Systems, describes tests necessary for code running in safety critical environments
- ISO 26262 standard part 6 prescribes how to test code in automotive systems
- SQLite is a software project that has an impressive and thoroughly described testing system
Importance of testing
Dynamic and static testing methods make Autoware.Auto reliable and robust, helping us to perform anomaly detection and handling that would otherwise be difficult to find. Through testing in Autoware.Auto, we can estimate the number of Heisenbugs, and find and eliminate undefined behaviors for which C and C++ languages are known for.
Dynamic analysis, simply called “testing” as a rule, means executing the code while looking for errors and failures.
Static analysis means inspecting the code to look for faults. Static analysis is using a program (instead of a human) to inspect the code for faults.
There are also formal verification methods (see the book, Chapter 15); note that the topics will not be covered in this document.
Testing in Autoware.Auto
This section introduces various types of tests that are run both manually and automatically.
Style / linter tests
Some examples of tools used for style and linting are cpplint, uncrustify.
Tests using the tools above allow Autoware.Auto to follow C and C++ style guides which results in uniform, easy to read code.
Static code analysis
The Cppcheck tool is used for applications written in Autoware.Auto.
Static code analysis tools detect the following types of errors:
- API usage errors
- Best practice coding errors
- Buffer overflows
- Build system issues
- Class hierarchy inconsistencies
- Code maintainability issues
- Concurrent data access violations
- Control flow issues
- Cross-site request forgery (CSRF)
- Cross-site scripting (XSS)
- Deadlocks
- Error handling issues
- Hard-coded credentials
- Incorrect expression
- Insecure data handling
- Integer handling issues
- Integer overflows
- Memory—corruptions
- Memory—illegal accesses
- Null pointer dereferences
- Path manipulation
- Performance inefficiencies
- Program hangs
- Race conditions
- Resource leaks
- Rule violations
- Security best practices violations
- Security misconfigurations
- SQL injection
- Uninitialized members
Unit tests
Unit Testing is a software testing method by which individual units of source code are tested to determine whether they are fit for use.
The tool used for unit testing in Autoware.Auto is gtest.
Integration tests
In Integration Testing, the individual software modules are combined and tested as a group. Integration testing occurs after unit testing.
While performing integration testing, the following subtypes of tests are written:
- Fault injection testing
- Back-to-back comparison between a model and code
- Requirements-based testing
- Anomaly detection during integration testing
- Random input testing
Memory tests
Memory tests allow the detection of unwanted calls to memory management APIs, such as:
malloccallocreallocfree
For more details on memory tests see the memory testing tool.
In-vehicle tests
Regular in-vehicle testing is performed as part of ODD development and demonstrations. These tests validate Autoware.Auto in a realistic autonomous vehicle product.
