embedded - How can I unit test Arduino code?

Question

I'd like to be able to unit test my Arduino code. Ideally, I would be able to run any tests without having to upload the code to the Arduino. What tools or libraries can help me with this?

There is an Arduino emulator in development which could be useful, but it doesn't yet seem to be ready for use.

AVR Studio from Atmel contains a chip simulator which could be useful, but I can't see how I would use it in conjunction with the Arduino IDE.

Answer 1

Don't Run Unit Tests on the Arduino Device or Emulator

The case against microcontroller Device/Emulator/Sim-based tests

There's a lot of discussion about what unit test means and I'm not really trying to make an argument about that here. This post is not telling you to avoid all practical testing on your ultimate target hardware. I am trying to make a point about optimizing your development feedback cycle by eliminating your target hardware from your most mundane and frequent tests. The units under test are assumed to be much smaller than the whole project.

The purpose of unit testing is to test the quality of your own code. Unit tests should generally never test the functionality of factors outside of your control.

Think about it this way: Even if you were to test functionality of the Arduino library, the microcontroller hardware, or an emulator, it is absolutely impossible for such test results to tell you anything about the quality of your own work. Hence, it is far more valuable and efficient to write unit tests that do not run on the target device (or emulator).

Frequent testing on your target hardware has a painfully slow cycle:

1. Tweak your code
2. Compile and upload to Arduino device
3. Observe behavior and guess whether your code is doing what you expect
4. Repeat

Step 3 is particularly nasty if you expect to get diagnostic messages via serial port but your project itself needs to use your Arduino's only hardware serial port. If you were thinking that the SoftwareSerial library might help, you should know that doing so is likely to disrupt any functionality that requires accurate timing like generating other signals at the same time. This problem has happened to me.

Again, if you were to test your sketch using an emulator and your time-critical routines ran perfectly until you uploaded to the actual Arduino, then the only lesson you're going to learn is that the emulator is flawed--and knowing this still reveals nothing about the quality of your own work.

If it's silly to test on the device or emulator, what should I do?

You're probably using a computer to work on your Arduino project. That computer is orders of magnitudes faster than the microcontroller. Write the tests to build and run on your computer.

Remember, the behavior of the Arduino library and microcontroller should be assumed to be either correct or at least consistently incorrect.

When your tests produce output contrary to your expectations, then you likely have a flaw in your code that was tested. If your test output matches your expectations, but the program does not behave correctly when you upload it to the Arduino, then you know that your tests were based on incorrect assumptions and you likely have a flawed test. In either case, you will have been given real insights on what your next code changes should be. The quality of your feedback is improved from "something is broken" to "this specific code is broken".

How to Build and Run Tests on Your PC

The first thing you need to do is identify your testing goals. Think about what parts of your own code you want to test and then make sure to construct your program in such a way that you can isolate discrete parts for testing.

If the parts that you want to test call any Arduino functions, you will need to provide mock-up replacements in your test program. This is much less work than it seems. Your mock-ups don't have to actually do anything but providing predictable input and output for your tests.

Any of your own code that you intend to test needs to exist in source files other than the .pde sketch. Don't worry, your sketch will still compile even with some source code outside of the sketch. When you really get down to it, little more than your program's normal entry point should be defined in the sketch file.

All that remains is to write the actual tests and then compile it using your favorite C++ compiler! This is probably best illustrated with a real world example.

An actual working example

One of my pet projects found here has some simple tests that run on the PC. For this answer submission, I'll just go over how I mocked-up some of Arduino library functions and the tests I wrote to test those mock-ups. This is not contrary to what I said before about not testing other people's code because I was the one who wrote the mock-ups. I wanted to be very certain that my mock-ups were correct.

Source of mock_arduino.cpp, which contains code that duplicates some support functionality provided by the Arduino library:

#include <sys/timeb.h>
#include "mock_arduino.h"

timeb t_start;
unsigned long millis() {
  timeb t_now;
  ftime(&t_now);
  return (t_now.time  - t_start.time) * 1000 + (t_now.millitm - t_start.millitm);
}

void delay( unsigned long ms ) {
  unsigned long start = millis();
  while(millis() - start < ms){}
}

void initialize_mock_arduino() {
  ftime(&t_start);
}

I use the following mock-up to produce readable output when my code writes binary data to the hardware serial device.

fake_serial.h

#include <iostream>

class FakeSerial {
public:
  void begin(unsigned long);
  void end();
  size_t write(const unsigned char*, size_t);
};

extern FakeSerial Serial;

fake_serial.cpp

#include <cstring>
#include <iostream>
#include <iomanip>

#include "fake_serial.h"

void FakeSerial::begin(unsigned long speed) {
  return;
}

void FakeSerial::end() {
  return;
}

size_t FakeSerial::write( const unsigned char buf[], size_t size ) {
  using namespace std;
  ios_base::fmtflags oldFlags = cout.flags();
  streamsize oldPrec = cout.precision();
  char oldFill = cout.fill();

  cout << "Serial::write: ";
  cout << internal << setfill('0');

  for( unsigned int i = 0; i < size; i++ ){
    cout << setw(2) << hex << (unsigned int)buf[i] << " ";
  }
  cout << endl;

  cout.flags(oldFlags);
  cout.precision(oldPrec);
  cout.fill(oldFill);

  return size;
}

FakeSerial Serial;

and finally, the actual test program:

#include "mock_arduino.h"

using namespace std;

void millis_test() {
  unsigned long start = millis();
  cout << "millis() test start: " << start << endl;
  while( millis() - start < 10000 ) {
    cout << millis() << endl;
    sleep(1);
  }
  unsigned long end = millis();
  cout << "End of test - duration: " << end - start << "ms" << endl;
}

void delay_test() {
  unsigned long start = millis();
  cout << "delay() test start: " << start << endl;
  while( millis() - start < 10000 ) {
    cout << millis() << endl;
    delay(250);
  }
  unsigned long end = millis();
  cout << "End of test - duration: " << end - start << "ms" << endl;
}

void run_tests() {
  millis_test();
  delay_test();
}

int main(int argc, char **argv){
  initialize_mock_arduino();
  run_tests();
}

This post is long enough, so please refer to my project on GitHub to see some more test cases in action. I keep my works-in-progress in branches other than master, so check those branches for extra tests, too.

I chose to write my own lightweight test routines, but more robust unit-test frameworks like CppUnit are also available.