How embedded-test works internally

embedded-test 0.7.0 has just been released and it brings support for unit tests, apart from the integration-test support we had for a while now. This version required me to refactor how the crate works under the hood. In this post I'm going to shed some light on the internals of embedded-test.

Recap: What is embedded-test?

embedded-test allows you to run your test cases on a bare-metal no_std microcontroller, while reporting the test results back to your IDE, just as if you were running the test on your host.

In order to make this work, you need to setup a custom runner for your project (probe-rs), add a custom linker file to your build script, and annotate your test modules with

#[embedded_test::tests].

Here is an example test:

pub fn factorial(n: u32) -> u32 { /*...*/ }

#[cfg(test)]
#[embedded_test::tests]  // <-- note this additional attribute!
mod tests {
    use super::*;

    #[test]
    fn test_factorial() {
         assert_eq!(factorial(5), 120); // Typical case
    }

    #[test]
    #[should_panic]
    fn test_factorial_overflow() {
        factorial(35); // This will overflow for `u32`
    }
}

If you invoke cargo test, the target runs your tests and you will get an output like the following:

$ cargo test

  Erasing ✔ 100% [####################] 192.00 KiB @ 556.44 KiB/s 
  Programming ✔ 100% [####################]  48.99 KiB @ 59.70KiB/s 
Finished in 0.82s
running 2 tests
test tests::test_factorial          ... ok
test tests::test_factorial_overflow ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.47s

As with rust's built-in test-framework you can filter the tests or generate JSON output (for seamless IDE, CI/CD integration). embedded-test also supports async test cases (similar to tokio::test), test case timeouts, init functions / state passing... You can find more info about how to set it up, in my previous blog post.

Host <-> Target communication

So how does this magic work?

Cargo allows configuring a "runner", which will then be invoked whenever you want to run some executable (normal binary, but also tests). We can use that runner to flash the test cases, run them on the target, and provide output which is compatible with what a normal rust test binary would output.

For embedded-test, "probe-rs run" needs to be setup as the runner. "cargo run" and "cargo test" calls will therefore invoke probe-rs run <ELF>. Probe-rs will look for the presence of the EMBEDDED_TEST_VERSION symbol in the ELF, to determine whether the binary in question contains embedded-test test-cases or not. In case it does, probe-rs will enter a special test mode, instead of flashing + running it normally (monitor mode).

probe-rs uses semihosting to communicate with the target. After flashing the test binary in one go, probe-rs resets the target and the target will run its main function. We'll talk about that main function later, but the first thing it does is issuing a semihosting call SYS_GET_CMDLINE. probe-rs will answer this call with the address of the first test to run. The target will then run the test in question, while probe-rs collects RTT log messages. The test exit/failure is reported via another semihosting call SYS_EXIT_EXTENDED and a corresponding exit code. After the test has ended, probe-rs will reset the target and pass the address of the next test case when asked.

Sequence Diagram of how probe-rs runs a test suite on a target device

In order to provide compatible output, probe-rs uses libtest-mimic. libtest-mimic takes care of test filtering and can output the test results in the various formats (compact, json, ...).

Macro Expansion of the tests Module

To better understand, how the tests are set up, and how the target signals ok/failure to the host, we have to look at the macro expansion of the tests macro.

Consider the following test code:

#[embedded_test::tests]
mod tests {
    struct Context {
        i2c0: esp_hal::peripherals::I2C0<'static>,
    }

    #[init]
    fn init() -> Context {
        let peripherals = esp_hal::init(/*...*/);
        Context {
            i2c0: peripherals.I2C0,
        }
    }

    #[test]
    fn can_connect_sensor(context: Context) {
        /* ......*/
    }
}

The Context struct, the init function and the can_connect_sensor test function are stripped of their attributes, but otherwise emitted as is. Additionally, a new function is emitted, which will serve as an entry-point to run that particular test case:

#[doc(hidden)]
fn __can_connect_sensor_entrypoint() -> ! {
    let outcome;
    {
        let state = init();
        outcome = can_connect_sensor(state);
    }
    embedded_test::export::check_outcome(outcome);
}

In case the init or the test function are async, an additional embassy task is emitted, and the generated entrypoint creates and runs an embassy executor.

#[embassy_executor::task]
#[doc(hidden)]
async fn ____can_connect_sensor_invoker() {
    let outcome;
    {
        let state = init();
        outcome = can_connect_sensor(state).await;
    }
    embedded_test::export::check_outcome(outcome);
}

#[doc(hidden)]
fn __can_connect_sensor_entrypoint() -> ! {
    let mut executor = esp_hal_embassy::Executor::new();
    executor
        .run(|spawner| {
            spawner.must_spawn(__can_connect_sensor_invoker());
        })
}

The embedded-test library contains a panic handler and the check_outcome function. TestOutcome is implemented for () and for Result<T, E>.

pub fn check_outcome<T: TestOutcome>(outcome: T) -> ! {
    if outcome.is_success() {
        info!("Test exited with () or Ok(..)");
        semihosting::process::exit(0);
    } else {
        info!("Test exited with Err(..): {:?}", outcome);
        semihosting::process::abort();
    }
}

#[panic_handler]
fn panic(info: &core::panic::PanicInfo) -> ! {
    error!("====================== PANIC ======================");
    error!("{}", info);
    semihosting::process::abort()
}

So, whenever the target is asked to run a particular test, it will invoke the corresponding entry-point function. Probe-rs will then wait until it receives the SYS_EXIT_EXTENDED call with the result.

Collect the test symbols from the ELF

Ok, but how does probe-rs know, which test cases exist? And how does the target's main function invoke a test case?

In the previous version of embedded-test, each #[embedded_test::tests] invocation, would emit a main function, that would know which test cases were defined inside the module. And when the target asked the host for what to do, one of the options was to list all the test cases and return them as JSON to the host...

The problem with this approach was, that you could only have a single #[embedded_test::tests] invocation per compilation unit, making it unusable for inline unit tests scattered over the project. As a workaround we could have collected the tests with the linkme crate. But that would still require probe-rs to connect to the target and fetch the tests.

Instead, the new version of embedded-test will emit a special symbol per test, which we add to a custom linker section, so that probe-rs can read the available tests directly from the ELF symbol table.

For our test can_connect_sensor, the exported symbol looks like this:

#[link_section = ".embedded_test.tests"]
#[export_name = "{\"name\":\"can_connect_sensor\",\"ignored\":false,\"should_panic\":false}"]
static __CAN_CONNECT_SENSOR_SYM: (fn() -> !, &'static str) = (
    __can_connect_sensor_entrypoint,
    module_path!()
);

So it's a symbol pointing to the test entrypoint, but as symbol name we use a JSON object, to store some metadata (an idea borrowed from defmt). Unfortunately, there is currently no way for a proc macro to get the module path of the call-site, so we need the symbol to be a tuple that also contains module_path!().

We can inspect our ELF file with objdump -t -j .embedded_test <ELF>, and find our symbol:

SYMBOL TABLE:
00000000 g     O .embedded_test	0000000c {"name":"can_connect_sensor","ignored":false,"should_panic":false}
...
00000060 g     O .embedded_test	00000004 EMBEDDED_TEST_VERSION

The test case symbol has a size of 12 bytes (hex 0xC). 4 bytes for the function pointer, and 8 bytes for the static string (which is another pointer + a length). With this information probe-rs can now build a list of tests, without connecting to the target. If you're interested in the full decoding logic in probe-rs, take a look at ElfReader.

The main function in our target binary therefore only needs to know the address of the function to run. It looks as follows, and is part of the embedded-test library code so that it is linked only once (and not emitted for every proc-macro invocation).

fn main() -> ! {
    let args = semihosting::experimental::env::args::<1024>();
    let command = match args.next() {
        Some(c) => c.expect("command contains non-utf8 characters"),
        None => {
            error!("Received no arguments via semihosting");
            semihosting::process::abort();
        }
    };

    match command {
        "run_addr" => {
            let addr = args.next().expect("addr missing");
            let addr = addr.expect("addr contains non-utf8 char");
            let addr: usize = addr.parse().expect("invalid number");
            let invoker: fn() -> ! = unsafe{core::mem::transmute(addr)};
            test_invoker();
        }
        _ => {
            error!("Unknown command: {}", command);
            semihosting::process::abort();
        }
    }
}

Summary

embedded-test 0.7.0 improves the developer experience quite a bit: Unit tests are now supported, and thanks to the test cases being read directly from the ELF, the tests are executed faster. I've also added support to override the init function on a per-test basis (with

#[test(init=my_custom_init)]).

You can try the new version today by:

• Installing probe-rs 0.30.0

• Adding embedded-test 0.7.0 to your project.

Source Code + Examples for ESP32, STM32: https://github.com/probe-rs/embedded-test/

I would love to hear your feedback and learn from your experience with this release.