Deterministic Testing

Deterministic Testing of Complex Systems

Timing, concurrency, and hardware interaction are the three areas where embedded tests most commonly become non-deterministic: they pass on a fast machine, fail on a slow one, depend on real elapsed time, or produce different results depending on thread scheduling. This chapter explains the patterns that eliminate that non-determinism.

Companion code lives in workshop/07_deterministic_testing/.

---

The Core Problem: Hidden Real-World Dependencies

Non-determinism always traces back to a hidden dependency on something outside the test's control.

Source of non-determinism	Hidden dependency
sleep(), usleep() in production code	Real clock
Timeout logic using std::time()	Real clock
Threads communicating via shared state	OS thread scheduler
ISR callbacks triggered by hardware	Real hardware interrupt
Polling loops that wait for hardware state	Real hardware, real time

The fix in every case is the same: inject an abstraction over the real-world dependency so the test can control it directly.

---

Section 1: Virtual Time

The wrong approach

// ❌ Test depends on real elapsed time; slow hardware causes false failures
TEST(WatchdogTest, ExpiresAfterTimeout)
{
    Watchdog wdog(500); // 500 ms timeout
    std::this_thread::sleep_for(std::chrono::milliseconds(600));
    EXPECT_TRUE(wdog.hasExpired());
}

This test takes 600 ms minimum, is flaky if the target is under load, and tells you nothing useful if it fails.

The IClock interface

// iclock.h
#pragma once
#include <cstdint>

class IClock {
public:
    virtual ~IClock() = default;
    virtual int64_t nowMs() = 0;
};

The ManualClock fake

Instead of advancing real time, the test manually sets the clock value.

// manual_clock.h
#pragma once
#include "iclock.h"

class ManualClock : public IClock {
public:
    void setTime(int64_t ms)    { now_ = ms; }
    void advanceBy(int64_t ms)  { now_ += ms; }
    int64_t nowMs() override    { return now_; }
private:
    int64_t now_ = 0;
};

Watchdog with injected clock

// watchdog.h
#pragma once
#include "iclock.h"

class Watchdog {
public:
    Watchdog(IClock& clock, int64_t timeoutMs)
        : clock_(clock), timeoutMs_(timeoutMs), startMs_(clock.nowMs()) {}

    void reset() { startMs_ = clock_.nowMs(); }

    bool hasExpired() const {
        return (clock_.nowMs() - startMs_) >= timeoutMs_;
    }

private:
    IClock&  clock_;
    int64_t  timeoutMs_;
    int64_t  startMs_;
};

Tests: instant, deterministic

TEST(WatchdogTest, DoesNotExpireBeforeTimeout)
{
    ManualClock clock;
    Watchdog    wdog(clock, 500);

    clock.advanceBy(499);
    EXPECT_FALSE(wdog.hasExpired());
}

TEST(WatchdogTest, ExpiresExactlyAtTimeout)
{
    ManualClock clock;
    Watchdog    wdog(clock, 500);

    clock.advanceBy(500);
    EXPECT_TRUE(wdog.hasExpired());
}

TEST(WatchdogTest, ResetRestartsCounting)
{
    ManualClock clock;
    Watchdog    wdog(clock, 500);

    clock.advanceBy(400);
    wdog.reset();
    clock.advanceBy(400); // total elapsed 800 ms but reset at 400
    EXPECT_FALSE(wdog.hasExpired());
}

Each test runs in microseconds and tests an exact boundary.

---

Section 2: Testing Sequences and State Machines

Embedded systems often have stateful behaviour: a sequence of inputs must arrive in order for a transition to occur. A non-deterministic test would use real time to pace inputs. A deterministic test drives inputs directly.

A connection state machine

// connection_manager.h
#pragma once
#include "iclock.h"

enum class ConnectionState { Disconnected, Connecting, Connected, Error };

class ConnectionManager {
public:
    explicit ConnectionManager(IClock& clock) : clock_(clock) {}

    void connect() {
        if (state_ == ConnectionState::Disconnected) {
            state_    = ConnectionState::Connecting;
            startMs_  = clock_.nowMs();
        }
    }

    // Call periodically to drive the state machine
    void poll() {
        if (state_ == ConnectionState::Connecting) {
            if (clock_.nowMs() - startMs_ > timeoutMs_) {
                state_ = ConnectionState::Error;
            }
        }
    }

    void onConnected()    { state_ = ConnectionState::Connected;   }
    void onDisconnected() { state_ = ConnectionState::Disconnected; }

    ConnectionState state() const { return state_; }

private:
    IClock&         clock_;
    ConnectionState state_    = ConnectionState::Disconnected;
    int64_t         startMs_  = 0;
    int64_t         timeoutMs_ = 1000;
};

Sequence tests

TEST(ConnectionManagerTest, TransitionsToConnectingAfterConnect)
{
    ManualClock clock;
    ConnectionManager cm(clock);

    cm.connect();

    EXPECT_EQ(cm.state(), ConnectionState::Connecting);
}

TEST(ConnectionManagerTest, TransitionsToConnectedOnCallback)
{
    ManualClock clock;
    ConnectionManager cm(clock);

    cm.connect();
    cm.onConnected();

    EXPECT_EQ(cm.state(), ConnectionState::Connected);
}

TEST(ConnectionManagerTest, TransitionsToErrorOnConnectionTimeout)
{
    ManualClock clock;
    ConnectionManager cm(clock);

    cm.connect();
    clock.advanceBy(1001); // past 1000 ms timeout
    cm.poll();

    EXPECT_EQ(cm.state(), ConnectionState::Error);
}

TEST(ConnectionManagerTest, NoTimeoutIfConnectedBeforeDeadline)
{
    ManualClock clock;
    ConnectionManager cm(clock);

    cm.connect();
    clock.advanceBy(500);
    cm.onConnected();
    clock.advanceBy(600); // would be past timeout if not connected
    cm.poll();

    EXPECT_EQ(cm.state(), ConnectionState::Connected);
}

The test drives the exact sequence and controls time explicitly. There are no sleep() calls and no race conditions.

---

Section 3: Concurrency

Concurrent code is hard to test for two reasons: thread scheduling is non-deterministic, and shared state can produce different results depending on interleaving order. The goal is to test the logic of concurrent components without relying on real threads.

Separate the concurrency mechanism from the business logic

The most effective technique is to keep threading infrastructure out of business logic classes entirely. The business logic runs on whatever thread calls it. The threading mechanism (a thread pool, a worker thread, a Qt event loop) is injected or tested separately.

// ❌ Threading baked in: untestable, non-deterministic
class DataProcessor {
public:
    DataProcessor() {
        worker_ = std::thread([this]() { run(); }); // thread started in constructor
    }
    void submit(const Data& d) { queue_.push(d); }
private:
    void run() { while(true) { process(queue_.pop()); } }
    std::thread worker_;
    BlockingQueue<Data> queue_;
};

// ✅ Threading extracted: logic testable synchronously
class DataProcessor {
public:
    // Business logic: callable from any thread, or directly from a test
    void process(const Data& d) {
        calibrated_ = d.raw + offset_;
        output_.write(calibrated_);
    }
private:
    float    offset_    = 0.0f;
    IOutput& output_;
};

// The threading wrapper is a thin shell: it only manages the thread and queue
class AsyncDataProcessor {
public:
    explicit AsyncDataProcessor(DataProcessor& processor) : processor_(processor) {
        worker_ = std::thread([this]() { run(); });
    }
    void submit(const Data& d) { queue_.push(d); }
private:
    void run() { while(true) { processor_.process(queue_.pop()); } }
    DataProcessor& processor_; // injected
    std::thread    worker_;
    BlockingQueue<Data> queue_;
};

Testing DataProcessor is now synchronous and deterministic: call process() directly, assert on the output. The AsyncDataProcessor wrapper can be integration-tested separately with a short real-time test.

Testing shared state: use a controlled executor

When the business logic must interact with shared state, use a fake executor that runs tasks synchronously in the test rather than on a real thread.

class IExecutor {
public:
    virtual ~IExecutor() = default;
    virtual void post(std::function<void()> task) = 0;
};

// Production: posts to a real thread pool or event loop
class ThreadPoolExecutor : public IExecutor {
public:
    void post(std::function<void()> task) override { pool_.submit(task); }
private:
    ThreadPool pool_;
};

// Test: runs tasks immediately on the calling thread, in order
class SynchronousExecutor : public IExecutor {
public:
    void post(std::function<void()> task) override { task(); }
};

With SynchronousExecutor, tasks that would normally run asynchronously run inline in the test. The sequence is predictable and the test is deterministic.

---

Section 4: Hardware Interaction

Hardware interaction tests face three challenges: the hardware may not be present, its state is not under test control, and it may take unpredictable time to respond.

Pattern 1: Inject an ISR dispatcher

Interrupt service routines (ISRs) cannot be called from tests directly. Abstract the interrupt as a callback that the hardware adapter invokes, and invoke the same callback directly from the test.

// iinterrupt_source.h
class IInterruptSource {
public:
    virtual ~IInterruptSource() = default;
    using Callback = std::function<void()>;
    virtual void registerCallback(Callback cb) = 0;
};

// Production: registers with the real IRQ
class GpioInterruptSource : public IInterruptSource {
public:
    void registerCallback(Callback cb) override {
        callback_ = cb;
        gpio_irq_register(GPIO_PIN_IRQ, &GpioInterruptSource::isr, this);
    }
    static void isr(void* ctx) {
        static_cast<GpioInterruptSource*>(ctx)->callback_();
    }
private:
    Callback callback_;
};

// Test: stores the callback so the test can fire it manually
class FakeInterruptSource : public IInterruptSource {
public:
    void registerCallback(Callback cb) override { callback_ = cb; }
    void fireInterrupt() { if (callback_) callback_(); } // called from test
private:
    Callback callback_;
};

// Testing interrupt-driven logic without real hardware
TEST(ButtonHandlerTest, SinglePressIncrementsCounter)
{
    FakeInterruptSource irq;
    ButtonHandler       handler(irq);

    irq.fireInterrupt(); // simulates the GPIO interrupt firing

    EXPECT_EQ(handler.pressCount(), 1);
}

TEST(ButtonHandlerTest, TwoPressesIncrementCounterTwice)
{
    FakeInterruptSource irq;
    ButtonHandler       handler(irq);

    irq.fireInterrupt();
    irq.fireInterrupt();

    EXPECT_EQ(handler.pressCount(), 2);
}

Pattern 2: Abstract hardware polling

Polling loops that wait for a hardware register to change must never appear in unit-testable business logic. Abstract the hardware state as a readable value.

// ❌ Polling loop in business logic: blocks, non-deterministic
bool waitForReady() {
    while (!(gpio_read(GPIO_PIN_READY) & 0x01)) { /* spin */ }
    return true;
}

// ✅ Abstract the hardware state; the test controls the returned value
class IGpioPin {
public:
    virtual ~IGpioPin() = default;
    virtual bool isHigh() = 0;
};

class StubGpioPin : public IGpioPin {
public:
    void setState(bool high) { high_ = high; }
    bool isHigh() override { return high_; }
private:
    bool high_ = false;
};

// Business logic: no polling, just reads the abstracted state
bool isDeviceReady(IGpioPin& readyPin) {
    return readyPin.isHigh();
}

Pattern 3: Simulate hardware response sequences

Some hardware responds to commands with a sequence of state changes. Use a fake that returns a pre-programmed sequence of values.

class SequencedSensor : public IFlowSensor {
public:
    void addReading(float value) { readings_.push_back(value); }

    float readLitresPerMinute() override {
        if (index_ < readings_.size()) {
            return readings_[index_++];
        }
        return readings_.back(); // hold last value
    }

private:
    std::vector<float> readings_;
    std::size_t        index_ = 0;
};

TEST(FlowControllerTest, OpensValveWhenReadingsRiseAboveThreshold)
{
    SequencedSensor sensor;
    sensor.addReading(5.0f);   // first call: below threshold
    sensor.addReading(5.0f);   // second call: still below
    sensor.addReading(12.0f);  // third call: above threshold

    SpyValve   valve;
    SpyLogger  logger;
    FlowController controller(sensor, valve, logger);

    controller.update(); EXPECT_EQ(valve.lastAction(), "close");
    controller.update(); EXPECT_EQ(valve.lastAction(), "close");
    controller.update(); EXPECT_EQ(valve.lastAction(), "open");
}

---

Summary of Patterns

Problem	Pattern	Key class
Timeout logic uses real clock	Virtual time	ManualClock
State machine paced by real time	Manual clock + direct event injection	ManualClock
Business logic runs on a thread	Separate logic from threading	SynchronousExecutor
ISR fires real hardware interrupt	Fake interrupt source	FakeInterruptSource
Polling loop waits for hardware pin	Abstract pin state	StubGpioPin
Hardware returns a sequence of values	Sequenced fake	SequencedSensor

---

What Belongs in Integration Tests

Some scenarios genuinely require real hardware and cannot be made deterministic in a unit test. Tag these clearly and run them separately.

• Real timing precision: verifying that a hardware watchdog fires within a specific window
• Actual ISR latency: measuring interrupt response time on the target
• Bus transactions: verifying SPI or I2C framing at the electrical level
• DMA behaviour: verifying that DMA transfers complete correctly under load

These are integration tests. They run on the target, may be slow, and are expected to be few in number. Everything else should be unit-testable with the patterns above.

---

See also: Test Architecture Patterns, Refactoring Toward Testability, Module 03: Hardware Dependencies