How TaskFused Was Tested With Killed Workers and Injected Storage Faults

TaskFused makes one promise above the others: work that two requests share is done once and kept safe, even when the process doing it dies. That’s easy to claim and easy to get wrong, so most of the tests try to break it. They kill real processes, make the storage fail on purpose, and drop connections at awkward moments.

Every result below comes from the files in test/results, recorded from version 0.1.0 on a two-core Linux cloud machine with Go 1.24 and SQLite 3.50.4, with every change flushed to disk. The same tests run on every push, on Linux and macOS.

Killing Workers

The crash tests don’t fake a crash. A worker is a real child process. It claims a job, confirms that its claim is saved, and then the test kills it outright, with no chance to clean up. A new worker starts on the same file.

A timeline of the crash test: worker one claims the job with token 1 and is killed, the lease runs out, worker two claims the same operation with token 2 and finishes it, and a late result from worker one is rejected.

In the recorded demo, the new worker took the job over once the one-second lease ran out and finished it about 1.1 seconds after the kill. The operation kept its ID. Its history shows two attempts: the first marked expired, the second with a higher token. Then the test does the thing a confused or delayed worker might do and tries to save a result with the old token. TaskFused turns it away.

The same test runs over the HTTP API, with every request going through a real local server and the workers running as separate child processes. It gives the same result, about 1.1 seconds from kill to finish.

One Hundred Requests, One Run

100 requests for the same work went in at once. One created the operation and 99 joined it. The worker ran it one time. 99 of the waiters got the result. The 100th had been given a deliberately short timeout and gave up, and the test checks that its giving up didn’t cancel the work for everyone else.

Breaking the Storage

The tests pick 13 kinds of write to the database: submitting, claiming, taking over an expired claim, running out of attempts, renewing a lease, completing, failing, scheduling a retry, cancelling, an operator retry, pruning results, and setting and changing limits. For each one, the tests make the write fail twice: once in the middle of a statement, and once at the moment of commit. That’s 26 forced failures. After each one, the test compares every table and counter with a snapshot taken before. Nothing may have changed. Then it removes the fault and checks the same action now works.

Other tests fill the database to its size limit, open it read-only, and hold a lock from a second connection until the wait times out. Each time TaskFused returns an error and leaves the file consistent.

Lost Replies

Over HTTP, the worst moment for a connection to drop is just after the server has saved something and before the client hears about it. The tests cut the connection at exactly that point for a submission and for an operator retry. The client then sends the same request again. The submission joins the existing operation, the retry returns its first answer, and no extra work or extra attempts appear.

Speed

TaskFused is about doing work once. It still shouldn’t be slow. On that two-core machine, with a 128-byte input and result and each change flushed to disk:

1,000 submissions, one after another, took 0.59 seconds; the median was 0.56 ms each.
Running 1,000 trivial jobs took 1.5 seconds; the median was 1.4 ms each.
100 callers waiting on one unfinished operation for a second wrote nothing to the file; they share one poll of the database.

These are measurements of one machine, so treat them as a rough guide only.

Every Release, Every System

Before each release goes out, the release workflow runs a smoke test on the binaries on their own kind of machine: Linux on x86-64 and ARM, an Apple silicon Mac, and Windows. The Intel Mac build is only checked for its architecture, because the Macs that run the tests are Apple silicon. It submits and joins work, runs it with the real worker, sets limits, prunes, serves the HTTP API, stops the server and worker cleanly, and runs the HTTP demo, which kills a real worker. If any of that fails anywhere, nothing is published.

What Isn’t Tested

The storage tests make SQLite report failures. They don’t pull the power or break a real disk, so durability through a power cut rests on SQLite and on the disk honouring its flushes. Version 0.1 is tested on one host only, because that’s all it supports. The status page’s HTML is checked by tests, but nobody has yet looked at it in every browser.

All of it is in the repository: the tests, the fault notes and the recorded results.