There's a specific kind of bug that has ruined more of my evenings than I'd like to admit.
You know the one. It shows up in production exactly once. Something gets charged twice, or a record quietly disappears, or a queue just stops moving. You go digging and find nothing conclusive, and the worst part is you can't make it happen again. You run the test suite forty times. Green, green, green. So you add a log line "just in case," close the ticket with a vague comment, and wait for it to come back. It always comes back.
For years I assumed that was just the deal. Some bugs you fix, some bugs you learn to live with. Then I fell down a rabbit hole reading about how FoundationDB was built, and it genuinely changed how I think about testing. Those people looked at "you can't reproduce it" and decided that was not an acceptable answer. What they did about it is kind of unreal.
Why you can't reproduce it
A test works because it's predictable. Same input, same output. You're basically doing controlled science.
A real backend is not controlled science. It's a street fight. Threads run in a slightly different order every time. The network adds a few milliseconds here, drops a packet there. A disk write gets cut in half by a crash. Two servers can't agree on what time it is. A retry fires twice because an acknowledgment got lost on the way back.
Any one of those on its own is fine. The bug only shows up when five or six of them happen to line up in exactly the wrong order at exactly the wrong moment. And the odds of that same alignment happening again, on your machine, during a test, while you happen to be watching, are basically zero. You're not debugging at that point. You're buying a lottery ticket and hoping to lose.
The idea I couldn't stop thinking about
Here's what FoundationDB did, and what TigerBeetle and a company called Antithesis have pushed even further since.
They took every source of randomness in the system. The network, the disk, the clock, the thread scheduler, every single call to random(). And they replaced all of it. Not the application code. The world around it. The network becomes a fake network. The disk becomes a fake disk. The clock becomes a fake clock. The whole thing runs single-threaded, and all of it is driven by one random seed.
Your actual production code runs on top, untouched. It thinks it's talking to real hardware. It's really talking to a simulation that a single number completely describes.
The clock is my favorite part of this. In the simulation, time isn't a real clock on the wall. It's just a counter the simulator owns. So when your code says "wait 30 seconds," the simulator doesn't actually sit there for 30 seconds. It bumps the number and keeps going. You can run what your system experiences as years of uptime in the time it takes to make coffee.
And then you get mean
Once you own the world, you can be awful to it on purpose, which is really the whole point. Drop a third of the packets. Split the network down the middle so the two halves can't talk. Corrupt a block on disk. Freeze a server for what it believes is three days. Reorder every message in flight. Kill a node halfway through a write and bring it back up. Then do all of that at once, thousands of times a second, each run shaped by a different seed.
Switching between the real world and the fake one isn't even that dramatic in the code:
// production
let now = SystemClock::now();
let sock = RealNetwork::connect(addr);
// simulation: same path, deterministic guts
let now = sim.clock(); // a counter the sim controls
let sock = sim.network(addr); // can drop, delay, reorder
Same path through your program. One version uses the real clock and a real socket. The other uses ones the test harness is free to mess with.
The part that actually matters
When the simulator finally trips the impossible bug, it hands you exactly one thing: the seed.
Put that seed back in and the entire run replays, identical. The same packets dropped in the same order, the same crash at the same microsecond, the same failure, every single time you run it. The bug that would've taken ten years to surface in production is now sitting in a debugger while you eat lunch.
FoundationDB ran this for years before they shipped anything at all. The story that always gets told: when Kyle Kingsbury, the guy whose entire job is breaking databases with a tool called Jepsen, came around looking for problems, he couldn't really find any worth writing up. They'd already simulated every disaster he'd reach for, plus a pile he never would.
What I actually took from it
I don't work on databases. Odds are you don't either. But this scales all the way down to the boring CRUD backends most of us actually ship, and it quietly changed how I write them.
Stop calling now() and the network directly. Pass them in. The moment your core logic takes the clock and the I/O as arguments instead of reaching out and grabbing them itself, you can fake them, fast-forward them, break them on purpose, and replay them. A function that's handed its inputs is testable like math. One that goes and fetches the real world can only be prayed over.
And the bigger thing I keep coming back to: most of the pain of the 3am bug was never the fix. It's the reproducing. If you can buy back the ability to make it happen on demand, you've bought back the part that actually hurt.
Those teams didn't outsmart the bug. They just took ownership of the tiny universe it lived in, and rewound time until it admitted what it did.