What I learned writing an append-only log

I set out to write a storage layer that could survive being killed mid-write. The naive version — open a file, append bytes, hope — falls over the first time the process dies between the write and the flush.

Durability is a claim you have to prove

Calling write() does not mean the data is on disk. It means the kernel has it. Until fsync() returns, a power cut takes your record with it. The fix is boring: fsync at record boundaries, and accept the throughput cost.

The interesting part is what happens on recovery. A torn record at the tail is normal, not exceptional, so the reader has to treat a bad checksum at the end of the file as the end of the file rather than as corruption.

The tail is always suspect

Every record gets a length prefix and a CRC. On open, scan forward until a record fails to parse, then truncate there. It sounds lossy, and it is — but the alternative is refusing to start, which is worse.