Redis Semantics Without Redis: An Embedded State Engine for Counters, Leases and TTLs
A command-line tool should run only one copy of its nightly export at a time. A small web app wants to cap each user at 60 requests a minute. A desktop app needs to remember when it last synced. All three are state: an integer, a timestamp, an owner. All three usually get solved by adding Redis, because Redis has the verbs for them: INCR, SET NX PX, EXPIRE. The app needed the verbs and got a server along with them, a port and a persistence config included.
The idea worth building is those verbs as an in-process library over one crash-safe file: get, set, del, incr, cas, expire, lease and a transaction. No network protocol, no cluster, no admin. It’s for apps on one machine; state shared across a fleet is what Redis is for, and this doesn’t replace it. Leaving things out is the point, the same argument the SQLite and nginx pattern makes as product strategy.
Embedded Stores Give You Bytes, Not Verbs
The embedded key-value stores are mature: LMDB, bbolt, BadgerDB, RocksDB, redb. They give you ordered bytes and transactions. Counters, compare-and-set and leases are code you write on top, and every team writes it a little differently, with its own bugs. Redka, by Anton Zhiyanov, reimplements the Redis API on SQLite, which shows the mapping works. Matching an existing command set is one project. Choosing a small one and getting every edge right is another.
The real competitor is a developer with SQLite and an afternoon, and that’s where most embedded state ends up: a kv table and three statements. A library wins only by owning the edge cases, and the Redis recipes are full of them. Take rate limiting, which has a post of its own on apicoding.com. It’s a counter with a TTL, and the classic Redis recipe does INCR and then EXPIRE as two commands, so a crash between them leaves a counter that never expires. One verb that sets the TTL when it creates the key closes that hole by design.
A Dozen Verbs and a Fencing Token
Storage is plain SQLite in WAL mode, which buys crash safety, multi-process locking and a file you can open in the sqlite3 shell to see what the app believes. One kv table holds key, value and an expiry deadline in unix milliseconds, with a partial index on the keys that have one. That’s the old idea of SQL and key-value over the same bytes: verbs for the app, a SQL prompt for whoever is debugging it at 2 a.m.
Counters are SQLite integers, and incr is one upsert with a typeof(value) = 'integer' guard, so incrementing a key that holds text fails the way Redis does instead of quietly coercing it. Expired keys count as absent, so incr on one starts over. cas(key, expected, new) is an UPDATE with WHERE value = expected that reports whether a row changed. Multi-key transactions map to BEGIN IMMEDIATE, which gives you real rollback, something MULTI/EXEC in Redis never offered.
Leases are where the engine earns its keep. lease(name, owner, ttl) returns a token or nothing, and the token comes from a counter stored with the lease that goes up on every grant and never on renewal. Martin Kleppmann’s 2016 essay “How to do distributed locking” explained why. A process holding a lease can pause (a garbage collection, a stalled disk, a container throttled by its CPU limit), wake up after the lease expired and another holder took over, and write anyway. The lease can’t prevent that, because the paused process doesn’t know it paused. A monotonically increasing token, checked by the resource being written, can: the resource remembers the highest token it has seen and rejects anything lower.
db = State.open("app.state") # one file, no server
n = db.incr("rl:u42:09:41", ttl=90) # ttl applies only when the key is created
db.set("session:ab12", blob, ttl=1800)
ok = db.cas("config:rev", expected=7, new=8) # True only if it was 7
lease = db.lease("nightly-export", owner="worker-3", ttl=30)
if lease: # None while someone else holds it
reports.execute(
"UPDATE reports SET body = ?, fence = ? WHERE id = ? AND fence <= ?",
(body, lease.token, report_id, lease.token),
)
The resource side is the fence <= ? clause: once a newer holder has written, a stale holder’s UPDATE matches zero rows. If the resource is a third-party API that can’t check tokens, the lease only stops duplicate work most of the time, and the docs should say so. Release sets the deadline to zero instead of deleting the row, because deleting it would reset the counter and the next holder could get a lower token than a stale one.
The Hard Part Is Time
Leases need monotonic time while the process runs, because a wall-clock step shouldn’t shorten or stretch a lease already granted. A deadline written into the file has to be wall time, since a monotonic clock means nothing after a reboot. So every restart is a risk. If the clock stepped back while the app was down, leases and TTLs outlive their promise; if it jumped forward, they die early. Two rules limit the damage. Store a high-water mark of the latest time the engine has seen, compute now at open as the larger of wall time and that mark, then advance it with the monotonic clock. And treat the token as the safety mechanism and the TTL as a liveness mechanism, so when the clocks lie the stale write still gets rejected.
Durability has one sharp edge. In WAL mode with synchronous=NORMAL, a power loss can roll back the most recent commits. For a counter that costs a few increments. For a lease grant it’s worse: the holder already used token 41 at the resource, the grant is rolled back, and the next holder is handed 41 again, so two holders share a token and the check passes for both. Lease grants have to sync on every commit, even when the rest of the file runs looser. Run them on a connection set to synchronous=FULL and let counters and caches use group commit.
Several processes can share the file, because SQLite’s locking coordinates them on one machine, but all writers queue behind one lock. A hot counter that eight workers bump on every request becomes a line of writers waiting their turn. The fix is a separate, clearly labeled verb that buffers increments in memory and flushes them every few hundred milliseconds, with a documented loss window on a crash. Hiding that trade inside incr would be a lie; offering it as incr_buffered is a choice. It suits page-view counts. A rate limit needs the exact version.
Expiry is lazy and swept. Every read checks the deadline, so a key vanishes the instant it expires. A sweep deletes expired rows in small batches to reclaim space, because one giant delete would hold the write lock. Redis does both as well, lazily on access and with a periodic active sweep. The wrinkle in a library is that there’s no daemon to run the sweep, so it runs on a background thread in each process that has the file open, plus opportunistically after writes.
The last hard part is the temptation to add networking. Someone will ask whether two machines can share the file, then whether a Python script can reach the Go service’s state, and each yes adds a wire protocol and an auth story. Then you’ve rebuilt Redis, badly. The honest answer is no, and that Redis is the right answer for that case.
What Version 0.1 Refuses
Version 0.1 is a library for one language with get, set, del, incr, decr, cas, expire, ttl, lease, renew, release and transactions, on SQLite, plus a CLI that dumps keys, shows live leases with their tokens, and sweeps on demand. Lists stay out. If you want a queue you want leases, retries and dead letters, which is the job queue post and not push and pop. Pub/sub stays out because pushing messages to readers is the job of an embedded event log. Also out are a network protocol, clustering and Lua scripting. The lease verb is the reason scripting isn’t needed: in Redis you’d assemble a token-issuing lock from SET NX PX and a counter, atomically, which means a script.
If it ever needs a network port, use Redis.