My baby is growing up. Just five months after launch, DoltLite is Beta, version 0.50.0.
DoltLite started as a lark. I needed a pet project to test with Steve Yegge’s innovative agent orchestrator, Gas Town. I wanted a real problem, not a toy. We wanted an embedded version of Dolt for years but rewriting Dolt’s storage engine in C or Rust was a bridge too far. SQLite seemed like a logical host for that engine. Could a team of agents pull it off? It only took about 2,000 pull requests but DoltLite going Beta proves a team of agents certainly could.
This article explains what DoltLite is and what a Beta launch means.
What Is DoltLite
DoltLite is a fork of SQLite. Everything above the B-tree layer is the same. That means the SQL parser and analyzer, the file system interaction layer, and test harness are stock SQLite. With other Dolt products we had to implement a SQL engine on top of version-controlled storage. Building a SQL engine is hard. With DoltLite we got one for the cost of writing a version-controlled storage engine in C.
The B-tree layer is swapped out for a Prolly Tree backed by a single file chunk store. Prolly Trees are content-addressed B-trees. This magical data structure powers the version control functionality in all Dolt products.
That means you get all the version control features of Dolt and Git in a SQLite package. Have you ever wanted to branch, merge, and diff SQLite? DoltLite is for you. Or maybe even more importantly, have you ever wanted a conflict aware SQLite sync engine powered by Git-style push, pull, clone, and fetch? DoltLite is for you. You can even use DoltHub as your sync backend.
What Does Beta Mean?
So what does DoltLite going Beta mean for you? Beta means four things:
- Storage Format Stability
- SQL Compatibility
- Full Version Control
- Production Performance
Reports from the field on functionality and stability are universally positive. SQLite’s testing battery is truly impressive. DoltLite passes those tests as well as a custom suite of Dolt oracle tests. DoltLite is ready for you to try.
Storage Format Stability
The biggest complaint about the development process for DoltLite was the storage format bumps. Storage format changes were not backward-compatible, requiring users to stay on the same version or manually dump and reimport their databases. It took 12 format changes to get to Beta. We’ve been on the current format for 57 releases, or over three months of calendar time. This format seems like one we can stick with.
The storage format is now stable. Any future breaking changes will have a supported migration path. DoltLite Beta means you can adopt without fear of a breaking storage format change.
SQL Compatibility
SQLite’s SQL layer is extremely well tested. DoltLite passes the vast majority of these tests as well, proving SQL compatibility.
DoltLite passes 100% of sqllogictest, a suite of 5.8M complex queries. The query layer is the same as SQLite so this is expected.
SQLite also ships with a suite of 892,277 TCL-based acceptance tests that test a broader surface of the SQLite API. DoltLite passes 99.46% with 4,809 known divergences. Each divergence must have a listed reason. The top reasons are:
- Tables are keyed by primary key, instead of
rowid, to support version control functionality. Some tests directly inspectrowid. - DoltLite has chunks, not pages. Some tests directly inspect pages.
- There is no WAL or journal sidecar. WAL and journal tests are skipped.
SQLite users will feel right at home with DoltLite. DoltLite operates like SQLite with extra version control functions.
Full Version Control
DoltLite implements the core suite of Dolt’s Git-style version control features.
Local version control features are supported. Branches, merges, diffs, rebases, cherry-picks, and resets, to name a few. Remote version control is supported: push, pull, clone, and fetch from a custom remote or DoltHub.
You even have the full Dolt Workbench GUI complete with agent mode. Unleash an agent on your SQLite and use dolt_reset('--hard') if it screws something up.
Production Performance
DoltLite gives you microsecond-scale embedded database performance but with a version control write performance tax. Reads are close to parity.
Each night, a performance report comparing DoltLite to SQLite on a standard sysbench-style benchmark is published on GitHub. Last night’s report shows in-memory DoltLite databases 10% slower on reads and 60% slower on writes. File-backed databases are at parity on reads and 10% slower on batched writes.
The big outlier is small autocommit writes which are 3.1X slower than SQLite. Performance sensitive DoltLite workflows should leverage batched writes as much as possible. Even for autocommit writes, we’re talking microsecond (i.e. sub-millisecond) individual write performance: ~125 microseconds in SQLite vs ~400 microseconds in DoltLite on a tiny GitHub runner.
Conclusion
DoltLite is Beta! All we’re waiting for now is users. Try DoltLite today for your embedded Dolt use cases. If you need any help, come by our Discord. Meet me in the #doltlite🪶 channel.
Tim's Weekly DoltHub Update
Stay in the loop and join the community on Discord
