After rewriting SQLite in Rust, Turso turns its sights on Postgres

Having built a SQLite-compatible database from scratch in Rust, cloud database provider Turso is attempting something more ambitious: a Postgres-compatible implementation on the same foundation.

"I don't think there is anything fundamentally wrong with Postgres. If there was something wrong with Postgres, we wouldn't rewrite it," Turso CEO Glauber Costa told The Register. But, like SQLite, Postgres could be freshened up for the cloud-native era.

Turso has grander ambitions than putting another notch on the Rustbelt. Its SQLite reimplementation, originally codenamed Limbo and later renamed Turso, uses a virtual machine architecture that the company believes can support multiple database frontends.

This month, Turso-the-company launched the Postgres effort and suggested that the same architecture could eventually support frontends for systems such as MySQL and Redis. "Turso is becoming the LLVM of databases. One modern, reliable core; many database frontends compiled down onto it," Costa wrote, announcing the release.

From fork to rewrite

Costa and fellow co-founder Pekka Enberg got the idea for Turso while studying the architecture of SQLite itself. Both had just left technical engineering jobs at ScyllaDB to start a San Francisco startup called ChiselStrike. They needed an online database, and SQLite was an obvious choice, but it needed updating to work as a cloud service. 

Software developer D. Richard Hipp created SQLite in 2000 as a compact, standalone transactional SQL engine. Hipp has steadfastly kept the scope of SQLite minimal, refusing outside contributors or bloating it with additional features (much like Andrew Tanenbaum kept tight control on Minix, inspiring a young Linus Torvalds to create a much more ambitious Linux).

ChiselStrike forked SQLite as libSQL to give the company's developers the ability to modify the database system. It also provided a much-requested avenue for outside contributors to add their own features. 

At first, contributors added features without making many changes to SQLite's core – not exactly a weekend job, even for an ambitious hacker. More fundamental work, such as introducing a richer type system, would require ChiselStrike to tackle the engine itself. The company therefore launched a complete rewrite, initially codenamed Limbo.

By then, ChiselStrike had already pivoted to a SQLite-based cloud service and renamed itself Turso. Limbo later became an official company project, adopted the Turso name – after the mythical Finnish sea creature Iku-Turso – and became the planned foundation for its future database engines.

Behind the SQL

While rewriting SQLite, the duo focused on its virtual machine architecture and concluded that the same approach could be expanded into a more general database execution engine. The software compiles SQL queries into its own bytecode language, called the Virtual Database Engine (VDBE).

They built Turso using the same virtual machine design, one based on VDBE. This foundation could work not only to mimic SQLite, but any database system.

"Look close enough, and every SQL database is just a fancy collection of B-Trees with a bunch of Indexes," Costa wrote. Any features that Postgres has but SQLite lacks could be written with VDBE. 

To demonstrate that their VM was sufficiently expressive, the duo built a demo that runs the 1990s classic video game DOOM in WebAssembly, with each rendered frame returned as a result row.

There are similar approaches to virtualizing the database. Google's Spanner can offer different modalities, spanning relational, graph, key-value, vector and other formats. Microsoft built the Cosmos database service on a similar premise.

Spanner and Cosmos operate higher up on the stack than Turso, Costa explained. Closer to Turso's building block style is Apache DataFusion, also built with Rust. 

"The idea of a database that can speak multiple models is nothing new," Costa said. "What is different about Turso is that it sits at the lowest possible level, exactly one layer above the storage engine."

Postgres on top, Turso under

Enberg and Costa put together a Postgres-compatible prototype on top of Turso called pgmicro. Now came the hard work of making it ready for production use. 

"Essentially we would offer a database that speaks Postgres, that plays the same role as Postgres, that is familiar to the developers in the Postgres ecosystem, but it's built in a different way underneath," Costa said.

When sufficiently mature, this Postgres-flavored Turso is intended to run most existing applications without modification, although the company is not promising complete compatibility with the OG Postgres. It also plans enhancements including automatically updated materialized views, a frequent request from the Postgres community.

One aspect of the old Postgres that may be jettisoned will be its one-process-per-connection model, which can be very expensive, computationally. "It's worth a fresh look," Costa said. 

While Postgres will require more work, it's not an order-of-magnitude more work, and the new features can be rolled back into Turso for use in other database variants down the road. 

On the commercial side, Turso is building out a database-as-a-cloud service that will offer any DB flavor the customer would want, but all running on Turso. 

Rust and AI on Turso

Rewriting open source software in the Rust programming language has become a sport, with official and unofficial rewrites underway of Kubernetes, the Bun JavaScript tool chain, and bits of Linux. Much of this work has been done with AI, which has a pretty good grasp of Rust fundamentals. 

But for Turso, Costa doesn't ascribe too much significance to the choice of the language for the rewrite.

Rust offers memory safety, yet plenty of Postgres bugs have nothing to do with memory safety. Rust works well with AI, which Turso uses quite a bit, but Costa and Enberg chose Rust for Turso before AI coding agents became widespread.

"I never understood the fascination programmers have with programming languages," he said, adding: "For us, Rust is just a tool." ®