A database that pretends to be a disk: building a virtual filesystem with Dokan

Most engineers go their whole career without writing a filesystem. For a couple of years, I maintained one — a filesystem with no disk of its own. Its files lived in cloud blob storage; a SQL database decided what existed, who was allowed to see it, and where the bytes were kept; and Windows treated the whole arrangement as just another drive letter. This is the story of why we built it, how it worked, and the corner of the problem that nearly broke us: Microsoft Office.

The problem: files that were a little too free

Insight is legal practice-management software. Law firms live and die by their documents — matters, correspondence, contracts, court bundles — and ours were stored in Azure Storage accounts, surfaced on each customer’s PC as a mapped network drive.

That worked, but it gave us almost no control over the thing that mattered most: the documents themselves. Once a folder is a network drive, it’s just there. It doesn’t care whether the user is logged into our application. It doesn’t understand that some matters are protected and should be invisible to most of the firm. It can’t route a file open back through our software so we can enforce our own rules, log access, or keep the interaction inside the app where it belongs.

We wanted the opposite of a mapped drive. We wanted documents that were:

only accessible while signed into the Insight desktop app — close the app, and the files quietly disappear;
governed by our security model — protected matters stay hidden from anyone without rights to them, at the filesystem level, not just in the UI;
interacted with through our software, so every open, edit and save flowed through logic we controlled.

You can’t get that from a share drive. We needed the operating system to ask us every time it wanted to touch a file — which meant we needed to be the filesystem.

The options that didn’t survive contact

We didn’t set out to write a filesystem. That’s an absurd amount of surface area to take on, and we went looking for anything that would let us avoid it.

WebDAV — HTTP-based, familiar, and something Windows can mount natively. But the Windows WebDAV client is famously temperamental, and it didn’t give us the tight, synchronous “ask us on every operation” control we were after.
Shell extensions — hooking into Explorer to fake files. This gets you something that looks like files in one window, but it isn’t a real filesystem: applications opening a path directly don’t go through the shell, so Word and friends would never see our documents.

One by one, the shortcuts turned out to be technically unfeasible or simply the wrong shape for what we needed. Every serious option pointed back to the same conclusion: to make the OS treat our database like a disk, we had to implement a real filesystem.

The lightbulb: I’d just installed Google Drive

Around this time I’d installed Google Drive for Windows on my own machine, and I noticed something: my Drive files appeared as an ordinary drive letter, but the bytes plainly weren’t all sitting on my SSD — they were streaming down from Google on demand. Something was standing between Windows and a remote store, answering filesystem calls on the fly.

That was exactly our problem. If Google could put a network service behind a drive letter, we could do the same for ours.

I took the idea to the engineer I was working with, and our digging led us to Dokan — a user-mode filesystem library for Windows, the spiritual cousin of FUSE on Linux. Dokan lets you implement a filesystem in a normal user-mode process: it hands you callbacks for every operation the OS wants to perform, and your code decides what happens. No kernel driver of our own to write, sign, and maintain.

Learning what a filesystem actually has to do

Here’s the humbling part. Once you commit to being a filesystem, you have to implement the whole contract — not the handful of operations you imagine a file needs, but everything Windows might ask of a volume.

We tracked down a copy of Rajeev Nagar’s Windows NT File System Internals — a dense, long-out-of-print O’Reilly tome that spells out, in exhaustive detail, exactly how the OS expects a volume to behave. It became our specification. From it we knew the full set of behaviours we had to honour:

Dokan surfaces this as a set of callbacks, and every one of them had to be correct:

CreateFile — open or create, with all of Windows’ sharing and disposition flags;
ReadFile / WriteFile — at arbitrary offsets, in whatever chunk sizes the caller felt like;
GetFileInformation, FindFiles — the metadata and directory listings Explorer and every app rely on;
Cleanup and CloseFile — the surprisingly subtle end-of-life of a handle;
move, rename, delete, set-attributes, and the rest of the long tail.

Miss a detail in any of these and applications don’t throw a clean error — they behave weirdly, which is far worse to debug.

Two filesystems, one interface

There was an extra wrinkle: we couldn’t flip every customer over on day one. The original Azure-backed storage had to keep working alongside the new database-backed one, sometimes for the same firm during migration.

So we designed to an abstraction. Dokan’s callbacks sat on top of a common interface, and behind that interface we had two implementations — one talking to the legacy Azure storage, one talking to our new database-backed store. The operating system neither knew nor cared which was mounted; it just made filesystem calls, and the right backend answered.

Metadata in SQL, bytes in the cloud

None of the actual file bytes lived in the database. The documents themselves stayed in cloud blob storage — moving terabytes of legal files into SQL would have been madness. What lived in SQL was everything about those files: the folder tree, the metadata, the security rules, the locks, and — crucially — the storage keys that pointed at the underlying blobs.

That split was the whole security model. The client never held the blob keys and never talked to storage directly. When Windows asked to open a document, the filesystem checked the request against our rules, looked up the file’s metadata and key in SQL, and only then fetched the bytes from blob storage on the user’s behalf. The database was the control plane; blob storage was just the data plane it pointed to. Sign out of the app, and the keys — and therefore the files — were simply gone.

On open, the bytes were streamed straight from Azure blob storage through the filesystem to the application — nothing was cached to local disk. That kept the security promise intact: no copies left lingering on the machine once you were done. The SQL side — all the metadata and key lookups — ran through Dapper, kept deliberately thin and fast on what was a very hot path. (I’ve reached for Dapper on nearly every project since — enough that I eventually wrote QueryKit, a small SQL builder to sit on top of it and take the boilerplate away.)

Locks in the database

Because the whole point was central control, filesystem state couldn’t live on any one PC. In particular, we held file locks in the database. When a user opened a document, the lock that said “this file is in use” lived in SQL, not in some local handle table — so the lock was authoritative across the whole system, survived the vagaries of individual clients, and meant our application was the single source of truth for who had what open.

The part that nearly broke us: Microsoft Office

Every filesystem project has a villain, and ours wore a familiar suit.

Microsoft Office does not open a document and quietly read it. When you double-click a Word file, Office performs an elaborate ritual: it creates a hidden owner/lock file alongside the original (the infamous ~$name.docx), scatters temporary files around, and — the real killer — saves your changes not by writing back into the original file, but by writing a brand-new temp file and then atomically renaming it over the top.

To a naive filesystem, this looks like chaos: phantom files appearing that were never real documents, and a “save” that manifests as create-plus-rename rather than a tidy write.

We had to teach our filesystem the difference between a document and Office’s scaffolding. The fix, in the end, was refreshingly blunt: we taught the filesystem to recognise those ~$ owner-lock files by their telltale prefix and simply refuse to treat them as real documents. They never made it into the database, never surfaced in a matter, never counted as files at all — as far as our document store was concerned, they didn’t exist. Office got to play its locking ritual in peace, and firms never saw a single phantom ~$ file cluttering their documents.

There’s a nice symmetry there. The locks that mattered — the ones that decided who genuinely had a document open — lived in our database, central and authoritative. Office’s little lock files, meanwhile, we just waved through and forgot. It’s a small, almost silly-sounding fix for something that had caused real grief, and exactly the kind of unglamorous detail that separates a filesystem that demos from one a law firm can run all day.

What I took away from it

Writing a filesystem is one of those projects that permanently changes how you see the machine. A few things stuck with me:

The OS is an API with an enormous, mostly-undocumented contract. The interesting work wasn’t reading and writing bytes; it was faithfully honouring behaviours applications assume without ever stating.
Abstractions earn their keep under pressure. The single interface with two backends is what let us run old and new storage side by side and migrate customers without a big-bang cutover.
The last 10% is other people’s software. We could open and save our own files early. Making Microsoft Office happy took a disproportionate share of the effort — and that’s usually where the real work of a systems project lives.

By the time I left Insight, the database-backed filesystem had rolled out across a customer base of around 800 law firms, roughly 80% of whom had adopted it in place of the old mapped network drives. Every one of those firms’ documents was now present only when someone was signed into the app, visible only to those with rights to the matter, and reached exclusively through software we controlled — exactly the model we’d set out to build.

We set out to make documents behave — to be present only when they should be, visible only to those allowed, and always reached through software we controlled. To get there, we put a SQL database in charge of a filesystem with no disk of its own, and convinced Windows to mount it as an ordinary drive. It’s still one of the most satisfying things I’ve built.