Out of inodes with 56 GB free - moving source trees into squashfs

webrtc.page is a set of MCP servers for reading real-time media codebases: libwebrtc, pion, coturn, Kamailio, Asterisk, FreeSWITCH, Janus and a few more. Each server keeps several releases of its project on disk, with a semantic index and a tree-sitter symbol index next to the source. libwebrtc is the big one: 32 Chrome milestones, the newest of them with their full DEPS checkout.

On 7 November 2026 I deployed M157 and the site went down. This post is about why, the fixes I shipped the same day, and the change that came out of it: every source tree is now a single squashfs image instead of a directory with a few hundred thousand files.

The failure

The deploy script packs changed versions into a tar archive, uploads it while the services are still running, then stops pm2, removes the old data and extracts the archive. The extraction step printed thousands of lines like this:

tar: ./versions/M157/src/third_party/blink/web_tests/fast/dom/Element/...: Cannot open: No space left on device
tar: Exiting with failure status due to previous errors

The script stopped there, with pm2 already stopped. Every endpoint returned 502 from 17:52 to 18:11 UTC, about 19 minutes, until a second deploy finished.

The disk-space check before the upload had passed. It said 59 GB would be free after extraction, and df -h agreed:

/dev/sda   199G  133G   56G  71% /mnt/volume_lon1_...

df -i did not:

/dev/sda   6553600 6553600  0  100% /mnt/volume_lon1_...

The data volume is ext4 with the default ratio of one inode per 32 KB, which gives 6.55 million inodes for 199 GB. Source code is millions of small files. The volume ran out of files long before it ran out of bytes, and my pre-flight check only counted bytes.

Where the inodes went

I counted files per directory on the server:

So a libwebrtc version is about 5% WebRTC and 95% third_party. That is expected once you look at DEPS: the entry src/third_party is Chromium's entire third_party repository, not a list of the libraries WebRTC links. Inside it, blink/web_tests, perf_tests and manual_tests alone were 203k files in M157. None of them were in the semantic index.

Two more things came out of the count.

Orphaned dependency trees. Locally I keep deps only for the newest versions and strip them from older ones with a compact command. On the server, 16 versions still had their third_party, not 7. compact deletes files but does not touch the version's updatedAt, and the incremental deploy only re-uploads versions whose updatedAt changed. So every compaction since the start stayed local, and the server kept nine trees of ~380k files that nobody had on disk anymore.

.git directories. Each DEPS checkout is a shallow git clone with its own .git. The deploy excluded only the top-level ./.git, so 45 nested ones per version, about 1.3 GB, went to production with every release.

Same-day fixes

To bring the site back I removed the half-extracted M157 and the orphaned third_party trees by hand, then re-ran the deploy. After that I made three changes so this would not happen again.

An inode check before anything is stopped. The deploy now counts entries in the archive and compares them to df --output=iavail on the server, keeping 200k spare. If the archive does not fit, the deploy fails while the old version is still serving.

Junk never reaches the disk. One list, PRUNE_PATHS, names the paths not worth keeping. The downloader turns it into a non-cone sparse checkout for the dependency that contains them, so they are never written:

/*
!/blink/web_tests/
!/blink/perf_tests/
!/blink/manual_tests/

The same list is used to prune versions that are already on disk, to exclude the paths from the archive, and to delete them on the server before the inode check. A version with deps went from ~396k files to ~193k.

compact follows the deploy. Compaction now bumps updatedAt, so the deploy re-uploads the version without deps and with the smaller index. The deploy also re-uploads any version that has third_party on the server but not locally, whatever its updatedAt says.

After the next deploy the volume was at 41% inodes (2.65M used) and 75 GB of disk. A good result, but the shape of the problem had not changed: one inode per source file, and every new milestone adds another ~190k.

Options

The hardlink idea was tempting: third_party/abseil-cpp in M156 and M157 is byte-for-byte identical, all 1,655 files. But hardlinks only survive inside one tar archive, and the deploy ships one version at a time. Keeping them on the server would mean a dedup pass after each deploy.

squashfs is a read-only, compressed filesystem in a single file. The kernel mounts it with a loop device; without root there is squashfuse. My versions are already read-only in practice: a milestone is downloaded once, indexed once, and then only read. An image fits that.

Measurements

Before changing any code I packed M157 (deps, after pruning) and M150 (no deps) on my workstation: 20 cores, NVMe, mksquashfs 4.7.5, squashfuse 0.6.3. .git was excluded from the image.

Reading is what the servers do all day, mostly ripgrep over the whole tree and single-file reads. Times are on a warm page cache unless marked "first run":

Plain squashfuse is single-threaded, and a full-tree search through it is 100 times slower than the directory. squashfuse_ll, the low-level multithreaded driver from the same project, is within 4–12x. The compression level did not change read speed (zstd-3 and zstd-15 both gave 2.1 s and 0.65–0.68 s), so I took zstd-15 for the smaller files.

On the server I do not need FUSE at all. The kernel there (Ubuntu 24.04, 6.8) has squashfs built in with CONFIG_SQUASHFS_ZSTD=y, and the services run as root, so a kernel loop mount uses the normal page cache and needs no daemon in a 4 GB box.

I repeated the test on the server with M150, the image placed on the same data volume as the directory, and the page cache dropped (echo 3 > /proc/sys/vm/drop_caches) before each cold run:

Here the image is 2.4–3.3x faster cold, and the same warm. The data volume is DigitalOcean block storage, attached over the network. Reading the tree as a directory means 16k small files scattered across it; reading the image means 54 MB of compressed data laid out together. On local NVMe the image only costs; on network storage it pays for itself.

The change

Every source tree keeps its path. What used to be a directory is now an image next to a mount point:

versions/M157/src/        directory          →  mount point (read-only)
versions/M157/src.sqfs                       →  image
versions/M157/semantic.db, ast.db            →  unchanged

Nothing that reads source had to change: ripgrep, file reads, path sanitising with realpath and the AST indexer all see the same directory as before.

The module is small. Packing is one mksquashfs call:

export function packArgs(treeDir, out, { level = 15 } = {}) {
  return [
    treeDir, out,
    "-comp", "zstd", "-Xcompression-level", String(level),
    "-noappend", "-quiet", "-no-progress",
    "-wildcards", "-e", "... .git",   // nested .git dirs stay out
  ];
}

Mounting depends on who is asking:

export function mountCommand(img, mnt, { root = false } = {}) {
  return root
    ? ["mount", ["-t", "squashfs", "-o", "loop,ro", img, mnt]]
    : ["squashfuse_ll", [img, mnt]];
}

Mounts are state outside the filesystem: a reboot or a deploy leaves empty mount points. Each MCP server therefore mounts the images of its own project when it starts. The call is idempotent, and a failed mount is logged and costs only those versions, not the server. The deploy also runs a mount-all before starting pm2, so a failure shows up in its log.

Three things broke on the way, all for the same reason: a mount point looks like a directory, so tools treat it as one.

• tar walks into mounts. Packing a version whose src/ is mounted put all 190k files back into the archive. The deploy now excludes the mount point of every packed tree.
• du walks into mounts too. The size estimate counted the uncompressed contents. It uses du -x now.
• rm -rf cannot remove a read-only mount. Replacing a version on the server first unmounts its trees, then deletes the directory.

The migration avoided re-uploading whole versions. A version that is packed locally but is still a directory on the server ships only its src.sqfs. The server deletes the old src/ directory, the image takes its place, and semantic.db and ast.db stay where they are.

Results

On my workstation, packing all 32 libwebrtc versions took about five minutes:

Images for versions with deps are 703–744 MB, without deps 50–60 MB.

Pros and cons

What I got:

• One inode per version instead of up to ~190k. Inodes are no longer the limit; disk is, and slowly.
• 4–5x less disk. M157 went from 3.3 GB to 710 MB, M150 from 276 MB to 54 MB.
• No .git in production, without a separate exclude rule.
• A version deploys as one file. No extraction of hundreds of thousands of files while the services are stopped.
• Immutable trees. A mounted image cannot be modified by a stray script, and the same file is what runs locally and in production.
• Faster cold search in production. On network block storage a cold whole-tree ripgrep went from 4.7–7.3 s to 1.9–2.2 s.

What it costs:

• Slower full-tree search without root. On my workstation, through squashfuse_ll, a cold whole-tree ripgrep is about 12x slower than a directory, 4x warm. Searches scoped to a subdirectory do not notice.
• One FUSE process per mount locally. About 6 MB RSS idle, 36 MB after a full scan of M157. With 32 libwebrtc versions that is fine. With pion's 418 small trees it is not, which is why pion is not packed yet.
• Mounts are state. A reboot, or a deploy that fails between unmount and mount, leaves empty directories. The servers remount on start, but it is one more thing that can be wrong.
• Changing a version means repacking it. Adding deps to an old version or pruning it now goes extract, modify, pack. The tools do this automatically, but it is slower than deleting a directory.
• No incremental upload inside a version. Any change to a tree produces a new image, uploaded whole.
• Root on the server. Kernel mounts need it. My services already run as root; if yours do not, you need a mount helper or FUSE there too.

What's next

libwebrtc was the obvious first target: 32 trees that held most of the inodes, and a small number of mounts. pion is the opposite: 418 trees of 2–3k files each, 648k inodes in total. One image per tag would mean 418 mounts. I will probably group tags, one image per minor release line, and accept repacking a group when a new patch release lands.

The broader lesson is about the check that failed. Disk space is the number everyone watches, and the deploy script watched it carefully. For a workload made of small files, inodes are a separate budget, and it runs out silently.