Two hundred and sixty-four gigabytes free

On Monday the Mac hung on boot for the second morning in a row. The progress bar filled most of the way, then nothing. The first guess was the obvious one: the disk is full.

Recovery mode disagreed. The container had 264 gigabytes free. Space was not the problem. The number of files was.

Thirty thousand directories in one folder

On macOS, os.tmpdir() does not return /tmp. It returns a per-user folder under /private/var/folders/<xx>/<hash>/T. Nobody looks there, and nothing in the UI tells you how full it is.

From Recovery we unlocked the data volume, renamed var/folders and tmp to .old, and let the machine boot. Then an agent counted what we had moved aside.

The T folder alone held 30,182 directories and 1,378,905 files. Every one of them was created between 23:00 on Sunday and 15:00 on Monday.

Three prefixes did most of it: construct-detach-* with 246,000 files, construct-attach-* with 234,000, and ghosts-preflight-test-* with 113,000.

The fix that made it worse

The first attempt was three commands in Recovery:

mv /Volumes/Data/private/var/folders /Volumes/Data/private/var/folders.old
mv /Volumes/Data/private/tmp /Volumes/Data/private/tmp.old
mkdir -m 1777 /Volumes/Data/private/tmp

The reboot that followed never finished. The machine went into a loop. macOS creates the per-user folders inside var/folders on its own. It does not create var/folders itself.

One more trip to Recovery:

mkdir /Volumes/Data/private/var/folders
chown root:wheel /Volumes/Data/private/var/folders
chmod 755 /Volumes/Data/private/var/folders

That booted. If you ever do this, create the empty parent before you reboot, not after.

Where they came from

Our agents run the test suite all night, in parallel, across many git worktrees. Every run is a small, sensible test. The total is the problem.

The tests. Fixtures were created like this and never removed:

const dir = mkdtempSync(join(tmpdir(), 'construct-detach-'))

One directory per test is harmless. Thousands of runs a night is 25,299 directories.

We had already fixed this. Each run now gets one root, mkr-<pid>-XXXXXX, with TMPDIR pointed inside it, and the root is removed when the run ends. The fix reached main at 00:50 on Monday. The leak stopped around 04:00.

The three-hour gap was the worktrees. Branches cut before the fix kept running the old code until they were rebased. Two of them still had not been when we looked. A fix on main does not reach a branch cut yesterday.

Vitest. The other 4,603 directories look like <nanoid>/ssr/<sha1>, and they still appear at about 40 an hour. Vitest 5.0.0 makes its temp directory at startup, before our run root rewrites TMPDIR. Its cleanup clears per-project folders, and only if the process lives to close. The root folder stays.

We reproduced it with a single test file and an empty TMPDIR. After the run, one directory was left behind.

Where temp files land

1. test fixture on main
2. run root
3. removed when the run ends

1. test fixture on a stale branch
2. system temp
3. nobody

1. Vitest at startup
2. system temp
3. nobody

One path ends in a folder that is deleted when the run ends. The other two end in a folder nobody owns.

We don’t know which part of startup walks that tree and chokes. We know that with 1.4 million entries in it the machine did not come up, and with an empty one it did.

What we are changing

Not a cleanup cron. A cron cleans up after a mechanism that is still broken.

1. Vitest starts through the same wrapper as everything else, so TMPDIR points into the run root before Vitest creates anything. Its folder dies with the root.
2. The reaper that removes orphaned run roots also removes Vitest-shaped folders older than an hour in the system temp. That catches a bare vitest run and runs killed by a signal.
3. A lint rule forbids mkdtempSync(join(tmpdir(), …)) in tests. The only way to get a scratch directory is through the run root.
4. Stale worktrees get rebased or deleted.
5. The Vitest behaviour goes upstream as an issue.

The part that stings

Every monitor we had was green. Disk space was fine. Memory was fine. Nothing counted files, because nobody thought files could be the thing that kills a machine with a quarter of its disk empty.

tmpdir() is a place to put things. It is not a promise that anyone will take them away. With one developer running tests by hand, that difference never shows. With agents running the suite all night, it took sixteen hours.

If you take one thing from this: every file a process creates needs an owner, a lifetime and a removal you can check. tmpdir() gives you none of the three.