September 24th

The whole incident is five commits long.

09-23  🚀 Enable optimized feed processing
09-24  🔧 Fix memory issues in optimized feed processing
09-24  🚨 HOTFIX: Revert to stable feed processing
09-24  🔧 Hotfix: Revert to stable feed processing
09-24  🚀 Fix ES modules and enable optimized worker

Read it in order. On the 23rd we turned on the optimised path — the one DEPLOYMENT.md promised would take bulk imports from Hours → Minutes, 10-50x faster. Overnight it ran out of memory. In the morning there was an attempt to fix it in place. That did not hold, so it was reverted. Then it was reverted again, forty minutes later, by someone who did not trust that the first revert had gone out.

That second Hotfix: Revert is the honest one. Nobody writes the same revert twice because the code needs it. You write it because a graph is still red and you cannot tell whether what you did has reached production yet, and doing it again is cheaper than finding out.

Then, the same day, the optimised worker came back on — through a change to the module format. Which means the failure the day before was at least partly a packaging problem wearing the costume of a memory problem, and we shipped the fix without ever separating the two.

What did not come back

bulk-db-operations.ts — 470 lines — was the thing being enabled and disabled all day. It has been dead since that afternoon. Nothing imports it; its only mention anywhere is a filename in a list inside a build script.

That would be an unremarkable ending, except for what was inside it: the only live call to calculateProductHash, an md5 over the fields that actually matter, so that unchanged rows could be skipped. On a marketplace feed maybe five per cent of rows change between runs. That one function was worth more than every batch size and concurrency limit in the project put together.

We built the optimised path. It broke, for reasons we half-understood. We reverted it twice and re-enabled a version of it that, as the shipped bundle confirms, does not contain the file at all — the string calculateProductHash appears zero times in dist/worker.js.

For the next three weeks the importer kept running, re-downloading the feed once per batch, writing every product whether it had changed or not.

The rule I keep relearning

A revert is a decision to stop losing, and on the day it is almost always right. What it is not is a decision about what happens to the code afterwards — but that is the decision it silently makes, because nothing ever comes back on its own.

Someone has to open a ticket while their hands are still shaking, and nobody does, because the graph is green again and green feels like finished.

Context in the full post-mortem.