The bug that ended shows before they started
We had a safety net that ended shows left running for weeks. It also ended shows that had been running for four seconds. Here is the whole thing, honestly.
Last week an operator pressed GO LIVE, saw the console confirm the show was on air, and watched it drop back to idle a few seconds later. Then it happened again. For anyone standing in a booth with a room full of people, that is about as bad as our product gets.
The safety net
Shows get abandoned. Someone finishes an event, closes the laptop, and the show sits in live mode forever. We once found one that had been live for 27 days. So we added a rule: if a show finished its schedule more than four hours ago, the server ends it. Cues untouched, so the post-show report survives.
The rule asked one question: is the last cue scheduled end more than four hours in the past? Reasonable-sounding. It never asked the other question, which turned out to be the important one: when did this show actually start?
Why that was fatal
A brand-new account gets a starter runsheet with cues at 09:00 through 09:50. Add four hours of grace and any new user who opened the console after roughly 13:50 local time was starting a show the server believed had ended hours ago. It ended it on the next tick, a tenth of a second later.
The same thing hit anyone who imported a rundown carrying yesterday times, or ran a show on a different day from the one its sheet was written for. Which is to say: the failure was not rare, it was systematic, and it targeted new users specifically.
What we changed
A live show must now satisfy both conditions before the net catches it: its schedule finished more than four hours ago AND it has actually been running for more than four hours. The 27-day show satisfies both and still gets cleaned up. A show that started forty seconds ago cannot be ended by the server no matter what its sheet claims.
There is now a test suite around that single function, including the 27-day case, the new-signup case, and both dry-run paths. Before this, the package that decides whether a show is running had no tests at all, while our spreadsheet parser had plenty. That imbalance was the real bug.
Three lessons we are keeping
- Test the catastrophic path, not the interesting one. Our effort had gone where the problems were fun, not where the failures were unrecoverable.
- A cleanup rule needs to know how long the thing has been alive. Any rule that ends something should ask when it started, not only when it was supposed to finish.
- Silent failures buy you nothing. The error was invisible in the interface and absent from our analytics, so the first person to notice was a user. Errors now name their cause on screen, and failures report themselves.
On timing bugs generally
This is a timing product, and both of the worst bugs we have shipped were about time semantics. This one confused a scheduled wall-clock time with an elapsed run. An earlier one read 0:05 in an imported sheet as five seconds when the author meant five minutes.
The pattern is the same each time: two quantities that both look like time, mean different things, and share a type. We now treat any code mixing scheduled time and elapsed time as guilty until it has a test proving otherwise.
A tool for live events earns trust by being boring at the worst possible moment. We were not, for a while, and we would rather write that down than quietly patch it.