I didn't think that was about too many alarms going off and so that a real one was being ignored, and re-reading the article doesn't give that impression either. It was more to do with a bug in the user interface (it appeared that you were re-editing a field you'd just filled in but actually weren't) and a memory corruption bug, along with assuming that software always works so doesn't need any additional checks. Are you thinking of another software failure?
(If by "dangers of creating too many errors" you just meant "don't create too many bugs" then that's not really relevant to this conversation, and also totally the wrong lesson to learn from that incident. The lesson is, software will inevitably have bugs, so make sure you have redundant checks to minimise their impact.)
The alarms were safety hardware interlocks. Nobody asked why and found the UI bugs that was causing this. The hardware interlocks were removed because the system had been working safely for so long. This was really feedback - nobody informed the developers that safety hardware interlocks were locking all the time.
Edit: see replies, originally I wrote fuses, but hardware interlocks is more correct.
The Wikipedia article the previous commenter linked doesn't mention this, and I've looked at a few sources linked from it and they don't mention it either.
They mention that previous Therac machines had "hardware interlocks" to prevent this sort of issue - maybe those are the safety fuses you're both talking about? - but it says they were never included in Therac-25 in the first place, because the software was assumed to be so reliable, not that they were removed because they blew.
Are you sure this is the same incident? Do you have a link to a source about it?
Edit: This article cited in Wikipedia is much more comprehensive and mentions that Therac-20 did have these fuses, in addition to hardware interlocks. But they weren't disabled because the real alarms were drowned out by false alarms - instead they all represented real instances of the issue.
(Middle column of the 12th page of PDF, numbered page 29 of the publication.)
> The Therac-20 at the University of
Chicago is used to teach students in a
radiation therapy school conducted by
the center. The center’s physicist, Frank
Borger, noticed that whenever a new
class of students started using the Therac-20, fuses and breakers on the machine tripped, shutting down the unit.
These failures, which had been occurring ever since the center had acquired
the machine, might appear three times a
week while new students operated the
machine and then disappear for months.
Borger determined that new students
make lots of different types of mistakes
and use “creative methods of editing”
parameters on the console. Through
experimentation, he found that certain
editing sequences correlated with blown
fuses and determined that the same computer bug (as in the Therac-25 soft-
ware) was responsible. The physicist
notified the FDA, which notified Therac-20 users.*
> The software error is just a nuisance
on the Therac-20 because this machine
has independent hardware protective
circuits for monitoring the electron-beam scanning. The protective circuits
do not allow the beam to turn on, so
there is no danger of radiation exposure
to a patient. While the Therac-20 relies
on mechanical interlocks for monitoring the machine, the Therac-25 relies
largely on software.
Yeah, the lack of hardware interlocks are part of what makes it a (literal!) textbook case for software engineering ethics.
I remember talking about it several times over my college career, and for good reason.
To this day when people start talking about software in the loop in a safety context I get extremely twitchy. I don't let that shit happen on my watch.
I had an interview once for a firmware position dealing with (relatively) high wattage transmitters. They wanted to know how I’d deal with some conditions where I’d need to turn off the power quickly. After talking a bit about my experience with hard real time deadlines, I sheepishly asked why they didn’t have diodes to mitigate the problem. I think it was one of the questions that got me an offer. Apparently having firmware engineers with even minor EE background and will think about analogue design is a little uncommon.
Today, I’d ask about more ways to push the responsibility away from the firmware to stop catastrophic failures. If the debugger pauses the CPU you can’t use the CPU to regulate things. I’ve worked on things where the failure mode can be explosive and it makes dev really harrowing if you can’t rely on hardware interlocks.
>Apparently having firmware engineers with even minor EE background and will think about analogue design is a little uncommon.
They often don't have to know how the nitty-gritty of the HW works. HALs are used everywhere, and some of them abstract a lot away. I've interviewed embedded SWEs that couldn't explain how the tri-state buffers work within GPIO.
The miscommunication between GGP and GP ITT exposes the contextual minefield of ambiguity we navigate daily. What is an "error"? A mistake made by the developer, or a fault condition exposed to users?
(If by "dangers of creating too many errors" you just meant "don't create too many bugs" then that's not really relevant to this conversation, and also totally the wrong lesson to learn from that incident. The lesson is, software will inevitably have bugs, so make sure you have redundant checks to minimise their impact.)