Therac-25
What happened
The Therac-25 was a medical linear accelerator built by Atomic Energy of Canada Limited (AECL) to treat cancer with either an electron beam or, by swinging an X-ray target and beam flattener into the path of a far more powerful beam, with X-ray photons. Eleven machines were installed in the United States and Canada. Between June 1985 and January 1987 the Therac-25 massively overdosed six patients, at Kennestone Regional Oncology Center in Marietta, Georgia; the Ontario Cancer Foundation clinic in Hamilton; Yakima Valley Memorial Hospital in Washington (twice); and the East Texas Cancer Center in Tyler (twice). Three of the six died of the overdose. Leveson and Turner describe these as "the worst series of radiation accidents in the 35-year history of medical accelerators."
The two Tyler accidents, in March and April 1986, are the best understood. In each case an experienced operator typed the prescription at the console, noticed she had entered x (X-ray) when she meant e (electron), moved the cursor up, fixed the mode, and pressed Return several times to get back to the command line. The screen showed the parameters "verified" and the system "beam ready," so she pressed B to turn the beam on. Within moments the machine halted with the cryptic message Malfunction 54, and the dose display showed a substantial underdose. Because pausing on a malfunction was a routine annoyance, she pressed P to proceed. Each patient later described the treatment as a sizzling, burning flash; each had in fact received an estimated 16,500 to 25,000 rads in under a second, against a prescribed daily dose in the low hundreds. The Tyler hospital physicist eventually reproduced the fault himself and found the key: the overdose occurred only if the prescription was edited quickly, as a practiced operator naturally would.
The engineering failure
-
A race condition in the data-entry code. Setting the bending magnets for a treatment took about eight seconds. If the operator finished editing the mode or energy during that window, the change reached the operator's screen and the mode/energy variable but was never noticed by the routine that set up the machine, because the routine that watched for edits only checked on its first pass. The result was a high-current electron beam meant for X-ray mode, delivered with the X-ray target and flattener rotated out of the way.
-
A counter that overflowed. The second Yakima overdose came from a different bug. A one-byte variable,
Class3, was incremented on every pass through the set-up test and was meant to be nonzero whenever setup was still incomplete. Every 256th increment rolled it over to zero, and if the operator happened to hit the set button on exactly that pass, the upper-collimator check was skipped and the beam fired with the turntable still in the field-light position. -
Software in place of hardware interlocks. The earlier Therac-20 shared much of the same code and had the same flaws, but its electromechanical interlocks simply blew a fuse when the software asked for something unsafe. On the Therac-25, AECL "decided not to duplicate all the existing hardware safety mechanisms and interlocks" and relied on software checks instead, so a software error went straight to the patient.
-
A machine that cried wolf. Malfunction messages were numbered 1 to 64 and were not explained in the operator's manual. One therapist reported an average of 40 dose-rate faults on some days, and an operator testified she had been taught it was "virtually impossible" to overdose a patient. The console showed no dose, or an underdose, while delivering a lethal one.
-
A risk analysis that excluded software. AECL's 1983 fault tree assigned "computer selects wrong energy" a probability of 10⁻¹¹ with no justification, on the assumption that programming errors had been removed by testing and that software does not wear out.
-
Denial instead of investigation. After each accident AECL told the hospital that an overdose was impossible and that no other incidents had occurred, even after receiving a lawsuit over the Marietta injury. Users first learned of one another's accidents by word of mouth. When the Tyler physicist and operator finally reproduced Malfunction 54 on demand, AECL's fix for the interim was a letter telling hospitals to remove the cursor-up key cap and tape the switch open.
Leveson's later reflection is that the accident was a system failure, and that its factors are still with us: overconfidence in software, confusing reliability with safety, assuming reused code is safe code, a lack of defensive design, unrealistic risk assessments, and interfaces that made the unsafe action easy and the safe one tedious.
Lessons
Focusing on the particular bug is "not the way to make a safe system." The lessons Leveson draws instead are general ones: identify the safety-critical requirements before writing code; never let software be the only interlock between an energy source and a person; design interfaces so that unsafe actions are hard and safe ones easy; give operators error messages they can act on and an independent way to see what the machine actually did; assume software is faulty until demonstrated otherwise; and treat every incident as something to investigate rather than explain away.
Try it: the operator console simulator
The console below is a JavaScript port of the Therac-25 operator console simulator written for MIT's 6.033 Computer System Engineering course. It runs entirely in your browser. Nothing is sent anywhere, and no rads are delivered. Click the console to give it keyboard focus, then follow the steps beneath it.
Step 1: a normal treatment
First, run the machine the way it was meant to be run. Press each key deliberately; speed does not matter here.
- Type your own name as the patient's name and press Return. It will appear in uppercase.
- Press
Xto select X-ray mode. The cursor jumps to the energy field. - Type
25and press Return. - Press Return three times to step past Unit Rate/Minute, Monitor Units, and Time.
- Press Return once on each of the six rows from Gantry Rotation to Accessory Number. Each press copies the ACTUAL value into the PRESCRIBED column and the row is marked
VERIFIED. - The cursor is now on the COMMAND line and the status reads
SYSTEM: BEAM READY. - Press
Bto turn the beam on. The console reportsTREATED <YOUR NAME> SUCCESSFULLY!. - Take a screenshot of the console showing this message. Then press any key to dismiss it.
This is what an operator saw many times a day. Notice that the machine gave no dose information beyond the message, and that the only confirmation that the parameters were correct was the word VERIFIED.
Step 2: the Tyler accident
Now repeat the treatment, but reproduce the correction the Tyler operator made. She typed x when she meant e, noticed it after finishing the rest of the prescription, and fixed it the natural way: cursor up, retype, cursor back down. The bending magnets start setting the moment BEAM READY appears and take about eight seconds, so the edit has to begin immediately after step 5 below and be finished within that time. Read all the steps before you start.
- Press
Qto restart the simulator. - Type your name and press Return, press
X, type25and press Return, and press Return three more times, exactly as before. - Press Return on the six rows from Gantry Rotation to Accessory Number, but stop before the last one: leave the cursor on Accessory Number.
- Put your fingers on the arrow keys. Your eight seconds begin with the next keystroke.
- Press Return. The status changes to
BEAM READY. - Hold ↑ until the cursor reaches the top of the screen (extra presses do no harm).
- Press ↓ once to land on Beam Type, and press
E. - Hold ↓ until the cursor is back on the COMMAND line. The screen now shows
BEAM TYPE: Eand still saysBEAM READY. - Press
B. - Take a screenshot of the console showing the
MALFUNCTION 54message and the dose delivered.
The console reports MALFUNCTION 54 and a dose of more than 10,000 rads. Every value on the screen was correct at the moment the beam turned on; the machine simply did not act on the value it displayed.
Turn in both screenshots: the successful treatment from Step 1 and the malfunction from Step 2, each showing your name in the PATIENT NAME field.
Think about it
Repeat Step 2, but pause for ten seconds after pressing E before coming back down and pressing B. The same keystrokes now produce a normal treatment. Why did the Tyler operator's experience, which made her fast at the console, make her more likely to trigger the fault than a trainee would have been? What would the machine have had to check, at the moment B was pressed, to make the fault impossible?
Credits
The simulator is a port of therac.c from the MIT 6.033 Therac-25 hands-on assignment (original C source), MIT Department of Electrical Engineering and Computer Science, Spring 2007. The layout and state machine are unchanged and the curses terminal has been replaced with a browser rendering; the guided steps above replace the open-ended questions of the original assignment. Two deliberate departures: the Return-key shortcut that the original assignment noted was "not working today" is enabled, and the eight-second blind spot, which the C program silently re-arms every eight seconds, here opens once when data entry is first completed (the bending magnets start to set) and closes for good eight seconds later, matching the behavior of the Datent, Magnet, and Ptime routines described on pages 29–31 of the Leveson and Turner paper.
Sources
Local copies of both papers are in this directory.