Engineered to Fail · Part 5 of 6
Accident Investigation and Design Failure: How the Frame Decides the Finding
When a system fails and people die, an investigation decides who failed. That decision either keeps a design failure visible or converts it into a personnel failure.
An investigation decides whether a design failure stays a design failure or becomes a personnel failure. The UK reviewing officers found the Chinook ZD576 pilots negligent to a gross degree, a finding set aside seventeen years later, while the Nimrod Review examined the safety case and called it a lamentable job from start to finish. The Tornado ZG710 Board of Inquiry named contributory factors in the Patriot system's classification criteria and rules of engagement. The same institution produced all three outcomes, and what differed was the frame brought to the evidence.
How did the same institution reach two opposite findings?
In 1994 an RAF Chinook, ZD576, crashed on the Mull of Kintyre. The pilots, Flight Lieutenants Jonathan Tapper and Richard Cook, the two other crew and 25 passengers were killed. The reviewing officers found both pilots "negligent to a gross degree."
The aircraft’s engine control software was already in question. A House of Lords Select Committee, summarising the Commons Public Accounts Committee’s findings, recorded that "In 1994 the FADEC software was both faulty and "unverifiable": it defied independent evaluation by either the Aircraft and Armament Experimental Establishment (A&AEE) at Boscombe Down or EDS-SCICON, a firm specialising in testing software."
The finding against the pilots stood for seventeen years. In July 2011 the Defence Secretary told the Commons that "the Reviewing Officers’ conclusions that Flight Lieutenants Tapper and Cook were negligent to a gross degree are no longer sustainable and must therefore be set aside."
What did the Nimrod Review do differently?
In 2006 RAF Nimrod XV230 was lost. "Fourteen members of the armed forces tragically lost their lives on that day," the Defence Secretary told Parliament when Charles Haddon-Cave’s Nimrod Review was published in 2009. The review did not look for a negligent crew. It looked at the safety case, and found that it "was a lamentable job from start to finish. It was riddled with errors. It missed the key dangers. Its production is a story of incompetence, complacency, and cynicism." The Secretary of State told the House that the review was "critical of both the MOD and our industrial partners, at both organisational and individual levels", and said: "I am sorry for the mistakes that have been made and that lives have been lost as a result of our failure."
The same institution produced both outcomes. What changed was the frame the investigation brought to the evidence.
Can an investigation hold an automated system’s design in view?
The land case sits between the two. In March 2003, in the opening days of the Iraq war, a Patriot air defence battery shot down RAF Tornado ZG710, and its crew did not return. The UK Board of Inquiry found that the battery, "in following its self-defence rules of engagement, misidentified Tornado ZG710 as a hostile anti-radiation missile and engaged it." The contributory factors it named sat in the system and its rules: "the failure of the Tornado’s IFF (identification friend or Foe), the wide classification criteria for anti-radiation missiles programmed into Patriot system and the Patriot rules of engagement, which were not sufficiently robust to prevent a friendly aircraft without a functioning IFF system being classified as an anti-radiation missile." Of the crew, it said they "conducted a dangerous and demanding combat mission in an exemplary manner yet did not return safely."
That is an investigation holding the design in view: what the system was programmed to classify, and what its rules allowed. The question for every land force fielding automated air and missile defence now is whether its own investigations are set up to reach findings like that one, or findings like the Chinook board’s.
What does fire investigation show about proximate cause?
I have investigated fires and their causes. Time and again the proximate cause was recorded as an error by someone installing or operating a system that was poorly designed in the first place, in a way that was predictable and engineered in. In one large commercial building fire that went to a coronial inquest, the building was lost because a valve had not been reopened after servicing. There was no failsafe to catch it. The only thing standing between that building and a total loss was a single valve that nobody was monitoring.
Where would this argument be wrong?
The investigation is where a design failure either stays a design failure or becomes a personnel failure, and the outcome depends heavily on the frame the investigation brings.
This is the most contested claim in the book, and I expect pushback from people who have run inquiries. If you think the evidence, rather than the frame, decides the outcome in almost every case, I want to hear your reasoning.
Frequently asked questions
What were the original findings against the Chinook ZD576 pilots?
The reviewing officers found both pilots negligent to a gross degree. In July 2011 the Defence Secretary told the Commons that those conclusions were no longer sustainable and must therefore be set aside, seventeen years after the crash.
What did the Nimrod Review find about the safety case?
The review found that the safety case was a lamentable job from start to finish, that it was riddled with errors, that it missed the key dangers, and that its production is a story of incompetence, complacency, and cynicism. The Secretary of State told the House the review was critical of both the MOD and its industrial partners, at both organisational and individual levels.
What contributory factors did the Tornado ZG710 Board of Inquiry identify?
The Board named the failure of the Tornado's IFF, the wide classification criteria for anti-radiation missiles programmed into the Patriot system, and the Patriot rules of engagement, which were not sufficiently robust to prevent a friendly aircraft without a functioning IFF system being classified as an anti-radiation missile.
Does an investigation's frame change whether operator error or design is found?
In the documented UK cases the frame made the difference. One inquiry examined the crew's conduct and found gross negligence, later set aside. Another examined the safety case and found systemic failure in its production. A third named the system's classification criteria and rules of engagement as contributory factors while describing the aircrew's conduct as exemplary.
Why does this matter for automated air and missile defence?
Because automated systems classify and engage on criteria set during design. If an investigation is not set up to examine what the system was programmed to classify and what its rules allowed, a design decision made years earlier can be recorded as a decision made by the operator on the day.
Sources
- House of Lords Select Committee on Chinook ZD 576, HL Paper 25, 2002: https://publications.parliament.uk/pa/ld200102/ldselect/ldchin/25/2503.htm
- House of Commons Hansard, Mull of Kintyre Review statement, 13 July 2011: https://publications.parliament.uk/pa/cm201011/cmhansrd/cm110713/debtext/110713-0002.htm
- House of Commons Hansard, Nimrod Review, 28 October 2009: https://hansard.parliament.uk/Commons/2009-10-28/debates/09102833000004/NimrodReview
- House of Commons Hansard, Written Ministerial Statement on the Tornado ZG710 Board of Inquiry, 14 May 2004: https://publications.parliament.uk/pa/cm200304/cmhansrd/vo040514/wmstext/40514m01.htm
Would your investigations reach the design, or stop at the operator?
The terms of reference decide the finding long before the evidence is weighed. Testing whether your investigation framework can reach a design decision is a short piece of work with consequences that last for years.
Book a strategy call