Thesis: The argument in the Pre-Implementation Safety Case for RVSM [1] demonstrates at most that RVSM operations without ACAS meet Target Levels of Safety. It does not demonstrate that RVSM operations with ACAS-equipped aircraft meet Target Levels of Safety; neither can a correct argument for this assertion be reconstructed from the document. Since most aircraft operating in RVSM are required to be ACAS-equipped, the safety case does not establish the safety level of RVSM operations as they are currently conducted and for the foreseeable future.
The remainder of this note establishes this thesis. I hope also to make it clear that this is not just a formal problem which can be quickly fixed, but a fundamental problem with the entire argument. Note also that the thesis says that the case has not been made; it does not say that the assertion, that RVSM with ACAS meets the Target Level of Safety (TLS), is not true. However, I do believe that recent incidents must lead us to look very carefully at it.
The safety case for European Reduced Vertical Separation Minima (RVSM), by which the altitude separation between Flight Levels 290 and 410 (that is, between 29,000 ft pressure altitude and 41,000 ft pressure altitude) is reduced to 1,000 ft vertically instead of the previous 2,000 ft vertically, argues that the performance of ACAS II does not affect the safety assessment of RVSM operations. There are three recent incidents which I believe are relevant to assessing this assertion
In particular, what is definitively known about the Bodensee accident suffices to demonstrate the safety case flaw.
Somewhat worryingly, the safety case document was published eleven months after the Swiss incident, on 14 August 2001, and Switzerland is a member of Eurocontrol, who published the document.
The general view of the safety case document is that ACAS (even in its TCAS II 6.04a Version which is not ACAS II and thus not RVSM-compatible) is an collision risk-reducing system and thus a safety case for RVSM which excludes ACAS is a conservative argument. I shall examine two versions of this argument, corresponding to two interpretations of "ACAS is a collision risk-reducing system" and both of them fail to establish the desired conclusion that RVSM-with-ACAS-meets-TLS by relying on the assertion that RVSM-without-ACAS-meets-TLS. There is no other argument in the safety case for concluding anything about RVSM-with-ACAS-meets-TLS from RVSM-without-ACAS-meets-TLS. The rest of the argument in the RVSM safety case attempts to show that RVSM operations without ACAS meet TLS. This is therefore the most that the safety case in the document can demonstrate. But it is barely relevant to current and future RVSM operations, since most operators are required to he ACAS-equipped.
Here is the detailed argument showing the flaw in the safety case.
(see, for example p197). They are speaking here specifically of collision risk assessment. They are also working to Target Levels of Safety (TLS) of 5 x 10**(-9) mishaps per flight hour. Finally,
Notice that the crucial part of this argument is the assumption 1a. Notice further that 1a has nothing to do with RVSM operations per se (according to Eurocontrol written documentation and correspondence [6, Section 2.2, p8]).
So RVSM SC is invalidated (10, above) by the invalidation (3) of an assumption (1a) that has prima facie nothing to do with RVSM.
One might be tempted to rescue (1a) by saying that it speaks to overall risk. Indeed, this may well rescue (1a) but now the reasoning (1b) does not work to establish the case. Here is why.
I think the argument (1b) is quantifying universally over states, like this.
This argument is valid. Here is the reasoning.
Justification for (1b, I1):
(1a): In every RVSM situation (or state), use of ACAS does not increase
risk of collision in that state. (This what "majorises" means).
Assume that for a given state, severity of collision remains constant with or without ACAS (in other words, everybody on both airplanes dies and we do not take account of the fact that the ACAS manoeuvres led to an airplane crashing onto a field full of festival-goers rather than on the empty field next door). This assumption is made by the SC, which considers risk of collision (which I take to mean that all collisions are taken to have severity 1).
Then since probability of collision in each state with ACAS is less
than or equal to probability of collision in each state without ACAS,
it follows by summing over states that overall risk with ACAS is less
than or equal to overall risk without ACAS.
QED (1b, I1)
Now suppose risk without ACAS can not be shown to majorise risk with ACAS. Then
But this assertion is invalid. So it cannot be used to derive the assertion (risk of RVSM with ACAS is less than TLS) from (risk of RVSM without ACAS is less than TLS).
Indeed the only way of proceeding is now the following.
One derives (or has already) some sort of taxonomy T of RVSM situations. Each situation in T is called T.i, for some value of i in an index set I, as usual. And, because we take it (proposition 3 in the argument) that (1a) is invalid, then for at least one member of T, say T.i0, risk with ACAS in T.i0 is greater than risk without ACAS in T.i0 (I would claim there are at least three such different situations: one instantiated by the Swiss incident [2], one instantiated by the midair collision [4], and one instantiated by the problematic three-airplane situation I described in [5]).
Let the (estimated) risk with ACAS for each class T.i in the taxonomy be R(T.i). Let the relative frequency of occurrence of instances of class T.i be P(t.i). Then one calculates the overall risk as Σ(i in I) R(T.i).P(T.i) [10, Chapter 10], and one shows that this is less than TLS.
That kind of calculation is not performed in SC. Indeed, the only things said in SC about any of this are various reformulations of (1a) and (1b).
But those are the only two alternatives. So, to summarise, either
(a) is not correct, as (3) asserts, and (b) is nowhere to be found.
So interpreting (1a) as in (1b I2) does not help rescue the safety case. The case at most shows that RVSM without ACAS meets TLS, and this does not suffice alone to demonstrate that RVSM with ACAS meets TLS.
[1] EUR-RVSM Pre-Implementation Safety Case, Edition 2, 14 August 2001, available from http://www.eur-rvsm.com/library.htm
[2] Federal Department of the Environment, Transport, Energy and Communications, Final Report of the Aircraft Accident Investigation Bureau concerning the incident (Airprox) between THY1944, BAG4608, IBE3514 and AZA467 on 13th September 2000 UIR Switzerland near TRA. (No date). Available from http://www.bfu.admin.ch/common/pdf/A024e.pdf
[3] Air Accidents Investigation Branch, AAIB Bulletin No.~6/2001, available from http://www.aaib.dft.gov.uk/bulletin/jun01/cggwd.htm
[4] Bundesstelle fuer Flugunfalluntersuchung, Presseinformation, Zusamme nstoss am Bodensee (available also in English), reviewed July 28th, 2002, http://www.bfu-web.de/aktuinfo-d28.htm
[5] Peter B.~Ladkin, ACAS and the South German Midair, Technical Note RVS-Occ-02-02, available from http://www.rvs.uni-bielefeld.de, under Publications.
[6] Eurocontrol, ACAS II Operations in the European RVSM Environment, Project ACTOR, available from http://www.eurocontrol.int -> Projects -> ACAS -> Training Materials -> Brochure f11, 2 August 2001.
[7] JAR Failure Condition Tolerability Matrix, Appendix to NLR Project ARIBA WP6 Final Report Part II, available from http://www.nlr.nl/public/hosted-sites/ariba/rapport6/part2/appendix.htm
[8] The Official Web Site for the European Reduced Vertical Separation Minima Programme, http://www.eur-rvsm.org
[9] U.S. Federal Aviation Administration, Reduced Vertical Separation Minimum, North Atlantic RVSM, http://www.faa.gov/ats/ato/north_atlantic.htm
[10] Nancy Leveson, Safeware, Addison-Wesley, 1995
Eurocontrol training materials, specifically [6, Section 2.2, p8], say that "The mandatory carriage of ACAS II in Europe and the implementation of RVSM are not linked even if there is an interaction between them. ACAS II is not, itself, a prerequisiste for RVSM."
I think it is misleading to say that the "mandatory carriage" of ACAS II and RVSM are not linked. In fact, the safety case document explicitly includes two Hazards identified by the FHA involving ACAS, namely [1 p164 Table D1]
"Remote" means, according to the JAR definitions, an occurrence frequency of between 10**(-7) and 10**(-5) per flight hour. See [7]. That means a frequency of 1 in 10,000,000 to 1 in 100,000 flight hours.
The risk due to the Nuisance Alerts from TCAS II Version 6.04a aircraft is regarded as "not tolerable" [1 p47]. It suggests [1 p93, Section 5.14, Conclusions on the RVSM Concept] that it is "relevant" that the risk is not unique to RVSM operations, and will be subject to ongoing monitoring. Maybe so, but neither factor can mitigate the risk. It does consider [1 p94] as "mitigating factors" (i) the comparability with the current environment in FL 245-295 and (ii) the low percentage of V6.04a flights in EUR RVSM airspace (3%, according to Point 5 of the Summary on p89 of Section 5.12.2, ACAS/TCAS Nuisance Alerts), and on that basis concluded that the risks should be deemed tolerable at that stage. The comparison (i) is an observation, so cannot mitigate any risk. The percentage (ii), though, may well be, The chances of two TCAS V6.04a aircraft meeting randomly can be argued to be 9 in 10,000, or about 1 in 1,000, and those of a TCAS II V7 aircraft meeting a TCAS II V3.04a aircraft 3 in 100. A risk deemed "not tolerable" could be argued to be mitigated if its occurrence is reduced by a factor of one thousand, respectively 3 in 100, depending.
In Section 5.12.2, in which it considers ACAS/TCAS Alerts, the safety case makes a number of assumptions and assertions which I am inclined to question. For example,
Section 5.12.2 [1 p88] says that "Any manoeuvre based on ACAS Resolution Advisories (RAs) ... when adhered to by the pilot .... do[es] not induce a risk [of collision]." I believe the Swiss incident [2] is a counterexample to this statement. However, it may be that the 400 ft vertical separation maintained is sufficient to classify the incident as "no risk of collision".
However, Section 5.12.2 [1 p88] recognises that ".... nuisance advisories may constitute a risk, e.g., in particular situations which result from pilots not following the ACAS RA". This appears to vitiate the interpretation of (1a) under which risk-without-ACAS majorises risk-with-ACAS.
It cites as a fundamental principle that the RVSM risk assessment should be conducted without "taking advantage" of the "risk reducing effect" of ACAS, and therefore that the actual risk in RVSM airspace (with ACAS benefit) is lower than the risk calculated in the assessment (performed without considering ACAS "benefits"). I regard this as an unfortunate choice of words that prejudice the reader towards the case being made. I would have preferred a more neutral phrasing. Further, it says "On this basis the ACAS events have not been taken into account in the C[ollision] R[isk] A[ssessment] unless they are caused by genuine operational errors." The balloon-climb incident is a case in which there was a genuine collision risk (an "operationally valid" RA), which was not caused by an operational error.
It says further that "The experience obtained with the early introduction of RVSM ..... has not indicated the presence of any issue regarding V6.04a equipped [sic] in flights operating under RVSM conditions." This is contradicted both by the Swiss incident and by the NAT Track E balloon-climb incident. Indeed, the AAIB recommended that the relevance of the incident to the RVSM safety case be examined. Further, "... a manoeuvre based on TCAS V6.04a RAs is not risk-increasing..." I believe this statement is contradicted by the Swiss incident.
The reasoning, that ACAS effects are risk-reducing and therefore that the collision risk in RVSM airspace not taking ACAS into account is lower than the actual collision risk in RVSM airspace, has been considered above as assertion (1b). Such reasoning is found, for example, on p197, a summary of an earlier argument.
The safety case does not appear to investigate any hazard due to an operationally valid (that is, non-nuisance) alert, such as the Bodensee midair [4] and the NAT Track E balloon-climb [3].