26 November 1998

Dear Mrs Dunwoody,

It is a year since I first wrote expressing my concern to you about the costs and viability of the NERC development. I believe there is still cause for concern about the system development, and I am glad that the Transport Subcommittee is keeping this topic in view. Although I believe that the role of formal acceptance tests has been misunderstood or misused by participants in the public discussion, and the chosen methods of review have not led me to alter my basic technical concerns expressed to you a year ago, I should like to confine myself here to offering the subcommittee a few thoughts about the costs of the NERC software development, based on events of the past year and other evidence presented to you.

Just about the simplest `back-of-envelope' cost model for development of a system involving significant software is a fixed-proportion model. Such a model sets a formal contract milestone as a point of reference, and attempts to assign the proportions of resources that will be used each side of this fixed point. I shall use the milestone of formal acceptance, which occurred during April and May, 1998.

Let us call system development before the point of formal acceptance `forward development', and system work after this point `post-acceptance' (it is more traditional to call it `maintenance', but I have found that this phrase carries inappropriate connotations for non-specialists, as well as somewhat varied meanings for specialists). The model assigns a fixed proportion of total costs, say x%, of a software project to forward development, and the remainder, (100 - x)%, to post-acceptance. Note that the post-acceptance phase includes modifications, transitioning to deployment, `upgrading', fixing problems occurring in development that persisted through formal acceptance, and so on. My experience is that non-specialists often do not realise that the major costs of a complex software system occur in its life post-acceptance. Some sample values of this x% from the most significant literature are:

I shall call these models SE (for `Software Engineering') when I refer to them collectively below.

NATS states that the cost of the NERC system up to formal acceptance was 216.9m pounds, out of a total budget of 475m pounds (NATS Annual Report, 1998, p13). It follows that the forward-development proportion that NATS is implicitly using is 216.9m/475m, which is 45.7%. (In fact, 216.9m pounds was what NATS paid on a fixed-price contract. The costs of the system up to formal acceptance can be estimated to be at least 37.5% higher than this, as argued below.)

The ratios of forward development costs to post-acceptance costs according to the four models are:

I should like to bring to your attention some conclusions one can draw about the NERC system costing using these fixed-proportion models: conclusions which, although crude and general, give cause for concern.

First I should note that the date of acceptance testing is used only as a formal boundary in the fixed proportion model. Nothing is implied about the quality or state of the product at the time of formal acceptance. This is a conservative assumption: if the system has suffered from problems during development, and both client and supplier had been motivated to perform acceptance testing sooner rather than later, the costs of forward development will be estimated lower than appropriate, post-acceptance costs will be estimated to be some fixed proportion of this lower figure, and consequently use of the model will tend to understate both post-acceptance costs and the total cost of the system over its lifetime.

I shall assume for the sake of these calculations the view that has been presented to you in other evidence: that the NERC system will work; that it will be introduced into operation in early 2002; and that its operational lifetime will be 20 years. I don't necessarily concur with these assertions, as you will infer from my previous evidence before the Subcommittee, but that is not my topic here.

The NERC software was in development for 5.5 years up to formal acceptance. This compares with the 4 years planned and bid. I assume that NATS constructed its forward-development planning at the time of bidding around this four-year figure, and not around a figure of 5.5 years. (May we assume that, had the supplier overbid, this would have been detected during bid appraisal?) Forward development thus took 37.5% longer time than planned. One can therefore crudely estimate the `actual' cost of forward development to be higher in this proportion than the fixed-price contract cost: approximately 298m pounds (this figure is to 216.9m pounds as 5.5 is to 4, that is, 37.5% higher, rounded to the nearest million pounds).

This estimate of an `actual cost' for forward development is needed in order to estimate a total-cost figure under a fixed-proportion model. If the forward development cost 37.5% more than planned, any fixed-proportion model entails that the post-acceptance costs will be 37.5% more than planned, as can be easily seen.

NATS claims (op. cit., p13) that money has been saved by letting a fixed-price contract for forward development. I think this could be misleading. Money is being used to represent resources, and it is important to know how many resources have actually been used in forward development, not just how many resources have been paid for by this point. To exaggerate somewhat to make the point, stating that one has `saved money' is rather like taking out a loan on a 20,000 pound car, paying a couple of thousand pounds down, and claiming that one has `saved money' by so doing. (In the short term, of course, one has!) At the end of the system lifetime, it's the actual resources used in total that will have been compensated. But paying less `up front' does have one short-term consequence. It would only be sensible for the supplier to base further offers for support of the system on estimates of actual resources used, minus what has been paid for already. A total-cost estimate based on forward-development actual-cost estimates is therefore important for all parties.

The figures for the four different fixed-proportion models work out as follows, using the figure for the estimated `actual cost' of 298m pounds. Using the proportions above, post-acceptance costs (to the nearest million pounds) are estimated to be

NATS currently has 258.1m pounds in budget reserve (op. cit., p13). The shortfall thus looks to be (to the nearest million pounds)

However, estimating actual costs of completion means estimating what has not been paid for, as noted above. Post-acceptance costs have not been paid for, and neither has the 81m-pound forward-development overrun (the difference between 298m-pound estimate and the 216.9m pounds paid). So this 81m pounds should be added in. The estimated actual costs to be paid post-acceptance are thus (to the nearest million pounds):

and the estimated actual budget shortfall (to the nearest million pounds) is thus

This shows that NATS themselves should be estimating a budget shortfall of 177m pounds, under their own assumptions; this estimate is itself 93m pounds short of the nearest estimate (Putnam's) from SE.

We may derive an estimate of the future cost-per-year of the NERC system from the total post-acceptance figure. Assuming that the system will be deployed in early 2002, as well as the assumption of a deployment for 20 years, the total system lifetime after formal acceptance will be 24 years. The average annual costs of the NERC will then be the total post-acceptance costs divided by 24, which are (to the nearest million pounds/year)

Notice that the planned cost of the system forward development at the time of bid was 216,9m/4 = 54m pounds/year (rounded to the nearest million pounds). Actual cost was 216,9/5.5 = 39m pounds/year. It follows that the costs of deploying and maintaining the NERC system for each of the next 23.5 years is estimated by SE to be between 0.4 and 0.98 times as much as was budgeted per year for its forward development, and correspondingly between 0.56 and 1.36 of actual average annual outlay over the elongated forward development period.

Three features of this discussion give cause for concern. First, there is a significant budget shortfall on all models, of 177m pounds (NATS's version) or at least 270m pounds (SE). Second, average annual costs of the NERC over its lifetime from here on out are at least 22m pounds (SE, unless one accepts NATS's 18% lower figure). Third, NATS's estimates for any of these figures are way more optimistic than the most optimistic estimate from SE.

This reasoning concerning costing estimates relied on certain assumptions. In order to gainsay the conclusions in the last paragraph, one could attempt to entertain doubts concerning the assumptions. Let me briefly consider such potential doubts.

The first major assumption is the fixed-proportion costing model. Doubt it if you like, but Barry Boehm is one of the most noted software engineering economics experts in the world, and wrote `the book' (Software Engineering Economics, 1981, which proposed the COMOCO model for forward development costing and planning). Balzer, Cheatham and Green's paper is a classic of software engineering. Their views were in part based on knowledge of the software development effort of the US Air Force up to that time. The Putnam Estimation Model is an early model, significant at and beyond its introduction, used sometimes in preference to COMOCO but less prominent nowadays. All these sources think it appropriate to suggest fixed-proportion models for crude estimates. All their figures represent significant experience. The differences between their estimates of the proportions could be explained by the differences in project types in their respective samples (this is, however, a supposition and I don't wish to present it as proven).

NATS has provided its own proportion, which is considerably more optimistic than either Boehm or BCG, but very roughly comparable with Putnam. Is NATS's estimation, that the forward development of the system would be a larger part of the total cost, and post-acceptance correspondingly a smaller part, at all plausible? In fact, the transition-to-deployment phase of the system, estimated by NATS as recently as this time last year to be one year, will last until early 2002 according to evidence recently presented to you, that is, it will take four years instead of one. When a planned one-year project phase takes four years, that cannot count as evidence that the post-acceptance phase is likely to be *less* costly than well-tried models predict. In fact, all experience shows that when forward development is harder than normal (as measured by the unusually high proportion of `bugs' in the original coding), post-acceptance costs are likely to be a higher proportion of total costs. This suggests to me at the least that Boehm's figure of 1:2 is likely to be more realistic than Putnam's 40:60. (On the evidence, I am myself more inclined towards a higher figure than Boehm's, maybe even BCG.)

Another assumption was that the project costs per year remained roughly constant over the forward development, but which took 5.5 years instead of the 4 years planned. In fact, in January 1996, the supplier appointed a cadre of senior engineers to the NERC project to `bring [it] back on track' (this information was offered to me by the Senior Vice President, Engineering of Lockheed Martin Air Traffic Management, Jack Clemons). This certainly cost more than simply letting the same level of formerly-planned resources continue for an extra 1.5 years. The conservative same-level assumption suggested that the forward development was underestimated by 81m pounds (to the nearest million pounds). It is thus likely that the true overrun cost of forward development will have been higher than this, leading to correspondingly higher estimates for the post-acceptance phase with all of these models. (On the evidence, I am inclined towards a figure of 85m pounds.)

Another assumption was that formal acceptance of the system occurred at the appropriate point. If formal acceptance occurred at a point earlier than appropriate (whatever that `appropriate' point might be), this would lead to an underestimate of the costs of post-acceptance (because one is setting the boundary point earlier, therefore more cheaply, than appropriate). The post-acceptance costs could only be overestimated by this assumption if the acceptance tests were in fact performed later than appropriate. Given the considerable discussion of the NERC project in the Committee's Fourth Report, and the somewhat adverse publicity surrounding this project, and the corresponding pressure to `deliver' which we may assume was there, I would invite the subcommittee to consider how likely it is that this was the case.

The final assumption made was that the NERC system will have an operational life of 20 years from deployment. From the point of view of costing calculations, I have no comment to make on this figure because I take it for my purpose here as a declaration of intent, not an estimate of anything.

The estimates above concerning the costs of the NERC software per year are not hard and fast calculations about how much the NERC will in fact cost. They represent rather an attempt to use public information offered by NATS, along with some of the most expert experience of software costing, and simple assumptions to reach some broad costing figures under optimistic assumptions. I have argued that these broad figures are likely to be conservative even under optimism, but even so give cause for concern. I suggest it would be in the public interest to inquire further about the costing of the NERC project.

My own intuition on the matters of overrun costs and likely total costs is as follows. First, given my general experience with the costing of engineers and engineering management in US companies, and given the pressure that was undoubtedly there to conduct acceptance proceedings in April 1998, I would consider an overrun cost of 85m pounds in forward development to be plausible. It could also be larger but I doubt very much whether it would be less than the 81m figure I have used here. Second, because the NERC system has had a troubled development history, I would estimate the proportions of forward development to post-acceptance costs to lie nearer the BCG model than the Boehm model. I would consider the Boehm model optimistic for this project. I emphasise that these are my intuitions. To establish more firm estimates, one would have to consider costing data in more detail.

Sincerely,

Peter Ladkin

Bibliography:

• Robert Balzer, Thomas E. Cheatham and Cordell Green, Software Technology in the 1990's: Using a New Paradigm, IEEE Computer, November 1983, pp39-45.
• Barry Boehm, Industrial software metrics top 10 list, IEEE Software, September 1987, pp84-85.
• Jack F. Clemons, Email letter to P. B. Ladkin, Wednesday 8 April, 1998.
• National Air Traffic Services, Chief Executive's Report, NATS Annual Report and Accounts, 1998.
• Putnam, L., A General Empirical Solution to the Macro Software Sizing and Estimating Problem, IEEE Transactions on Software Engineering 4(4):345-361, 1978.