Developing Robust Software

As one of many steps in developing robust software, Westinghouse Electric Corp. chose the Ada programming language for a Czech nuclear reactor's secondary shutdown system. The company needed to assure all interested parties, from the Czech Republic's regulatory agencies to its Western neighbors to the U.S. regulatory agency, that the system was accident-proof. The parties needed the assurance because some questioned the Soviet-designed plant's structural safety.

While in most of the world the specter of nuclear disaster is named "Chernobyl," in Czechoslovakia it is called "Bohunice." For over a decade, the four-unit VVER-440 plant in Bohunice suffered mishaps and accidents that culminated in April 1990 when the coolant level rose and caused the building to flood. Nearby Austria began handing out free potassium iodide (KI) tablets and lobbying internationally to stop the Soviet-designed plants from operating.

Immediately after the 1993 break with the Slovaks, the Czechs decided to continue the construction and retrofitting of another Soviet-designed reactor, a VVER-1000, in the town of Temelin, 60 kilometers (36 miles) from the Austrian border. Because of Bohunice, which is now part of western Slovakia, the Czechs needed to earn Western European confidence in Temelin's safety. Otherwise, obstacles to international licensing and financial backing could frustrate their goals to generate 50-percent of their national power through nuclear energy by the millennium, and to replace the high-sulfur domestic brown coal whose smoke has laid waste to a third of Bohemia's forests. In 1994, Westinghouse signed a $419-million contract to provide new instrumentation and control (I&C), to design new fuel and a reactor core, and to supply U.S.-manufactured fuel. Despite Austrian opposition, the U.S. Export-Import Bank awarded the company a much-needed $317 million loan guarantee.

Although the Czech utility Skoda is in charge of testing and licensing the I&C system, the company has taken steps to assure the neighboring countries that Temelin's software could pass U.S. Nuclear Regulatory Commission (NRC) standards. Most of the I&C system is already licensed in Europe: Westinghouse is reusing much of the software that it previously installed in a Sizewell, England, plant. Also, the NRC has trained Skoda technicians in licensing and testing reactor software.

While much of the Westinghouse software is reused, the secondary shutdown system is completely new. Both the primary and the backup protection system contain three redundant divisions as a safeguard against error. However, the same software bug could simultaneously bring down all three divisions. In order to raise the odds to impossible of that happening simultaneously in both the primary and secondary systems, Westinghouse redesigned the backup system's architecture with different software and hardware. For the software, the Monroeville, Penna., division decided to program the backup shutdown system in Ada; the primary shutdown system is in PLM-86.

Why choose Ada?

Westinghouse came to its decision by conducting a survey of languages and narrowing the field to Ada and C. It chose Ada because of several factors. First, the Ada programming language is an international standard (ISO/IEC 8652). If Westinghouse wins a similar contract in Poland, for example, the Czech software should successfully compile on the different or newer computer system.

Second, Westinghouse chose Ada because the behavior of an Ada program can be deterministic at the lowest level through logical access of the target hardware's basic elements, and through control over the visibility of types, operations, and data. Third, Ada featured strong type and range checking. Fourth, those features and its modular construction had already made it attractive to other organizations for safety-critical applications. The language is used extensively by international airplane manufacturers, the U.S. Federal Aviation Agency (FAA) and Department of Energy, and many nations' organizations for air traffic control and aerospace exploration. As a result, government agencies had already subjected Ada software-development tools to stringent tests for safety.

Westinghouse found such a tool through the compiler manufacturer Thomson Software Products (now called Aonix). Thomson offered a run-time system, C-SMART, that was used on several of the Boeing 777 aircraft's systems, including the brakes and the power ignition. The system was therefore already documented for safety-critical applications in accordance with FAA standards.

Westinghouse's I&C system will control the Temelin reactor's everyday operations as well as protect the plant from accidents. A Unit Information System (UIS) processes the data from Temelin's control and protection systems via a WESTNET highway, which is a standard high-speed redundant Fiber Distributed Data Interface. Its redundancy ensures that one data highway can fail without crashing the system. The highway distributes the data to a technical support center, and to the main and emergency control rooms, which have switches that communicate directly to the two shutdown systems.

The secondary protection system monitors the sensors of various conditions in the plant, such as temperatures, flows, and pressures, to detect if they are below or above a set point. If an acceptable range is exceeded, then the Ada-driven system sends a signal to the non-logical programming (NPL) control room, where software is embedded in the hardware. The primary system also sends a signal to the NPL. If the two systems agree, then the NPL implements one or more of its nine reactor trip functions, which usually means closing or opening a valve or a pump. If they disagree, then the NPL decides which signal to obey according to a set of logical choices that cannot be changed ^since they have been burned into the hardware.

The digital backup protection system consists of three protection cabinets, two monitoring cabinets, and a Sun SPARC laptop computer. The cabinets receive analog and digital input from the sensors of water pressure, level, and temperature in the reactor's core. The cabinet's microprocessors perform the required calculations and write to a digital NPL output board, which connects the cabinet to the pumps and the valves. The protection cabinet contains three VME-bus backplane chasses. One chassis contains three processor boards; the other two chasses are made up of input/output (I/O) boards. Inputs from the monitored values come into the I/O boards, from which the processor boards read the I/O data.

Each processor board has a 68040 microprocessor, three for each of the backup system's three protection divisions, which equals nine, plus one 68040 board for each of the two monitoring systems. The five cabinets in the system communicate via a fiber-optic network. The three processor boards in a protection cabinet communicate with each other via the VME bus. Two of the processor boards perform safety tasks, while the third board performs testing on the cabinet. This third board is used to communicate with the portable SPARC notebook.

Normally, an embedded system of this type has no graphical user interface (GUI). The SPARC laptop, however, provides a maintenance interface, through which it displays a graphical picture of the algorithms using the real data, as well as the calculated results. Text fields display the actual values, such as the analog inputs from the system. If the data goes through a comparitor, then the technician can check whether the system is operating correctly while the display shows the result.

The portable SPARC notebook's other maintenance features include the ability to change the set points at which the comparitors decide to take some action. For example, if the analog input reads a temperature at 500 degrees Celsius, which the comparitor has set as too high, then the technician can theoretically change the limit to 510 via the SPARC laptop. The interface is programmed in C using a GUI-builder.

A different architecture and a different language were not the only ways in which Westinghouse satisfied the ambiguous requirement that its secondary protection system be "diverse." The software development laboratory also hired a new staff of programmers who were unfamiliar not only with the other systems' software design, but also with nuclear plants and with Ada. "We purposefully went after programmers with no little or no experience in nuclear-reactor software," said Jeff Pike, the diverse protection system's lead engineer. "We wanted a completely clean slate." All the new programmers had a core knowledge of nuclear physics and experience in other languages, such as C and PLM. After a week of in-house training in Ada, the programmers began designing the new system from scratch.

The secondary backup system developers decided to use a combination of object-oriented design (OOD) and structural analysis. While OOD is a current popular buzzword that programmers often equate with clear and easily maintainable software, in safety-critical applications it has drawbacks. In order to control the complexity of a solution, for example, OOD programmers will conceal unnecessary details throughout an algorithm. Such information hiding is almost forbidden in many safety-critical software requirements. When a piece of software's failure in a new airplane would be "catastrophic," for example, then the FAA requires that every line of code be demonstrably executed. In other words, do not hide anything. For Westinghouse's secondary shutdown system, the engineers used the design techniques that characterize Ada and OOD in order to facilitate the separately developed software's integration. Some information is hidden, such as the I/O, so that other components cannot mistakenly corrupt the data.

The Westinghouse software developers further ensured that the Ada software would meet safety-critical criteria by choosing the Thomson C-SMART library and its standard cross compiler, which translates the host's code into Motorola 68040 executable software. When used together, the compiler and C-SMART library flag and do not compile code that uses Ada features that are frowned on in safety-critical applications. Tasking is forbidden, for example, which can communicate data from a dozen sensors simultaneously, because the timing of executed code is not set in granite.

C-SMART links in code to perform library functions and runtime software. It is drastically smaller than Thomson's standard library, for the company has stripped out anything that is not deterministic or cannot be verified for safety. The runtime software has the same documentation and level of testing as the Westinghouse I&C software. C-SMART's documentation for Lockheed's new Hercules aircraft, for example, ran to 68 kilograms (150 pounds) of paper for a runtime system of only 6800 source lines of code (SLOC). The documentation included tests, test results, source code, etc., and contained over 3,000 signatures.

The runtime system's documentation met the FAA's criteria for safety-critical applications but not necessarily those of the NRC. Westinghouse's developers first created a cross-reference document that mapped the FAA's requirements to the IEC 880 standard, "Functional Safety: Safety Related Systems." Thomson produced a document that traced each requirement in the IEC 880 standard into the material developed for FAA certification to enable the Nuclear Certification Agencies to find the information they need. The compiler company also added 50 extra test cases for boundary conditions that the FAA did not require, which Westinghouse's verification and validation department is currently reviewing and spot checking with retests. Because the safety-critical runtime software already satisfied most of the reactor's requirements, the companies were able to concentrate their efforts on creating and running tests rather than fixing code.

After another round of editing the documentation and implementing any customer changes, Westinghouse will try to ship its new I&C system in November 1997. In the fall of 1996, the company started to test, verify, and validate its new backup protection software, which will be fewer than 100,000 SLOC. Testing for the entire system should be completed in May 1997.

The Temelin power plant still has hurdles ahead, and has drawn opposition from inside as well as outside the new Czech republic. Moreover, the requirements for the secondary shutdown system have been volatile, and continue to change as the Czechs fit Western standards into a Soviet design.

Despite the moving baseline, the programmers are integrating the individual parts of the new modules into a complete system, which Ada's strong typing has made easier than they expected. Pike said that, unlike his experience with C integration, he does not worry that another developer "was supposed to pass me three parameters and only passes me two and they're of a different type."

"At this point in the development, most people see that Ada has significantly cut down on the integration time," Pike said. "Problems are in logic mistakes or requirements that are not lined up at the system level, more than mismatches that you would find if not using a strongly typed language."

In time, Westinghouse expects that Skoda and the Austrians will share the programmers' confidence in Ada's strength and safety.