Nuclear Instruments and Methods in Physics Research A293 (1990) 421-426North-Holland

Section XIII. The human interface

W

TNUKES A CONTROL SYSTEM USABLE?AN OPERATIONAL VIEWPOINT

Marc CLAYMP-6, Mail Stop H812, Los Alamos National Laboratory, Los Alamos, NM 87545, USA

This report discusses the generally accepted successes and shortcomings of the various computer and hardware-based controlsystems at the Los Alamos Meson Physics Facility (LAMPF) from an operator's standpoint. LAMPF currently utilizes three separatecontrol rooms that, although critically co-dependent, use distinct operating methods. The first, the Injector Control Room, which isresponsible for the operation of the three ion sources, the 750 keV transport lines and the 201.25 MHz portion of the linac, uses apredominantly hardware-based control system. The second, the LANSCE Control Room, which is responsible for the operation ofthe Los Alamos Neutron Scattering Center, uses a graphical touch-panel interface with single-application screens as its controlsystem. The third, the LAMPF Central Control Room, which is responsible for the overall operation of LAMPF, primarily uses atext-oriented keyboard interface with multiple applications per screen. Though each system provides generally reliable humaninterfacing to the enormously complex and diverse machine known as LAMPF, the operational requirements of speed, usability, andreliability are increasingly necessitating the use of a standard control system that incorporates the positive aspects of all three controlsystems.

1 . Introduction

From LAMPF's inception to the present day, con-trol-system development for a machine as large anddiverse as LAMPF has been an unending challenge tothe personnel assigned to the task . Through the lasttwenty years, the rapid and ever expanding capabilitiesof LAMPF have required the implementation of newerand more sophisticated technologies. The people whodesign, construct and program these systems have hadthe difficult task of finding, developing and implement-ing the best suited, most reliable and fastest systemsavailable for the money . They have also had the nearlyimpossible task of somehow making the systems usableto everyone who needs to use them.

What makes an operating system usable? Some mea-

sure a system's usability by its user-friendliness and

ease o ¬ operation . Others believe that a system is usable

if it is fast, efficient and flexible . At LAMPF, these

differing viewpoints have, through the years, resugicu in

the development of three distinct operating systems in

three separate control rooms . This paper discusses some

of the positive and negative aspects of the control

systems currently in use at LAMPF, with an emphasis

on the facility operator perspective.

* Work supported by the U.S . Department of Energy .

0168-9002/90/$03.50 n 1990 - Elsevier Science Publishers B.V . (North-Holland)

2 . Evolution of the Injector Control Room (ICR)

421

.°rior to the completion of the linac and experimen-tal areas, the operation of LAMPF was limited to onebuilding and one control room, ICR . ICR had thecapability of monitoring and controlling the H + ionsource, the Cockroft-Walton high-voltage power supplyand all the magnets and vacuum valves on the H +low-energy beam transport. This operation was almostentirely accomplished with the use of oscilloscopes (for

beam-current-monitor signals, dome-voltage transients,

etc .), DVMs (for magnet-shunt voltages) and hardware

binary and analog control modules (for vacuum valves

and magnets) . Although a remote indication and con-

trol system (RICE) was gradually being installed, which

would eventually allow remote operation of these de-

vices, there was no way to access the main control

computer from the ICR, so all the beam-transport de-

vices were controlled locally . The H + ion source and

Cockroft-Walton were, however, remoieiy conigvlled

by ineans of a NOVA computer, due to personnel safety

requirements.Even when a beam was first being accelerated by the

201 MHz section of the linac, the ICR operator still

controlled almost everything locally, now including,

along with the H + injector and transport, the magnets,

vacuum system and controls of the 201 MHz amplifier

systems . To maintain the beam, an ICR operator used

local controls distributed along a 100 m long equipment

"XIII . HUMAN INTERFACE

422

aisle . Clearly, with the bear. now being introduced intothe 805 MHz section of the linac (with its associatedhalf-mile long equipment aisle), the total hardware con-trol system was woefully inadequate .

Concurrently, the Central Control Room (CCR) wasbecoming operational . The heart of the control roomwas a Systems Engineering Laboratories SEL 840-MPcomputer. As this computer, along with RICE, becamefully operational, the operating emphasis shifted fromthe ICR to the CCR. Although a small group of oper-ators was now running the accelerator from the CCR, atleast one operator was always stationed in the ICR (asone is today) to monitor directly the operation of theion sources, their respective beam transports and the201 MHz amplifier systems .

The Injector Control Room and Injector Buildinghave remained primarily locally controlled even thoughCCR can now control almost all of the equipmentwithin the domain of the ICR operator .

3 . Evolution of the CCR

Unlike the ICR, there were almost no hardwarecontrols built into the CCR. The function of the CCRhas continued to be the remote control of the entirelinac and main experimental area proton beam lines .Initially, this was accomplished with an SEL 840 con-trol computer, a RICE Interface Unit (RIU) and theRICE itself. The system used was very similar to today'ssystem, i.e. a text-oriented keyboard interface with mul-tiple applications per CRT screen .

The operator could manipulate several different pro-grams with one keyboard and CRT. The displays ofthese programs could be monitored on either the CRTor on one of the several surrounding graphics scopes .Analog devices could be controlled from a series ofcontrol knobs located adjacent to the CRT. All of thesecomputer-hardware devices were arranged in a generallyefficient manner to allow relatively easy access to each.This arrangement of operator-controlled computerhardware devices was called a control console.

Other than significant improvements in the technol-ogy of the peripheral hardware, the only significantchanges to CCR operation have been the replacement..F s!�. OUT OAn W_tL .. AYva the SEL 840

l~il a v till O /UV änu lne addition of

VT-100-size touch screens.

4. Evolution of the LCR

The LAMPF Control System (LCS) had alreadybeen operating successfully for several years when in1973 the Weapons Neutron Research (WNR) experi-mental facility became operational . At that time it was

M. Clay / What makes a control system usable?

decided that the WNR would not be controlled by theCCR. This meant a new control room hhd to be built .

The designers of the WNR system incorporated manyof the aspects of the LCS. These included the text-ori-ented keyboard/CRT interface, the graphics scopes andthe control knobs. But, due to budget limitations, thedesigners' desire to use CAMAC instead of RICE andthe use of a different type of control knob hardware, theWNR control system was not quite as efficient as theLCS .

Before constuction of the Proton Storage Ring (PSR)began, the decision was made to build an entirely newstate-of-the-art accelerator control room . With this inmind, the design personnel chose a system that wouldbe based solely upon Eliographics touch screens. Thescreens would have color capability, would be muchlarger than their CCR counterparts, and would be pres-sure-sensitive, making them true touch panels .

All of these design criteria were implemented andare now being used to control the Proton Storage Ring,Los Alamos Neutron Scattering Center (LANSCE) andWNR beam delivery systems from the LANSCE Con-trol Room (LCR).LAMPF operations personnel have now taken con-

trol of the LC:R and plans are being made to implementan LCR operating console in the CCR, thereby unitingthe two operating systems .

5. Program interaction

First and foremost, with the LCS an operator needseither a good memory or the ability to develop a selec-tive memory . There are dozens of programs which canbe run from the CCR that require an operator toremember alphanumeric designators . This is not as dif-ficult as it sounds, however, for a number of reasons .

Firstly, the programmers have generally been verysuccessful in their use of mnemonics and abbreviationsas program names. For example, the main programused to monitor the status of the entire machine at anygiven time is addressed as "STAT". During the yearswhen the SEL 840 was in use, all program names usednumeric characters only . The "STAT" program, forinstance, was referred to as "734" . Though some didlike the old numeric names, alphabetic names are _aener-ally preferred .

Secondly, on a day-to-day basis, operators actuallyuse only a handful of all the programs available tothem . So the average operator only needs to rememberperhaps a dozen commonly used programs . Other pro-grams, if not remembered by one of the other shiftoperators, can always be looked up in a manual detail-ing all currently used programs . (As an aside, the pres-ence of this manual, as well as its continual updating, isan essential requirement in every control room.)

M. Clay / What makes a control system

Thirdly, the placement of a push-botton panel,labeled with the twenty or thirty most commonly usedprograms, directly above the main CRT has limited thenumber of programs one must remember. Pushing oneof the buttons automatically calls up the programlabeled on the button. An important feature of thispanel is that the program is merely called up and notexecuted. An additional "end hit" in the appropriatefield of the program display is required to execute anyfunction of the program.

Another method for accessing programs in the CCRis by the use of VT-140-size touch panels. These touchpanels have black-and-white displays that use coordi-nated infrared beams for picking up commands . Theyare not true touch panels as one does not actually haveto touch the screen to execute a command .

The initial touch-panel. display lists all the programswhich can be run from the touch panel . After touchingone of the programs listed, the program is called up onthe touch screen and can be executed with appropriateadditional touches. Programs manipulated from thetouch panels do not use the keyboard or CRT in anyway . Functions such as assigning analogue devices tocontrol knobs and hard-copy printing are done auto-matically from the touch panel .

Most operators, although approving of the idea ofthe touch panels, have been disappointed in theirhardware implementation for two reasons . Firstly, theprograms that can be operated from the touch panelscannot be operated. with the keyboard and CRT. Withthe SEL 840, the function of the touch panel wasperformed with a program card reader . Should anythingever happen to the card readers, any function that couldbe performed with the card reader could ,dso be per-formed from the keyboard and CRT, with menu-drivenprograms .

Secondly, with, the LCS touch panels, the. commandexecuted is all too often not the command desired.There appear to be two reasons for this. One reason isthat the touch panels frequently lose their calibration .The other is that the screens are just too small, therebyplacing the functions too close together . The problemswith the panels in the CCR are generally just an an-noyance, but during critical operating periods, e.g . peri-ods of extreme beam instabilities, beam spill or duringtune-up, poor touch-panel performance can be a realhindrance.

One recent innovation that could eliminate the LCStouch-panel problem has been the implementation of ascreen which uses a track ball to execute commands.The only difference in operation of the two panels isthat the one requires a touch and the other uses a trackball. This new scheme, which is currently being used tooperate a new LCS-style wire scanner program in theLCR, will, with a few refinements, greatly increase theoperational efficiency .

usable? 423

All programs available to operators in the LCR arelisted on each of the PSR touch panels. These programshave been grouped together as much as possible intosimilar functions, e.g., the six pulsed-power-system pro-grams can all be found in a group on the touch-panelmain menu.

The PSR touch panels work, at least outwardly, verymuch in the same way as the LCS touch panels . Onedemands the desired program by touching the box withthe program label. The touch-panel menu then disap-pears, giving way to the desired program menu .

However similar the PSR and LCS touch panels arein general operation, the usability of the two is verydifferent. As mentioned earlier, the PSR touch panelsare much larger than those in CCR. This size differencemeans that an equivalent number of LCS touch panelbuttons can be spaced much further apart on the PSRpanels, which means that, with larger room for error,touch-panel operation is not nearly so sensitive to smallchanges in calibration.

Another advantage of the PSR touch panels is theircolor capability. This capability provides some veryuseful beam-parameter monitoring programs. Two pro-grams in particular, the Loss and Beam Position Moni-tor programs, are perhaps the most useful of all of thePSR system programs because of this color capability.Each program uses differing color tones to indicatevarying degrees of spill or temperature. The black-and-white screens in the CCR (as well as the monochromegraphics scopes) simply do not have the capability todisplay these types of programs.

Clearly, operations personnel much prefer the PSRtouch panels to the ones in the CCR. There are, how-ever, a few disadvantages to their sole use as programmanipulators .

Firstly, unlike the LCS touch panels, a few programsin the LCR will begin running and executing functionsas soon as they are "called up" rather than waiting foran additional command. For example, a program whichhas some direct or indirect beam interaction or e x-ergizes some delicate equipment may be inadvertentlycalled up. If one of these programs were left un-attended, even for a short period of time, real problemscould develop. It has generally been proven to be betterto require additional operator action to execute anyprogram function after it has been called up initially.

Secondly, and by far most notably, once a PSRtouch-panel program has been called up on a screen, noother program can be executed simultaneously fromthat same screen . Even with the six touch panels pro-vided in the LCR, simultaneous operation of severalprograms can be very cumbersome .

This is where the LCS has a distinct advantage overthe PSR system . An operator can manipulate severalprograms simultaneously with one keyboard and CRTand a few graphics scopes. Because of the LCS's ability

XIII . HUMAN INTERFACE

424

to run several programs at a time from one CRT, theexperienced operator has much more flexibility and canoperate much more efficiently.

6. DSP vs scan screens

Earlier it was mentioned that a certain amount ofoperator memory is required for program manipulationin the CCR. That amount of memory is insignificantcompared to the amount required to operate a featureof the CCR system known as DSP .

DSP, which is the abbreviation for "display", is afeature whereby any data or device associated with theRICE or CAMAC can be displayed at any time on aline of the CRT by typing in the appropriate operatordesignator. All of the data and devices controlled by theremote computers have an operator designator associ-ated with them. For example, TDFilZI is the analoguecommand designator for the left fourjaw actuator inthe H+ low-energy transport . Another example isLDCM5D1. This is the analogue data designator for theaverage beam current in the fifth current monitor in thebeam line known as LINE D. 1here are only two of theseveral thousand operator designators associated withLAMPF.

If an operator wishes to monitor a certain parameteror manipulate a certain device and needs to know theoperator designator, there are three options available .An individual can either recall it from memory, tapsomeone else's memory, or look it up in one of themany large books containing all of the operator desig-nators known as the channel tables . (This is anotherbook that must be continually updated and left availa-ble to operators in the control room.)

Again, as with remembering the various programs,although difficult, remembering all of the designatorsone usually will run across is not as hard as it sounds .There is an easily recognizable pattern to the designa-tors, and there are methods by which one can rapidlymake a good educated guess.

With DSP one can change the state of binary de-vices, assign analogue devices to control knobs andmonitor the status of binary and analogue data in eitherengineering units or in raw counts (for example, 0-2000counts might be equivalent to 0-10 A on a magnet).'there are an infinite number of combination : of dataand devices that one can monitor at any one time withthe CRT screen . The only limitation to DSP is thenumber of available lines on the CRT.

An :aid to remembering the designators comes in theform of two programs that work in conjunction withDSP: °`SAV" and "RST". Once an operator has dis-played a desired combination of designators, it can be

M. Clay / What makes a control system usable?

saved with an alphanumeric name with the "SAV"function . Once saved, an operator can re-display thatdesignator combination with "RST".

With some experience, the DSP program is a veryuseful tool . When used in conjunction with the SAVand RST programs, DSP becomes perhaps the mostversatile and powerful tool available to aa operator.

The PSR counterpart to DSP is the scan screen .Most of the available buttons on the PSR touch screenmenus are for scan screens and not programs . Whencalling up a scan screen, one is greeted with a list ofoperator designators similar to those used in the LCS.There are two things one can do -with the displayeddesignator. It can be assigned to a control knob (if it isan analog device) or it can be "expanded" . The expandfunction brings up a new menu displaying the controland read-back states of the device and provides accessto other submenus for controlling alarm limits and soforth. There is no typing involved with the scan screensand one only needs to know the general location orcategory of a device to be able to call it up on a scanscreen .

This system is user-friendly and very simple . Withonly a vague knowledge of beam-line components andtheir locations, and with a little help, new operators canuse the s.:-an screens with relative ease almost im-mediately . The experienced operator, however, longs forsomething faster.

Going through the scan screens to execute the manycommands hidden within the operator designators' re-spective expand menus is tedious at best . Combine thiswith the fact that there are too many devices crammedonto a screen, thereby producing the same problem ofunwanted commands that the LCS touch screens areprone to, and the use of the scan screens can be quitecumbersome .A major drawback to the scan screen is the inability

to easily change the operator designator combinations .With DSP, an operator can easily obtain any desiredcombination of designators with a few simple keystrokes .With the scan screens, the designator combinations arepredetermined and written into a data file . Changingthe designator combinations requires a programmer toperform the nontrivial task of reorganizing and recom-piling the scar-screen files .

It has generally been accepted that DSP, in conjunc-tion with SAV and RST, is far superior to the scanscreens in speed and efficiency . It does, however, takesome time to learn the patterns of operator designatorsand to quickly manipulate the keyboard, whereas thePSR scan screens are easy to learn and require notyping . But an experienced operator with DSP canusually run circles around an equivalently experiencedscan-screen operator .

7. Bearn availability

The. ability to run the entire machine generally hingesupon the operation of the control computers. Whatwould happen if they should fail? This brings outanother deficiency in the PSR system . In the event of acomputer failure, the PSR system will not allow beamto be run. This is because of a system known as runpermit, which requires the complete operation of thecomputer in order to run beam. In the PSR system, runpermit is a program and therefore an integral part ofthe computer . In the LCS it is not .

In the CCR, the run-permit system is a hardwaresystem completely separate from the control computer.If the computer fails, the beam can remain on becausethe machine is still protected by the external run-permitsystem. In the LCR, with the ran-permit program downbecause of computer failure, it is not safe or possible torun beam . In the opinion of operations personnel,neither the run permit nor any other protective inter-lock should be part of the computer .

Another reason for this is that run-permit programglitches (or computer glitches) turn the bears off whenthere is no real problem . If the glitch does not clear� thebeam remains off until a programmer or computer-hardware person can correct the problem, even thoughno real beam-line-related problem may exist .

Yet another problem is that with a run-permit pro-gram a device monitored by run permit may go off andremain undetected until the computer completes its nextdata read. This could take up to 0.5 s and so couldcause severe problems. With an external hardware run-permit syste:., a run-permit drop turns the bearn offalmost instantly.

An external hardware run-permit system, regardlessof implementation (e.g . Allen-Bradley, binary-indicatorlogic relays, etc .) is much more reliable and increasesbeam availability .

S . Response

M. Clay / What makes a control system usable?

Operators rarely agree unanimously about anything,but one thing upon which they do agree its that at timestenth the i C v. anri the PSR cvstem . re nond tc1o slowly .--- ---- ___ ____ ____ _ __ _ _, __

_ _ _I .

,,

Unfortunately, there seems to be no clear-cut reasonwhy . Some speculate that it is a communication prob-lem between the control and remote computers. But noconcrete evidence exists which shows clearly th,rt anysingle piece of interface hardware is the cause . Othersbelieve that program priorities are the root cause . Stillothers consider the problem to be saturation of thecontrol-computer CPU by outside users . Regardless ofthe cause, the usability of a control system is directlyrelated to the response time of the system . As the

computer bogs down, operations bog down . It is suffi-cient to say that every effort must be made (and usuallyis, once the problem is known) to enhance control-sys-tem efficiency . This may include denying anyone exceptthe operating crew (and other essentials) access to themain control computers during normal operations (andespecially during; abnormal operations) . If the number-one priority is a. high-quality beaus at maximum availa-bility during scheduled production times, then this isnot an unreasonable request . The operators' generalresponse to any proposed change is, if it will make thecomputer run faster and reduce the response time, do it .

9. Reliability

This topic has been saved until the last, to discussthe last remnant of a hardware-based control system.As mentioned earlier, virtually everything within therealm of the ICR operator can now be controlled fromthe CCR, yet for some reason a person is still continu-ously stationed in the ICR. Why is that?

It is because there are some things which an operatorsimply cannot do from a control console. A CCR oper-ator cannot give undivided attention to one particular201 MHz rf stand or ion source. In fact, even if undi-vided attention were to be given by a CCR operator, itwould not be enough . A person actually standing at thedevice, resetting, monitoring and controlling it locally,is faster and more efficient than the CCR operator withan elaborate control system could ever be . With the rfstands and the ion sources still producing more beam-offtime than anything else, it has been decided that anoperator should continue to directly oversee and controlthe operation of these devices (in the case of the 201MHz rf stands, the only parameters that the CCRcontrols are the amplitude and phase set-points of the rffield in the accelerating module) .

The H- and P- (i .e . polarized H - ) ion sources arenow both controlled by Micro VAXs and CAMACleaving the H+ source control to be the last vestige ofthe old ICR control room. It is the only device now thatis not routinely operated from a computer . A hardwarepanel with binary and analogue command and measure-ment facilities, called an Injector Loral Control panel(ILC), is used to control the H + source and Cockroft-Walton .

The H - CAMAC system has had relatively frequenttroubles due to high-voltage arc-downs, and the P-system is fairly new and not yet tested, but thesystem hardly ever has a problem. "Mis does not meanthat a hardware system is necessarily more reliable thana computer system, but, according to operators, it seemsto be so in this case .

The H+ hardware console will be replaced by aMicroVAX and CAMAC system, similar to the othersources, by 1991 .

425

XIII . HUMAN INTERFACE

426

10 . Summary

As can be seen from this paper, operators believethat the LCS is superior to the PSR system in nearlyevery way . The LCS is much more difficult to learn andmaster and is not as user-friendly as the PSR system .JBut, with sufficient experience, an operator (even withminimal typing skills) can generally operate the ma-chine much faster and more efficiently with the LCS.

M. Clay / What makes a control system usable?

Combining the LCS with the PSR touch screens andtheir color graphics capabilities, keeping run-permit andother protective systems separate from the computerand restricting the use of the main control computerduring periods of production beam delivery wouldgreatly improve system efficiency and machine oper-ation .