HPS Slow Controls: Performance and Future

N. Baltzell HPS Collaboration Meeting

November 16, 2016

1

HPS Controls Framework

•  EPICS R3.14.12.5 –  Mostly softIOCs (RHEL7) –  A few VME crates (vxWorks)

•  beamline motors/scalers

•  Control Screens –  Motif-based medm –  A few good summary screens –  Lots of detailed screens

2

•  Alarms –  Motif-based ALH –  Audible/Visual Alarms + Emails

•  Strip Charts –  StripTool (old & loved) –  MyaViewer (archive access)

•  Time Histories –  archived with JLab’s Mya

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Overview

3

The  main  overview  screen  

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Overview

4

Beamline  Strip  Charts  

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Overview

5

Alarms  

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Overview

6

ECAL  FADC  Scalers  

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Overview

7

Trigger  Rates  &  Live8me  

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Upgrades for 2016 Run

8

•  ECAL/SVT Summary Screens •  Chicane Controls Simplified •  Neutron Counters in EPICS •  ECAL LV & FADCs in EPICS •  Automatic BTA (ABU/BANU) •  SVT FSD-Only Bias Interlock •  Harp Scan Quirks Fixed •  Added Hall-B collimator controls •  Left-click Bug in StripTool fixed •  Watchdog Squares •  DAQ Reboot Buttons

Lessons  Learned  JLab  Security  scans  can  freeze  vxWorks  2H02  Harp  got  stuck  –  need  yearly  maintainence  

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

9

(MaS  Solt)  

Example Upgrades for the 2016 Run Detector  Summary  Screens  

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Example Upgrades for the 2016 Run •  Chicane Controls

–  New sequencer to ramp up to max, hold to saturate for N minutes, then ramp down to setpoint

•  Status lights & messages & progress meter for shift operator

•  Setpoint can be calculated based on beam energy via a button click, based on the Bdl calculations

•  Setpoint is saved for next time chicane is turned on

–  No shifter’s manually fiddling with and entering current setpoints

–  Worked •  except when magnet trips, then

fallback to old method (only happened once in 2016 run)

10

•  Automatic BTA –  Uses 2C21A nA BPM current and

Halo FSD counts to determine beam availability, and DAQ livetime and trigger rate to determine HPS usage

à BNA or BANU or ABU

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Security Scans during the 2016 Run •  JLab security scans had been disabled on the slow controls subnet for

year(s) and our vxWorks hardIOCs had run stably since the HPS 2014 run

•  One weekedend during the HPS 2016 run they started misbehaving •  Investigations were complicated by a simultaneously failing network switch

•  Switch was replaced, and remote reset implemented on hardIOCs (as a fallback) during a weekend controlled access

•  The next workweek the security scans were discovered and terminated….

•  While investigating what turned out to be caused by security scans and failing network switch

–  Removed the cause of the long pause at beginning of harp scans –  And added scan abort button in case beam trips early in the scan

11 Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Performance Summary

12

•  Many upgrades between 2015 and 2016 runs •  A couple issues resolved during the 2016 run •  Result is a well performing, reliable system

with easy usability by shift workers

Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016  

Many  people  contributed  to  the  improvements  …  

Future •  CLAS12 is coming online

–  HPS IOCs run in parallel •  currently separate software builds •  although based on same OS and all dependency versions

–  CLAS12 is using a new frontend user interface •  CS-Studio

–  allows faster development for big systems, dynamically generated screens –  integrated alarm handler frontend, plotting tools –  server/database alarm/notifier/messenger/log system –  medm requires tons of copy/paste, manually fidgeting with widgets, writing scripts to generate static

screens –  Motif (medm/alh) support is dying

»  actually OSX just had a recent Xquartz release that accidentally “broke” Motif completely

•  By the HPS 2018 run –  ECAL improvements

•  LV GPIB-ETH converter will need blocked network port (new heavy traffic) •  HV/LV Interlock based on Chiller/Temps

–  SVT Bias à Assert DAQ Busy (maybe) –  have a merged CLAS12/HPS software build for IOCs (“easy”) –  ideally convert HPS screens and alarms over to CS-Studio

13 Slow  Controls:  Performance  &  Future  –  HPS  Collabora8on  Mee8ng  –  November  2016