White Paper: Synchronization for High Frequency Trading Networks

www.spectracomcorp.com 1 | Timing & Synchronization White Paper

For many financial institutions, high frequency trading volume is growing at an accelerating pace and demanding new requirements on their IT infrastructure. Drivers in their business such as pricing of equities moving from decimal to penny resolution and the growing need for markets to provide improved liquidity are resulting in huge opportunities for financial gain. Taking advantage of these opportunities is, in part, dependent on the care taken in the network’s time synchronization and the management of latency. Wall Street firms who were involved in the early phases of High Frequency Trading have been early adopters of high performance timing solutions utilizing a variety of signals including GPS, IRIG, 1PPS, NTP and now the Precision Time Protocol (PTP) which allows for precision time transfer on Ethernet networks. The implementation of specific timing solutions depends on the trading infrastructure and the network topology. Through a combination of hardware, software, and careful network management, it is reasonable to expect microsecond level time-transfer from traceable time sources to Linux applications. Introduction IT managers in financial services institutions, especially those who are participating in high frequency trading, find themselves facing multiple converging and difficult to meet system-level requirements:

• Surging network traffic in capital markets requiring accurate time-stamping of market data feeds which can approach a pace of over one million per second.

• Processing data and making trading decisions in an ever more real time environment with high performance trading algorithms requires precision timing within the Linux kernel and software application.

• Pressure to reduce trading latency and transaction times to sub-millisecond levels across a geographically disperse set of trading servers and exchanges.

• Increasingly stringent regulatory oversight requiring high precision timed log files. • Ability to post-process nearly any network scenario for optimization.

Spectracom is able to provide a time synchronization solution for financial institutions: precision time sources referenced to worldwide time standards, a variety of time distribution protocols and standards, important precision time transfer to trading server operating systems (then into the application), verification solutions and portable clock calibrations where GPS is not available. The specific application of these products is dependent on the needs of the scenarios that are common for financial institutions now participating in High Frequency Trading. However, there are several common approaches such as using GPS as a traceable time source, the new PTP protocol for precise time transfer over a LAN, and sophisticated client software to improve precise time transfer to the application within the Linux environment. Legally Traceable Time Through GPS An important aspect of any time synchronization deployment is the notion of accurate, traceable time, versus relative time. Time is absolute and one of the seven legally-defined units of measure.

www.spectracomcorp.com 2 | Timing & Synchronization White Paper

Since 1972 the world’s time standard is UTC (coordinated universal time). It is based on atomic clocks maintained by National Metrology Institutes and is distributed in a number of ways. The most accurate and simplistic method (when afforded a clear view of the sky) of synchronizing to traceable time is through the GPS broadcast. Spectracom offers synchronization to GPS, in addition to a number of other time sources, to transfer time to where it is needed in many different ways. Because of the myriad of choices of timing references and timing signals, Spectracom offers SecureSync®, a network-based master clock that is feature-rich and configurable so you only pay for the functions that you need. Configured with a GPS receiver, SecureSync, like any other GPS-based timing hardware, offers better than 50 nanosecond accuracy to UTC. A good start for precision timing. A Combination of PTP Hardware and Software Improves Synchronization Leveraging the network is a low-cost way to synchronize multiple servers (blades or 1U appliances) in the trading network. This has been done for years through network time protocol (NTP). Today, a new network protocol is available to offer improved time transfer over the network. Precision time protocol (PTP) uses hardware time-stamped messages to measure the delay between the master and slave. The result is better accuracy when adjusting the slave clock for minimum offset compared to the master. A typical SecureSync configuration combines a GPS receiver and a PTP module. This “PTP grandmaster clock” is used to synchronize PTP slaves (clients) within the local area network with high accuracy when variation in bi-directional packet delay is low. Even under the worst case scenario, PTP performance will be equivalent to NTP. An example of a PTP slave is the Spectracom TSync-PCIe-PTP time code processor card. The PCI express card has an integral LAN port to send and receive time-stamped PTP messages across the network to synchronize an on-board clock with 4 nanosecond resolution. This PTP slave offers the best possible time precision for a computer that can accommodate plug-in cards. Another example of a PTP slave is TimeKeeper Linux PTP client software. The software is a transparent and very lightweight “applet” that does not alter the Linux kernel and does not require modification of existing applications. It responds to the standard Linux time calls such as gettimeofday. However it greatly improves the accuracy of timing functions in a number of ways:

1) Improves the accuracy of the local clock by disciplining it to the PTP time reference. 2) It quickly and smoothly converges local time to the master time with a typical offset less than 10 microseconds. 3) Time is monotonic and never goes backward which can occur with standard Linux system clocks.. 4) Time is computed without leaving the processor, therefore, is immune to interrupts and other operating system

“overheard” that can delay time reads and affect timing accuracy. 5) Allows the application to build a high precision time-scale by making the current time available every 30-50 nanoseconds.

Specialized time synchronization hardware improves TimeKeeper accuracy by a factor of 5. Therefore the best performance is achieved with a combination of TimeKeeper PTP client software and the TSync-PCIe-PTP card. The following describes the implementation of synchronization solutions in several scenarios using GPS, PTP, hardware master clocks, PCI express cards, and timing management software. Also included are solutions for GPS-denied environments and NTP improvement software. Typical Network Scenarios Scenario I: Time synchronization within a local trading center, or between trading centers. Assumptions:

• Timing packet delay variation (PDV) is within acceptable limits to produce the required time synchronization performance

• GPS is available in all trading center locations

• Most servers requiring synchronization are blade servers, however, some critical 1U servers require the highest level of synchronization

• All servers are running Red Hat enterprise Linux

www.spectracomcorp.com 3 | Timing & Synchronization White Paper

Spectracom Solution: • Network Master Clocks: SecureSync® PTP Grandmaster (redundant pair)

• Linux Time Client: TimeKeeper™ PTP Client

• Local Hardware Clock/PTP Slave (for 1U application servers): TSync-PCIe-PTP plug-in card

• Synchronization Hardware Testing: STA-61 Sync Tester/Analyzer

As critical infrastructure, it is typical to deploy network master clocks as redundant pairs. Two SecureSync GPS PTP grandmaster clocks require their own GPS antenna (and RF cabling) with a clear view of the sky. PTP includes a “best master clock” algorithm so no special switches are required to accommodate a failover from one SecureSync to the other. Special care is required to minimize packet delay variation (PDV) by eliminating as many indeterminate time delays in the network as possible and practical. Timing accuracy is degraded due to timing packet path asymmetry. Use low traffic or directly connected PTP masters and slaves via hubs since switches and routers queue messages depending on network conditions. Alternatively, special PTP-enabled routers can be deployed. Known as PTP transparent clocks, they are able to compensate for any delay of PTP messages. PTP slaves are deployed as PCI express cards for 1U application servers that can accept such hardware, and PTP Linux clients. Timing performance across the network can be analyzed by measuring the phase of the 1 pulse-per-second signals from the PTP grandmaster and the PTP hardware slave. This configuration can be duplicated across all data centers involved in the trading scenario. When each LAN is synchronized to a traceable time source via a local master clock, then any data passed from one LAN to another can be perfectly correlated – extremely important when managing latencies.

Sidebar: Table 1 provides typical performance of a variety of time transfer protocols assuming minimal asymmetry between the PTP grandmasters and slaves. Table 1: Time Synchronization Performance of GPS-based Time Transfer Technologies

PTP w/ SW timestamps via TimeKeeper alone

PTP w/HW timestamps via PCIe card and TimeKeeper

NTP w/existing standard NTP Server

4-5 microseconds 1 microsecond 10 milliseconds

www.spectracomcorp.com 4 | Timing & Synchronization White Paper

Scenario II: Time synchronizing within a GPS-denied trading center. Assumptions:

• Either the remote trading center is connected to another network with access to GPS and the variable path delay between networks is managed to minimize asymmetry or,

• A procedure can be followed to calibrate a local master clock to GPS to keep it within the desired accuracy limits

Spectracom Solution: • Network Master Clock: SecureSync PTP Master/Slave

• Linux Time Client: TimeKeeper PTP Client

• Portable GPS Calibrator: GPS-12R Time-Transfer Standard

www.spectracomcorp.com 5 | Timing & Synchronization White Paper

In the case of a network without access to GPS, there are 2 choices:

1) Manage the variable packet delay from a network with access to GPS (scenario 1) through the use of PTP-enabled infrastructure, or a directly connected dedicated low-traffic timing network, so a PTP slave can achieve accurate time from another network’s GPS PTP grandmaster.

2) Routinely calibrate a local PTP master using a “flying clock” portable calibrator. As previously mentioned asymmetric packet delay degrades the accuracy of PTP so it offers no advantage over the existing NTP infrastructure. One approach is to manage the packet delay variation. A local SecureSync can be configured with two PTP modules; one to act as a PTP slave and one to act as a local PTP master for all the other clients on the network. If the PDV is unmanageable, then the local SecureSync should be configured as an “atomic clock” with a Rubidium oscillator to provide the best timekeeping performance in the absence of continuous access to a traceable time source. (Configuring a SecureSync with a Rubidium oscillator is also a redundancy option in GPS-accessible environments to protect against failure of the GPS antenna system, RF cabling, or the entire GPS system itself.) The drift rate of a Rubidium oscillator is on the order of a part in 10 to the 11th (<10 microseconds per day) and is several orders of magnitude better than crystal oscillators even when combined with techniques to reduce the effect of temperature variation. While a Rubidium clock solves the “relative seconds” problem, traceability, or accuracy, to absolute time is required so events on the local network can be correlated to other networks for managing latencies, etc. Spectracom offers the model GPS-12R battery-powered GPS time-transfer standard to be used as a “flying clock”. Simply synchronize the unit in an area that can receive GPS signals while connected to AC mains. Then transport it to the SecureSync unit under its internal battery power (2-3 hours, but a +12VDC option extends transportation time by using a vehicle battery). Then connect it to the SecureSync and reconnect mains power. The SecureSync’s internal Rubidium oscillator will use the GPS-12R 1PPS signal as its primary reference. Once it is synchronized, disconnect the GPS-12R, and the SecureSync will resume in an accurate “holdover” condition until the next calibration cycle. Sidebar: The following table below gives you the typical performance you will be able to achieve synchronizing remote locations held over with Rb oscillators.

Table 2: Timing Performance between remote sites

Timeframe

Between two sites both running GPS-based servers constantly

One site running on servers with Rb Oscillator holdover

Between two sites both running Rb servers with Rb holdover

3 months 50 ns + PTP time transfer < 1 ms < 2 ms 6 months 50 ns + PTP time transfer < 2 ms < 4 ms 12 months 50 ns + PTP time transfer < 4 ms < 8 ms Perpetual 50 ns + PTP time transfer <10 us/day < 20 us/day Where: ms = milliseconds us = microseconds ns = nanoseconds

Scenario III: Optimizing Time Transfer to a Server Rack Assumptions:

• GPS is available in the trading center

• A low traffic, highly symmetric, dedicated timing network can be established between the GPS master clock and the server rack or the network is PTP enabled

• All servers are running Red Hat Enterprise Linux

www.spectracomcorp.com 6 | Timing & Synchronization White Paper

Spectracom Solution: • Network Master Clock: SecureSync® PTP Grandmaster

• Server Rack Master Clock: SecureSync PTP Master/Slave

• Linux Time Client: TimeKeeper PTP Client

Similar to scenario II, a SecureSync GPS clock PTP grandmaster can accurately synchronize another SecureSync through a low PDV network. This allows for physical separation of the GPS grandmaster and the server rack from a GPS accessible location to the “top-of-rack”. Network infrastructure can be used to transfer time via PTP. The top-of-rack SecureSync acts as a slave to the GPS grandmaster and a master for all servers in the rack. The rack becomes a “mini timing network” to eliminate the PDV in the time transfer between PTP master and client/slaves. Scenario IV: Improving NTP time synchronization within a local trading center Assumptions:

• NTP is preferred over PTP

• There is a desire to improve the accuracy of time transfer

• All servers are running Red Hat Enterprise Linux

Spectracom Solution: • Network Master Clock: Existing GPS NTP Server

• Linux NTP Time Server: TimeKeeper NTP Server Software

• Linux NTP Time Client: TimeKeeper NTP Time Client

NTP was designed for wide time distribution across many devices and networks in a cascading hierarchy. Accuracy of NTP time transfer can be greatly improved by optimizing the algorithm for point-to-point synchronization. A version of TimeKeeper does just that. Install this software application in any Linux server. Synchronization to a traceable time source is achieved from a timing signal through hardware such as a 1PPS signal from existing GPS timing hardware. The TimeKeeper NTP client synchronizes to the NTP server in a similar way as the PTP client. TimeKeeper is offered in a combined NTP/PTP client, so a new PTP network can be overlaid on an NTP network. The same principals for managing packet delay variation apply to the NTP configuration as well as the PTP configuration.

www.spectracomcorp.com 7 | Timing & Synchronization White Paper

Conclusion Financial institutions, hedge funds and market-makers need to address a variety of challenges with accurate, traceable time for their critical networks and network elements. High frequency trading requires real-time decisions based on microsecond time accuracy in order to remain competitive. This level of accuracy is available today through a combination of a GPS timing, new network synchronization protocol (PTP), PTP masters and slaves, time-transfer Linux software, time calibration for GPS-denied environments and/or carefully managed networks, and measurement solutions.

USA | 1565 Jefferson Road, Suite 460 | Rochester, NY 14623 | +1.585.321.5800 | sales@spectracomcorp.com FRANCE | 3 Avenue du Canada | 91974 Les Ulis, Cedex | +33 (0)1 64 53 39 80 | sales@spectracom.fr UK | 6A Beechwood | Chineham Park | Basingstoke, Hants, RG24 8WA | +44 (0)1256 303630 | info@spectracom.co.uk

June 8, 2011 - WP06-101 (A)