Post on 18-Apr-2015
FAST Protocol - FIX Adapted for Streaming
FAST Protocol – Bandwidth and low latency
Kevin Houstoun
Factors
• Speed of light
– 299,792,458 m/s
– 299 m/us
– 670,616,629.4 miles per hour
– 1 ft / nanosec
• Rapid Addition
– FIX ~5 microseconds
– FAST ~ 1 microseconds
• Measurement
• IP Stack
– TCP/IP
– Windows 30 microseconds
– Windows HPC 8 microseconds
– UDP
2
• People v’s Computers
– Computers nanoseconds
– People 140 - 200 milliseconds
• Type of business– Solutions
–Smart Order Routing–Proximity Hosting–DMA–Co location
– Care Orders– Active Trading– Human Involvement?– Credit / Limit checking– Booking
–Allocation–Confirmation
• Connectivity– FIX Hub– FIX Direct– IP
–Internet–Private–Switches, Routers and Hubs
3
Role of Market Data in Electronic Trading
• The trend toward electronic trading is creating an explosion in market data volumes and peak message rates which present a challenge for even the best technology
– Improved and ubiquitous trading technology is responsible for the rapid upward spiral in market data volume
– Broader use of standards have created a “quick and easy”way to connect to the market
– Market participants around the world make critical decisions on real-time information
– Without timely market data a trader is at an extreme disadvantage
– The pervasiveness of electronic trading is a tremendous stride forward but it also presents equally tremendous challenges
4
The market data catalyst
Market DriversBest executions
Direct market accessNew Products
New ParticipantsInnovative Trading Strategies
Market Structure ChangesRegulatory (Reg NMS, MiFID)Dynamic pools of liquidity
Innovation in Technology
Electronic TradingAlgorithmic ModelsReliance on Market Data for Decision MakingUbiquitous Trading Technology
Market Data
5
Peak message rates outpacing effective solutions
• Business demands are outstripping communications
services and cost-effective bandwidth solutions
– More products, more listings,
– More price levels, more depth/data
– Increased demands for low latency
• Proprietary solutions are increasingly costly to
maintain
• The industry is ripe for a radical shift
Financial Information Forum Area Chart
U.S. Equities Exchange Feeds - 1 Minute Peaks (Messages Per Second)
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
Dec
-01
Jan-
02Fe
b-02
Mar
-02
Apr
-02
May
-02
Jun-
02Ju
l-02
Aug
-02
Sep
-02
Oct
-02
Nov
-02
Dec
-02
Jan-
03Fe
b-03
Mar
-03
Apr
-03
May
-03
Jun-
03Ju
l-03
Aug
-03
Sep
-03
Oct
-03
Nov
-03
Dec
-03
Jan-
04Fe
b-04
Mar
-04
Apr
-04
May
-04
Jun-
04Ju
l-04
Aug
-04
Sep
-04
Oct
-04
Nov
-04
Dec
-04
Jan-
05Feb
-05
Mar
-05
Apr
-05
May
-05
Jun-
05Ju
l-05
Aug
-05
Sep
-05
Messages p
er
Second
CTS UTDF CQS NQDS UQDF ArcaBook TotalView
Source: Nasdaq, SIAC, Arca
OPRA Peak 1 Minute based on SIAC data
(FIF Projections using Past Average % Change of 5.9%)
-
25,000
50,000
75,000
100,000
125,000
150,000
175,000
200,000
1999
Feb-
00A
pr-0
0Ju
n-00
Aug
-00
Oct
-00
Dec
-00
Feb-
01A
pr-0
1Ju
n-01
Aug
-01
Oct
-01
Dec
-01
Feb-
02A
pr-0
2Ju
n-02
Aug
-02
Oct
-02
Dec
-02
Feb-
03A
pr-0
3Ju
n-03
Aug
-03
Oct
-03
Dec
-03
Feb-
04A
pr-0
4Ju
n-04
Aug
-04
Oct
-04
Dec
-04
Feb-
05A
pr-0
5Ju
n-05
Aug
-05
Oct
-05
Dec
-05
Feb-
06A
pr-0
6Ju
n-06
Aug
-06
CME Bandwidth Utilization Trend
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
5000000
5500000
6000000
6500000
7000000
7500000
8000000
8500000
12/29/2004 2/17/2005 4/8/2005 5/28/2005 7/17/2005 9/5/2005 10/25/2005
Bandw
idth
(m
bits)
Moving Avg Avg
Moving Avg Peak
Daily Average
Daily Peak
6
Challenge of Market Data Growth - OPRA
Source: Financial Information Forum, Market Data Capacity Committee
7
Challenge of Market Data Growth - US Equities
8
FIX Response to an Industry Problem
2004
– Q4 – Market Data Optimization Working Group (”mdowg”) formed
• 2005
– Q1 – Initial protocol proposals submitted
– Q2 – Proof of Concept (”POC”) development
– Q3 – POC testing performed
– Q4 – POC results released
• 2006
– Q1 – FAST 1.0 specification released
– Q2 – FAST 1.1 development work started
– Q3 – FAST 1.1 specification development
– Q4 – FAST 1.1 specification released
The Financial Services Industry faces seemingly ever growing market data bandwidth requirements, existing open standard are unable to cope with this volume and the industry faces developing another round of proprietary standards.
9
MDOWG Charter & Objectives
• What are the challenges of engineering a new Market Data Solution?
• Develop a comprehensive solution that will mitigate exponentially increasing market data volume across all industry sectors
• Re-engineer Market Data so that it is no longer the weak link in the trading cycle
• Provide real-time, low-latency feeds that can be effectively delivered to the average subscriber
• Shrink the gap between trading technology and market data technology
• Plan for advances in bandwidth capacity but hedge your bets
• Create a common, flexible, straightforward solution that will reduce integration costs
10
What are key aspects of developing a global market data solution?
• Architecture – Scalable, fault-tolerant Architecture that minimizes latency and hops
while providing flexibility
• Interface – Standardized market data interface that supports all market sectors
and business models
• Transport – Efficient means of dissemination that reduces overhead while providing
a lightweight session layer
• Data Layer – Optimized data-transformation layer that radically reduces the size of a
market data payload
– Typically 100 bytes becomes 20 bytes
• Book Management – Intelligent book management practices that utilize thin-content market
data to manage a book
11
Flexible, Scalable, Standards-based Architecture
12
FIX Adapted for Streaming (FAST)
• Implicit tagging– Removes the need for the “tag=“ portion of “tag=value” without imposing the constraints that
fixed-length protocols do.
– Requires specifying message templates
– Retains flexibility and extensibility
– Retains repeating group optimisation from FIX.4.4
• Field encoding– Reduces data volume by leveraging the data affinities between messages
– If I don’t send you a value for Symbol on this Exec Report, use the value from Symbol on the previous Exec Report I sent you
– If I don’t send you a TimeInForce on this Order, use the default value of “0” (day) from the template
• Serialization– Applies binary encoding to the data
– Uses continuation bit encoding
– Eliminates need for explicit separators
– Uses a presence map to indicate which fields are present, and which ones are not
– Special bytes in the header contain the template identifier, so that the other side can deserialize and decode appropriately
– Also considering adding a start-of-frame delimiter or message length to facilitate reading
13
MDOWG Accomplishments (to date)
• Multi-asset class requirements review
• Book Management Recommended Practices
• Multicast Transport Recommended Practices
• FIX Extensions (Draft) for Market Data
• FAST Serialization and encoding specifications
• Initiated a FAST proof of concept (POC)
• Winter 2005 – Published POC Results
• www.fixprotocol.org
14
Proof of Concept Framework
• Data sets provided by POC sponsors (selected for high volume days where possible and type of data feed)– Top of Book – OPRA
– Full Depth – ARCA, NOREX
– Aggregate Depth – CME
• Measurements pre-defined and coded into test harness
• Preliminary testing to determine optimal field encoding
• Strict version control of the code base enforced
• Data sets were redundantly and reciprocally tested by two separate engineering teams – primary and secondary responsibility assignments
– primary results were compared to secondary
– results checked for uniformity; discrepancies researched
– test results correlated and re-run until identical results
• Code base changes resulted in re-testing
15
POC Phase 1A Results Summary
16
FAST Protocol Performance – Native Format
Bandwidth Utilization OPRA Feed vs FAST
0
1000
2000
3000
4000
5000
6000
09:30 10:00 10:30 11:00 11:30 12:00 12:30 13:00 13:30 14:00 14:30 15:00 15:30 16:00
Kb
its
/se
co
nd
OPRA FAST T1 T1x3
Bandwidth Utilization ARCA Feed vs FAST
0
500
1000
1500
2000
2500
3000
3500
08:30 09:00 09:30 10:00 10:30 11:00 11:30 12:00 12:30 13:00 13:30 14:00 14:30 15:00 15:30 16:00
Kb
its
/se
co
nd
ARCA FAST T1 T1x2
Bandwidth Utilization CME Feed vs FAST
All Channels
0
200
400
600
800
1000
1200
1400
1600
8:00 8:30 9:00 9:30 10:00 10:30 11:00 11:30 12:00 12:30 13:00 13:30 14:00 14:30 15:00
Kb
its
/se
co
nd
CME FAST T1
NOREX Bandwidth Utilization
0
10
20
30
40
50
8:30 9:00 9:30 10:00 10:30 11:00 11:30 12:00 12:30 13:00 13:30 14:00 14:30 15:00 15:30 16:00
Kb
its
/se
co
nd
Norex FAST
ARCA OPRA
CME NorEx
17
FAST – ARCA Compression Ratio
18
FAST vs. Native ARCA Bandwidth Utilization
19
FIX Adapted for STreamingSM (FAST ProtocolSM)
• FPL’s Market Data Optimization Working Group developed and demonstrated a series of
techniques to radically reduce message size
• Very timely solution for real world volume and bandwidth challenges
– ↑ Number of trades, ↓ Avg trade size, ↑ Speed
– Quote volumes doubling year after year
– Algorithmic trading engines and DMA driving volume and need for speed
– Stat-arb and other trading strategies predicated on speed
• The FAST Protocol is another example of global market participants working together to solve
real world needs
• A true solution, combine:
– Standardized message definition, business model, and content in FIX
– FAST Protocol applied to reduce message size and serve high-volume needs
20
FAST 1.1 Overview
•One single specification containing all the pieces
•Transfer Encoding
–Support of non US ASCII Strings (byte vector)
–Presence map extension (now zero or one bit per field)
•Field Encoding
–Additional field operators
•Templates
–XML based notation to provide unambiguous template definitions.
•FAST 1.0
–Subset of FAST 1.1
–High degree of compatibility
21
FAST 1.1 Specifics
•Transfer Encoding
–Support of non US ASCII Strings (byte vector)
–Presence map extension (now zero or one bit per field)
•Field Encoding
–Additional field operators
•Templates
–XML based notation to provide unambiguous template definitions.
•FAST 1.0
–Subset of FAST 1.1
–High degree of compatibility
22
Business Case for FAST, SA faces bandwidth challenges
• Bandwidth costs are reduced in proportion to the reduction in data volume. Here’s an
example:
– A feed is projected to use 36 Mbps in the coming year
– Assume a 40 Mbps line is $6000/mo and a 10 Mbps line is $3000/mo
– If FAST is able to deliver 80% compression it now fits on a 9 Mbps line
– Savings is $36K annually times the number of customer connections
• Hardware costs materially reduced
– Lowers CPU utilization
– Reduced data storage requirements
– Server farms and routers don’t need to increase
• Flexible Architecture
– New fields added with little effort
– Ability to plug-and-play any market data format
23
FAST Summary
• FAST POC completed a year ago
• FAST scalability demonstrated to very high volumes
• Committed Implementations
– CME
– OPRA
– Archipelago, NYSE
– ISE
• Proof of Concept
– LSE
– OMX
• Hedge fund and high volume community using it.
Factors
• Journey
– Idea
– Order
• Speed of light
– 299,792,458 m/s
– 299 m/us
– 670,616,629.4 miles per hour
– 1 ft / nanosec
• Rapid Addition
– FIX ~5 microseconds
– FAST ~ 1 microseconds
• Measurement
• IP Stack
– TCP/IP
– Windows 30 microseconds
– Windows HPC 8 microseconds
– UDP
24
• People v’s Computers
– Computers nanoseconds
– People 140 - 200 milliseconds
• Type of business– Solutions
–Smart Order Routing–Proximity Hosting–DMA–Co location
– Care Orders– Active Trading– Human Involvement?– Credit / Limit checking– Booking
–Allocation–Confirmation
• Connectivity– FIX Hub– FIX Direct– IP
–Internet–Private–Switches, Routers and Hubs
25
Resources
• FIX website: fixprotocol.org
• FIX website: fixprotocol.org/fast
• Technical / Specifications - fixprotocol.org/specifications
– Spec docs, FIXML resources, FIXimate, Repository
– Technical Presentations / Documentation
• FIX Implementation Guide - fixprotocol.org/implementation-guide
• FIXGlobal – FPL’s quarterly publication with free subscription -fixglobal.com
• Algorithms – www.fixprotocol.org/working_groups/algowg
• Market Data – www.fixprotocol.org/working_groups/mdoptwg
• Remember
– Visitors see something;
– Registered members see more;
– FIX Contributors see all.
•Get Involved!!