Latin America Electronic Trading Conference 2008 Sao Paolo, Brazilian Mercantile & Futures Exchanges...

Latin America Electronic Trading Conference 2008Sao Paolo, Brazilian Mercantile & Futures Exchanges

The FAST ProtocolSM

Optimising the Communication of Financial Information

FPL Latin America Electronic Trading Conference 2008

2

Background Increased market data volumes

High bandwidth and processing costs Classic FIX

Widespread adoption Inadequate performance for

Market data Quote driven markets

Proprietary Alternatives Complicates integration (many different interfaces) No cost sharing of development and maintenance Limited adoption


3

FAST ProtocolFactors that Influenced the Design

Efficiency Compression Ratio Resource Consumption (CPU, memory) Ease of Use (Conceptual, Implementation, Operation)

Costs Implementation Validation Deployment Operation Maintenance

Proof of Concept•CME•Archipelago•ISE•Microsoft•Singapore Exchange•London Stock Exchange


4

FAST ProtocolCharacteristics

Basic Feature Set Optimized for message streams Content aware (requires knowledge about message structure) Byte-oriented binary representation Variable-length fields Each message contains one or more fields A presence map enables efficient use of default values Several ways of deriving default values

Empirically validated Very fast processing (encoding / decoding) High compression ratios on tested exchange feeds Simple implementation


5

FAST Protocol Layers

Transfer Encoding

Field Encoding

Session Control

Template Management

TCPUDP

IP

Application Messages


6

FAST Vocabulary A data stream consists of a sequence of messages. A message consists of a sequence of one or more fields. A field consists of a sequence of one or more bytes.

Wire format refers to the byte representation that is used to transfer data on the wire.

Encoding is the process of translating to wire format. Decoding is the process of translating from wire format. A CODEC (enCOder/DECoder) provides support for encoding or decoding a

stream of messages.

A byte consists of seven data bits and a stop bit (the ‘SBIT’) that when set indicates the byte is the last byte of a field.

A presence map (‘PMAP’) is a field that is interpreted as a vector of bits, one bit (‘PBIT’) for each field in a message.

A field operator enables previous values to be reused.


7

FAST Transfer Encoding – Fields

b1 b b b b b b

b0 b b b b b b b0 b b b b b b

b0 b b b b b b b1 b b b b b b

b1 b b b b b b

7 bits

14 bits

21 bits

nn bits

FAST Transfer Encoding - SBIT Detail•SBIT coding provides space efficient delimitation of fields

•More compact than byte delimiter for fields shorter than 7 bytes

•Empirical data shows average FAST field sizes of ~2 bytes~33% more compact than a byte delimiter at 2 bytes

•Serialization of SBIT format is simple and efficient

•There are exceptional cases when SBIT coding is suboptimalByte vector can be used for these cases


8

FAST Transfer Encoding – Messages

The tail of a pmap may be compressed

The number of fields in a message depends on the pmap content

pmap field field field pmap fieldfieldpmap

A message may be as short as one byte (an empty pmap)


9

FAST Field Data Types

Unsigned Integer Signed Integer Decimal (aka Composite Scaled Number) ASCII Character String UNICODE Character String Bit Vector Byte Vector Additional FAST 1.0 Data types that were dreprecated

Unsigned Integer without Null Signed Integer without Null Scaled Number

The FAST transport representation supports unlimited size fields.

Implementations will likely limit the supported size to match the native data representation in applications.


10

Some helpful refreshers Some important values to remember

0x80 = 128 decimal = b‘10000000’ 0x80 is an Octet value of 0x00 with the SBIT (Stop Bit) set 0x7f = 127 decimal = b’01111111’

Operations & bitwise AND operation

0x00 & 0x80 => 0x00 0x7f & 0x80 => 0x00

| bitwise OR operation 0x00 | 0x80 => 0x80 0x7f | 0x80 => 0xff


11

For FAST the fundamental data type is the bit

SBIT – Stop Bit is the high order bit in the byte set to b’1’ (8th byte) Bit 7 to indicate the end of a field, b’0’ otherwise

DBITS – Bits 6 through 0 contain data

«datatype»Bit

{documentation = b'0' or b'1'}

+SBIT[1] : StopBit = b'0'+DBITS[7] : Bit

«type»ContinuationByte

+SBIT[1] : Bit = b'1' || b'0'

StopBit

+SBIT[1] : StopBit = b'1'+DBITS[7] : Bit

«type»StopByte

+Bits[8] : Bit

«type»Octet

18


12

Integer Data Types

«datatype»Integer

«type»Enum

{documentation = Enums in FAST are encoded as unsigned integer}

+HighOrderByte[0..*] : ContinuationByte+LowOrderByte[1] : StopByte

«type»UnsignedInteger

{documentation = All positive values adjusted +1}

+HighOrderBytes[0..*] : ContinuationByte+LowOrderByte[1] : StopByte

«type»SignedInteger

{documentation = Twos Complement EncodingSign Bit is the highest order DBIT

All postive values adjust +1}

+LowOrderByte[1] : StopByte = 0x81

ZeroForNullableIntegers

+LowOrderByte[1] : StopByte = 0x80

NULLForNullableIntegers

+LowOrderByteOnly[1] : Octet

«type»SignedIntegerByte

{documentation = Used for ScaledNumber}


13

Unsigned Integer Without Null Support Data Type Transfer from low order to high order 7 bits at a time Set the Stop Bit in the low order byte

Encoded Bytes Stop Value(s) Integer value 0x80 1 0 0 0x04 0xff 0 1 0x04 0x7f 0x027f (639)1 0xff 1 0x7f 0x0000007f (127)1 0x7f 0xff 0 1 0x7f 0x7f 0x00003fff (16383)


14

Unsigned Integer Without Null Transfer Encoding

Value (hex) Seven Data Bits per Byte (bin) Len

127 0000007F 0000000 0000000 0000000 0000000 1111111

0xff

1

16383 00003FFF 0000000 0000000 0000000 1111111 1111111

0x7f 0xff

2

2097151 001FFFFF 0000000 0000000 1111111 1111111 1111111

0x7f 0x7f 0xff

3

268435451 0FFFFFFF 0000000 1111111 1111111 1111111 1111111

0x7f 0x7f 0x7f 0xff

4

4294967295 FFFFFFFF 0001111 1111111 1111111 1111111 1111111

0x0f 0x7f 0x7f 0x7f 0xff

5


15

Unsigned Integer The number is incremented by 1 in the transfer process to account for the

NULL value being supported Why is the NULL value 0x00 encoded as 0x80?

Encoded Bytes Stop Value(s) Integer value 0x80 1 0x00 NULL 0x81 1 0x00 Zero 0xff 1 0x7e 0x0000007e (126) 0x7f 0xff 0 1 0x7f 0x7e 0x00003ffe (16382) 0x10 0x00 0x00 0x00 0x80 0 0 0 0 1 0x10 0x00 0x00 0x00 0x00 0xffffffff (2^32-1)


16

Signed Integer Data Type Two’s Complement Arithmetic When the 7 bit value is converted to a 32 bit value the first data bit is taken

as a sign bit and is copied to the 25 high end bits of the 32 bit value yielding 0xffffffff (which corresponds to -110 in 2’s complement representation).

The most significant data bit of the first byte encoded is the sign


17

Signed Integer Transfer Encoding for Positive Values


0 00000000 0000000 0000000 0000000 0000000 0000000

0x80

1

63 0000003F 0000000 0000000 0000000 0000000 0111111

0xbf

1

8191 00001FFF 0000000 0000000 0000000 0111111 1111111

0x7f 0xff

2

1048575 000FFFFF 0000000 0000000 0111111 1111111 1111111

0x3f 0x7f 0xff

3

134217727 03FFFFFF 0000000 0111111 1111111 1111111 1111111

0x3f 0x7f 0x7f 0xff

4

2147483647 7FFFFFFF 0000111 1111111 1111111 1111111 1111111

0x07 0x7f 0x7f 0x7f 0xff

5


18

Signed Integer Transfer Encoding for negative values


-1 FFFFFFFF 1111111 1111111 1111111 1111111 1111111

0xff

1

-64 FFFFFFC0 1111111 1111111 1111111 1111111 1000000

0xc0

1

-8192 FFFFE000 1111111 1111111 1111111 1000000 0000000

0x80

2

-1048576 FFF00000 1111111 1111111 1000000 0000000 0000000

0x40 0x00 0x80

3

-134217727 F8000000 1111111 1000000 0000000 0000000 0000000

0x40 0x00 0x00 0x80

4

-2147483648 80000000 1111000 0000000 0000000 0000000 0000000

0x78 0x00 0x00 0x00 0x80

5


19

Signed Integer with Null Support Still Two’s Complement Arithmetic – does not change for negative numbers Plus, just like with the Unsigned Integer with Null Support Data Type, we get

to increment positive values by 1 PLUS ONE ENCODING

Good thing we have computers to do this stuff for us

Encoded Bytes Stop Value(s) Integer value 0x80 1 0x00 NULL 0x81 1 0x01 0 0x04 0xff 0 1 0x04 0x7e 0x027e (638)1 0x08 0x00 0x00 0x00 0x80 0 0 0 0 1 0x08 0x00 0x00 0x00 0x00 0x7ffffffff (2^31-1) 0x78 0x00 0x00 0x00 0x80 0 0 0 0 1 0x78 0x00 0x00 0x00 0x00 0x800000000 (-2^31)


20

FAST Scaled Number (decimal) Data Type Base-10 Single field Requires one field in the PMAP First byte is interpreted as a

Signed Integer specifying the Exponent

Remaining bytes are treated as a Signed Integer representing the Mantissa

Provides exact decimal representation

Mantissa X 10Exponent

Mantissa Exponent Value

-1 0 -1*10^0 = -1

127 -2 127*10^-2 = 1.27

1 4 1*10^4 = 10000

+ExponentByte[1] : SignedIntegerByte+MantissaBytes[0..1] : SignedInteger

«type»ScaledNumber

{documentation = Value is treated as a single field}

+ExponentByte[1] : SignedIntegerByte = 0x80

NullScaledNumber X Obsolete


21

FAST Decimal Datatypeaka Composite Scaled Number Data Type

Mantissa Exponent Value

-1 0 -1*10^0 = -1

127 -2 127*10^-2 = 1.27

1 4 1*10^4 = 10000

Base-10 Consists of two fields Requires two places in a PMAP if

field encoded Exponent maximum is -63 to +63

Mantissa X 10Exponent

+ExponentField[1] : SignedInteger+MantissaField[1] : SignedInteger

«type»CompositeScaledNumber

{documentation = Composed of two separate fields}

+ExponentField[1] : SignedInteger = 0x80+MantissaField[1] : SignedInteger = 0x80

NullCompositeScaledNumber{documentation = Composed of two separate fields}


22

FAST ASCII String Data Type

7 Bit ASCII Character Set Supported Last Byte in string has the Stop Bit set

0x54 0x68 0x61 0x6e 0x6b 0x20 0x59 0x6f 0x75 0x20 0x42 0x4d 0x46 0xa1

What is the decoded string in ASCII?

+StopChar[1] : StopByte = 0x80

NullString

+ContinuedChar[1] : ContinuationByte = 0x00+StopChar[1] : StopByte = 0x80

EmptyString

+ContinuedChar[0..*] : ContinuationByte+StopChar[1] : StopByte

«type»ASCIICharString


23

What is in the String?

Thank You BMF!!

0x54 0x68 0x61 0x6e 0x6b 0x20 0x59 0x6f 0x75 0x20 0x42 0x4d 0x46 0xa1


24

UnicodeString DatatypeByte Vector Data Type

Length specified as a FAST Unsigned Integer Value

Length number of Octets follow Full 8 bit values 0x00 – 0xff A NULL Byte Vector is

represented by setting the length to Null

Can be used for non-ASCII character sets

+length[1] : UnsignedIntegerWNull+vector[0..*] : Octet

«type»ByteVector

{documentation = Can be used for non-ASCII Character strings}

+length[1] : UnsignedIntegerWNull = 0x80

NullByteVector


25

Bit Vector Data Type

+ContinuedBits[0..*] : ContinuationByte+StopBits[1] : StopByte

«type»BitVector

Series of zero or more continuation bytes followed by one stop byte

There is no NULL value defined for this type

Bytes Stop Bit Vector 0xff 1 b’1111111’ 0x06 0x83 0 1 b’0000110’ b’0000011’ 0x60 0x07 0xf0 0 0 1 b’1100000’ b’0000111’ b’1110000’


26

A Transfer Encoding ExampleBeginStr SeqNum SenderID SendingTime Price Symbol

8=FIX.4.4|34=10000|49=CLIENT1|52=20060126-13:06:58.100|44=1200|55=FOO1|8=FIX.4.4|34=10001|49=CLIENT1|52=20060126-13:06:58.200|44=1210|55=FOO1|8=FIX.4.4|34=10002|49=CLIENT1|52=20060126-13:06:58.300|44=1190|55=BAR2|

Original size 71 bytes

FIX.4.4|10000|CLIENT1|20060126-13:06:58.100|1200|FOO1|FIX.4.4|10001|CLIENT1|20060126-13:06:58.200|1210|FOO1|FIX.4.4|10002|CLIENT1|20060126-13:06:58.300|1190|BAR2|

FIX.4.4nnCLIENT120060126-13:06:58.100nnFOO1FIX.4.4nnCLIENT120060126-13:06:58.200nnFOO1FIX.4.4nnCLIENT120060126-13:06:58.300nnBAR2

Implicit Tagging54 bytes (-24%)

SBIT Encoding48 bytes (-33%)

SBIT+Binary Encoding43 bytes (-39%)

FIX.4.410000CLIENT120060126-13:06:58.1001200FOO1FIX.4.410001CLIENT120060126-13:06:58.2001210FOO1FIX.4.410002CLIENT120060126-13:06:58.3001190BAR2

Greg


27

Beyond Transfer Encoding

Further compaction requires information about data affinity and predictability.

BeginStr SeqNum SenderID SendingTime Price Symbol8=FIX.4.4|34=10000|49=CLIENT1|52=20060126-13:06:58.100|44=1200|55=FOO18=FIX.4.4|34=10001|49=CLIENT1|52=20060126-13:06:58.200|44=1210|55=FOO18=FIX.4.4|34=10002|49=CLIENT1|52=20060126-13:06:58.300|44=1190|55=BAR2

Affinity

Predictability

Field Encoding ...


28

Templates

A template specifies how to encode an application message into the transfer encoding

A template conveys Message Structure Data Types Field Operators


29

Template Overview

<template name="ExampleOrder"> <messageRef name="NewOrderSingle"/> <string name="BeginStr"> <constant value="FIX.4.4"/> </string> <u32 name="SeqNum"> <increment/> </u32> <string name="SenderID"> <copy/> </string> <string name="SendingTime"> <delta/> </string> <decimal name="Price"> <delta/> </decimal> <string name="Symbol"> <copy/> </string></template>

Data Types

Field Operators

Structure


30

Characteristics of FAST Templates Semantics are formally and unambiguously defined

Enables consistent behavior and interoperability among implementations Concrete Syntax

Default format for authoring, storing and interchanging templates Human and machine readable XML Syntax

Supports evolution of the template specification Extensible – enables inclusion of custom data in templates Widespread knowledge and good tool support

Alternative concrete representations Hard coded in encoder and/or decoder FAST encoded as specified by the Session Control Protocol (SCP)


31

The Transfer Encoding Example RevisitedBeginStr SeqNum SenderID SendingTime Price Symbol 8=FIX.4.4|34=10000|49=CLIENT1|52=20060126-13:06:58.100|44=1200|55=FOO1|8=FIX.4.4|34=10001|49=CLIENT1|52=20060126-13:06:58.200|44=1210|55=FOO1|8=FIX.4.4|34=10002|49=CLIENT1|52=20060126-13:06:58.300|44=1190|55=BAR2|

Original size 71 bytes

FIX.4.4|10000|CLIENT1|20060126-13:06:58.100|1200|FOO1|FIX.4.4|10001|CLIENT1|20060126-13:06:58.200|1210|FOO1|FIX.4.4|10002|CLIENT1|20060126-13:06:58.300|1190|BAR2|

FIX.4.4nnCLIENT120060126-13:06:58.100nnFOO1FIX.4.4nnCLIENT120060126-13:06:58.200nnFOO1FIX.4.4nnCLIENT120060126-13:06:58.300nnBAR2

Implicit Tagging54 bytes (-24%)

SBIT Encoding48 bytes (-33%)

Binary Encoding43 bytes (-39%)

FIX.4.410000CLIENT120060126-13:06:58.1001200FOO1FIX.4.410001CLIENT120060126-13:06:58.2001210FOO1FIX.4.410002CLIENT120060126-13:06:58.3001190BAR2


32

Data Types

Strings, Integer and Decimal numbers, Boolean, Byte Vector Restricted types for numbers – 8-bit up to 64-bit sizes

<template name="ExampleOrder"> <messageRef name="NewOrderSingle"/> <string name="BeginStr"/> <u32 name="SeqNum"/> <string name="SenderID"/> <string name="SendingTime"/> <decimal name="Price"/> <string name="Symbol"/></template>

BeginStr SeqNum SenderID SendingTime Price Symbol Sz8=FIX.4.4|34=10000|49=CLIENT1|52=20060126-13:06:58.100|44=1200|55=FOO1 448=FIX.4.4|34=10001|49=CLIENT1|52=20060126-13:06:58.200|44=1210|55=FOO1 448=FIX.4.4|34=10002|49=CLIENT1|52=20060126-13:06:58.300|44=1190|55=BAR2 44

Original Size: 71 bytesOriginal Size: 71 bytes


33

Field Operators

Constant – Always the same value Increment – Frequently previous value incremented by one Copy – Frequently the same as the previous value Delta – Values differ slightly Default – Frequently a specific value


34

Constant Operator

<template name="ExampleOrder"> <messageRef name="NewOrderSingle"/> <string name="BeginStr"> <constant value="FIX.4.4"/> </string> <u32 name="SeqNum"/> <string name="SenderID"/> <string name="SendingTime"/> <decimal name="Price"/> <string name="Symbol"/></template>




35

Increment Operator

<template name="ExampleOrder"> <messageRef name="NewOrderSingle"/> <string name="BeginStr"> <constant value="FIX.4.4"/> </string> <u32 name="SeqNum"> <increment/> </u32> <string name="SenderID"/> <string name="SendingTime"/> <decimal name="Price"/> <string name="Symbol"/></template>




36

Copy Operator


<template name="ExampleOrder"> <messageRef name="NewOrderSingle"/> <string name="BeginStr"> <constant value="FIX.4.4"/> </string> <u32 name="SeqNum"> <increment/> </u32> <string name="SenderID"> <copy/> </string> <string name="SendingTime"/> <decimal name="Price"/> <string name="Symbol"> <copy/> </string></template>



37

Delta Operator


<template name="ExampleOrder"> <messageRef name="NewOrderSingle"/> <string name="BeginStr"> <constant value="FIX.4.4"/> </string> <u32 name="SeqNum"> <increment/> </u32> <string name="SenderID"> <copy/> </string> <string name="SendingTime"> <delta/> </string> <decimal name="Price"> <delta/> </decimal> <string name="Symbol"> <copy/> </string></template>



38

Naming Names identify messages, templates and fields Names of messages are typically global

Maintained by the owner of a protocol, for example FPL Names of templates are typically local

Maintained locally by companies using or implementing FAST Standard templates can be maintained by for example FPL

Name collision must be avoided Templates use the same method as in Java and XML

Names belong to a namespace based on the company domain name


39

Naming Example

<templates templateNs="http://www.example.org/ns/example" ns="http://www.fixprotocol.org/ns/fix44">

<template name="ExampleOrder"> <messageRef name="NewOrderSingle"/> … </template>

<template name="ExampleTrade"> … </template> …</templates>


40

Additional information Session Control Protocol Version 1.1 More and more vended solutions

Majority of FIX engine vendors: ORC, TransactTools, NYFIX, Rapid Addition, and others

Open source solutions http://www.openfast.org (Java only) At least two more efforts underway


41

Questions?

Latin America Electronic Trading Conference 2008 Sao Paolo, Brazilian Mercantile & Futures Exchanges...

Documents

Transcript of Latin America Electronic Trading Conference 2008 Sao Paolo, Brazilian Mercantile & Futures Exchanges...