Extent 2013 Obninsk Trading Systems: Testing at the Confluence of FT & NFT
-
Upload
extentconf-tsoy -
Category
Technology
-
view
966 -
download
1
description
Transcript of Extent 2013 Obninsk Trading Systems: Testing at the Confluence of FT & NFT
Trading Systems:Testing at the Confluence of FT & NFT
Alexey Zverev, Managing DirectorAlyona Bulda, QA Project LeadIvan Bobrov, HFT Analyst
Introduction
1. Introduction2. Our topic - how to find / reproduce "toxic issues" which can happen under
the load or in real live environment conditions3. We will talk about:
a. A classical approach where FT and NFT are separate streamsb. What is missed herec. Some approaches we use or may apply to fill this gap between FT and
NFT (applying to trading systems)4. Our presenters
FTfocus on:
• Test scenarios are based on functional requirements
• Test tools which allow to run simplistic tests such as to send an order -> to verify a response
• Prepare a functional library starting from simple scenarios and adding more complex ones step by step
• Test scenarios are automated
• A test Library is enhanced as we proceed with its testing and production clients scenarios
• An automated library will be reused as a regression test library which is executed against each release after go live and can be taken for consequent projects
NFTfocus on:
• Testing is based on non-functional requirements
• Load scenarios are driven by the technical characteristics in PROD.
• The main target of the tests is to assess non-functional requirements: latency, capacity and fault tolerance
• A lot of efforts are invested in correct latency measurements and determining conditions when the system breaks
• Further regressions are done to compare system performance indicators with the previous releases
What do we miss when we keep FT and NFT separated ?
- Any complex system must sustain a certain level of
concurrency. Some issues appear when several events happen
simultaneously (Race conditions).
Example 1: Clients send a CxlReplace request and approx. at the same
time
the ME sends a fill back.
These 2 events meet at
the Order Management
server, and the state of
the order can get corrupted.
What do we miss when we keep FT and NFT separated ?
- Production order books and various system object states are
more complex than in our FT scripts, and no amount of test cases
can cover all possible scenarios
Example 2: A User tries to
hit two different implied
orders coming from the
same parent order
which is being restated
in the middle of the process.
What do we miss when we keep FT and NFT separated ?
- Components may behave differently under the load
Example 3: The System must support various
throttling limitations and the requirements of
the system behaviour. The correctness
of functional system behavior can be
tested only under the load.
- Integer memory overflow
Example 4: Turnover (VWAP) fields in the statistics messages and
incremental OrderID overflow;
* the picture was taken from https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcS9nlDdLAmuvmyRqqZsanbyfwL601nJlOvtHGEfJBS4DKnmSTANUA
What do we miss when we keep FT and NFT separated ?
- Dynamic Mass events which happen during
DLC
Example 5: Market Open,
Market Close, Auctions,..
The state of the system
at exact instance
when the system goes
from one session to another
should be verified with the load
applied against the system.
Detailed Verification Of The System Behavior Under Random Load
We need to make sure that we provide extensive coverage We need to make sure that the load test mimics what happens in the production system
We need to verify that the system works as expected
- basic consistency check - market data must be internally consistent- order book should be available and not be different
+ Recreation of exact steps of real production issues
- It is difficult to record multiple network streams
Performing Functional Testing Against The System Under The Load
to leave a part of the system intact by the load
to adjust functional tests to cope with the activity of the load
to make scripts in case when we perform tests actively interacting with the load * scenarios must be flexible* sustain interaction with the script
Testing With The Baseline
The example of the cycle of a Clearing System
A detailed, step by step analysis
The “Mini Robots” Solution
The “Mini Robots” Solution
Questions & Answers-tO
Thank You-tO