Finance Trading in The Cloud - AWS Michigan Meetup
-
Upload
eric-detterman -
Category
Technology
-
view
1.927 -
download
4
description
Transcript of Finance Trading in The Cloud - AWS Michigan Meetup
1
Legal Informatics w/ AWS CloudSearch & High-Performance Financial Market Apps
AWS Michigan Meetup
October 9th, 2012
http://www.solidlogic.com
Objectives
‣ Introduce Quantitative Trading
‣ Present a case study on AWS usage in Quantitative Trading System Evaluation.
‣ Discuss potential improvements upon our presented architecture.
2 http://www.solidlogic.com
4
Solid Logic Technology develops innovative custom technology solutions across a variety of industries using leading software, infrastructure and business practices.
http://www.solidlogic.com
About us
5
Industry experience ‣ Financial and legal services ‣ Logistics ‣ Automotive ‣ Defense and homeland security ‣ Consumer sales and service ‣ Academic and scientific research
Our expertise Infrastructure and cloud computing ‣ Scalable, programmatic infrastructure
management
‣ Strategic data center design
‣ VMware architecture and management
‣ Multi-cloud development and deployment
‣ Scalable web infrastructure with CDN
‣ Security and compliance methods and implementation
Software development ‣ Analytical solutions - simulation, optimization, big
data, natural language processing, quant. finance
‣ Enterprise content management, workflow solutions, system integration
‣ Oracle Transportation Management
‣ Database technology (Oracle, Vertica, Postgres, Cassandra, etc.)
‣ Web application and website development
Company Information ‣ Founded in 2011 ‣ Entirely mobile company ‣ Develop both internal projects (IP) and
client software solutions
http://www.solidlogic.com
Solid Logic Management Team ‣ Eric Detterman, CEO and Co-Founder
• Professional Experience - Legal IT Business Analyst, Lean Startup, Cloud Computing, Processing Engineering and Consulting
- Researched and developed core investment strategies for Birmingham, MI RIA
- Currently in production and managing > $20M, AUM growth > 50% annually
- Proprietary trading (equities, futures, options), web and software development
• Education: B.S. Economics – Oakland University
‣ Michael Bommarito, CIO and Partner • Relevant Experience
- “Big data” consultant, Oracle ERP architect, Linux cluster administrator. - Software developer - NYC-based quantitative hedge fund
- Consultant - multiple quantitative hedge funds
• Education : M.S.E Financial Engineering, M.S. Political Science, B.S. Mathematics – University of Michigan
‣ Ronald Redmer, Board Member and Lead Technical Advisor • Relevant Experience
- CIO, National Default Exchange (NDeX), a business unit of The Dolan Company (NYSE:DM)
- CEO defense supplier company, Airport systems software, CEO auto testing company, Affina – software dev mgr, EDS - tech lead
6
http://www.solidlogic.com
Case Study: Proprietary Trading Simulation
Quantitative Trading and Investment Systems: ‣ (Loose) Definition:
• Rules-based mathematical ‘model’ created by testing and validating a hypothesis about how a tradable market acts or optimizing parameters to create an equation to describe the market.
• The goal is to outperform the broad market (S&P 500) or some benchmark after costs.
‣ Example Strategy: • Investment universe = ~50 Fidelity Mutual Funds • Strategy #1: Invest in the top six ranked mutual funds based on proprietary
momentum (p0 > p-1) based ranking algorithm. Analyze and rank fund universe every 45 days and re-allocate.
• Strategy #2: Invest in the top six ranked mutual funds based on proprietary mean reversion (p0 > p-1) based ranking algorithm. Analyze and rank fund universe every 45 days and re-allocate.
8 http://www.solidlogic.com
Case Study: Proprietary Trading Simulation
9
Hypothetical Example
http://www.solidlogic.com
Case Study: Proprietary Trading Simulation
10
Challenge: ‣ Characterize the performance and
sensitivity of an equity trading system across input parameters and market conditions
‣ Optimize parameters based on profit and risk measures
‣ Estimated runtime is unacceptable on local workstation (>1 month)
‣ Primary bottlenecks are in dense linear algebra operations
• Spectral decomposition (ARPACK) • Pairwise comparison of higher-order
distribution moments (M-M arithmetic)
Scope: ‣ Assets 62 ‣ Tests/asset 96 ‣ Total tests 5,952
Test Information ‣ Mean components/asset 395 ‣ Points/component 3,135 ‣ Points/test 1,238,325
‣ Total elements 7,370,510,400
http://www.solidlogic.com
Case Study: Proprietary Trading Simulation
Potential solutions: ‣ Run on existing hardware – wait for results ‣ Physical or virtualized servers with supporting job schedulers –
requires hardware, software, and specialized labor ‣ Setup cloud infrastructure to process work – requires software
and specialized labor
11 http://www.solidlogic.com
Trading Simulation: Architecture
12
This was our initial version – Not overly elegant, but works very well with minimal effort to setup. Easy to improve upon.
US East Region
Availability Zone
Strategy Test Results
(S3 Buckets) Trading System
Source Code and Config Data (Git Repo)
Custom Created
AMIs (x16)
http://www.solidlogic.com
Trading Simulation: Test Process
13
US East Region
Availability Zone
Strategy Test Results (S3
Buckets)
Local Development Environment
Custom Created
AMIs (x16)
Trading System
Source Code (Git Repo)
http://www.solidlogic.com
Trading Simulation: Overview
14
Technology Solution: ‣ Built an optimized simulation
environment as virtual image (AWS EC2 AMI)
‣ Provisioned and configured centralized storage (AWS S3)
• Experiment configuration • Simulation input • Simulation output • Post-processed results
‣ Fully automated deployment of simulation to instances through master source control system (git)
Compute Instance (x16): ‣ 88 Elastic Compute Units (ECU) ‣ 2x Xeon E5-2670s-16 cores ‣ 60.5GB RAM ‣ 10GbE, dual NIC ‣ 3+TB instance scratch
Total Compute Resources: ‣ 1408 ECUs ‣ 512 concurrent threads (HT) ‣ 968GB RAM
(1 ECU~=5GFlops)
http://www.solidlogic.com
Trading Simulation: AMI Creation Process
‣ Use standard Ubuntu Server 12.04.1 LTS for Cluster Instances AMI x64 (ami-eb7bcf82)
• cc2.8xLarge – 88ECUs, 16 cores, 60.5GB RAM
‣ Install git, s3cmd, PostgreSQL JDBC drivers
‣ Install and configure test environment and all dependencies
‣ Create new AMI based on the above
15 http://www.solidlogic.com
Trading Simulation: Test Execution
‣ For each test instance….. • ssh -X -i /home/ericd/.aws/first/name.pem ubuntu@IP • cd /home/ubuntu/testcode/tradingsystemsales • git pull • cd /usr/local/testcode//bin • sudo ./testcode -nodesktop • parameterSweepSingleNode('Yes','Yes',
'\home\ubuntu\testcode\tradingsystemsales\models\daily\AdaptiveStateSpaceSPY\data\masterlist.mat', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/data/etfs', 'csv', '', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/data/etfs', 1, '\home\ubuntu\testcode\tradingsystemsales\models\daily\AdaptiveStateSpaceSPY\data\ETFsToTest.csv', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/data/etfs', 'mat', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/src', 'No', 'No')
• parameterSweepSingleNode('No','Yes',
'\home\ubuntu\testcode\tradingsystemsales\models\daily\AdaptiveStateSpaceSPY\data\masterlist.mat', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/data/etfs', 'csv', '', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/data/etfs', 1, '\home\ubuntu\testcode\tradingsystemsales\models\daily\AdaptiveStateSpaceSPY\data\ETFsToTest.csv', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/data/etfs', 'mat', '/home/ubuntu/testcode/tradingsystemsales/models/daily/AdaptiveStateSpaceSPY/src', 'No', 'No')
16 http://www.solidlogic.com
Trading Simulation: Initial Test Results
‣ Result sets saved to S3 buckets using S3cmd
• Approximately 6000 result sets
17 http://www.solidlogic.com
Trading Simulation: Output
‣ Run Time:
• Cloud: 45 hours
• Single-seat: 1-2 months
• Order of magnitude improvement in time!
18 http://www.solidlogic.com
Trading Simulation: Economics
19
On-Site Co-Location
On-Demand
Spot Pricing
Server Hardware / Instance Usage $69,045 $69,045 $1,647 $193
Network Hardware 13,809 13,809 - -
Hardware Maint. 24,856 24,856 - -
Operating System - - - -
Power and Cooling 9,907 - - -
Data Center Construction / Co-Location Expense
8,618 65,136 - -
Admin. / Remote Hands Support 105,000 240 - -
Data Transfer 1 4 1 1
Total $231,237 $173,091 $1,647 $193
Cost Savings1 N/A 25.12% 99.29% 99.92%
$ / Compute Hr.2,3 $26.40 $19.76 $2.40 $0.28
Cost Model Details ‣ Cost estimates using assumptions
and calculations in Cost Comparison Worksheet
‣ Costs represent one year annualized costs. Assumes a useful life of three years for purchased equipment
‣ 1 = Cost savings using On-Site as baseline
‣ 2 = On-Site and Co-Location assume 100% usage
‣ 3 = Based on actual 686 machine hours used
http://www.solidlogic.com
Trading Simulation: Next Steps
Potential Improvements: ‣ Develop improved cloud infrastructure management tools
• Allocation of work across instances • Allow user defined completion time and programmatically
scale compute resources to work towards goal • Spread work across unused internal and available external
compute resources
20 http://www.solidlogic.com
Thank you
21
Eric Detterman CEO, Co-Founder
Michael Bommarito CIO, Partner
Direct: (248) 792 – 8001
Direct: (646) 450 – 3387
(248) 792 – 8000 www.solidlogic.com
330 East Maple Rd. #231 Birmingham, MI 48009