Post on 22-Jan-2021
Detection of Abnormal Changes in Financial Markets
Hideki TakayasuADAM, Tokyo Institute of Technology
(Advance Data Analysis and Modeling Unit)and
Sony Computer Science Laboratories., Japan
Foreign Exchange Market as a complex systemUniversal mechanisms of random walks
Detection of abnormal changes
What are financial markets ?“The coming together of buyers (borrowers) and sellers (lenders) to trade financial securities”
cf. Wikipedia
There are more than 1 million markets as a whole in the world.
FX market data is very interesting because:
●Total volume of transaction is the largest in the world
400x1010 USD (4 Trillion) /day
(The whole world trade (materials) is 4x1010 USD /day)
Foreign exchange market: currency marketEx. Buy 1 million US‐dollar by JP‐yen by a rate 123.456 JPY/USD
●The market is open 24 hours from Monday to Friday
(Continuity of data is better than any other markets)
●Time resolution is very high, 1 msec
●All orders (buy, sell, cancel) are visible
(except user’s names)
Mostly for speculation
From the viewpoint of complex systems
Financial markets are consisted of millions of different components (individuals, companies)
Millions of markets are interacting
Any news can affect the markets
Sometimes rules change
Human desire and emotion also affects(the ultimate intangible complex systems)
Changing every time, no steady state ever existed !
Scientific approaches seem very difficult, but they are extremely important for our society
News
Good properties for scientific study
The structure of one market is rather simple:Assembly of buy and sell orders on 1 dimension (price)(1‐dim data with time stamps)
The aim of each dealer can be simple, buy at a low price and sell at a high price to earn(Agent‐based modeling is available)
We can get high quality data for scientific research that enable us detail analysis (Empirical laws are derived from the data)(Number of direct dealers are limited, about 1000)
Fractal property = Random walk
Random walk property of market pricesDollar-Yen rate for 13 years
zig-zag Tick level
T.Mizuno, S.Kurihara, M.Takayasu, and H. Takayasu[Physica A324(2003), 296‐302]
Mandelbrot
For shorter time scale the fractal property breaks down
Time scale of financial markets
10-3 1 102 105 107 1010
sec min hours day month year
dealers' strategy
fractal properties
Financial technology
"10-minutes is an epoch"
millisec
Algorithm trader’s time scale
Langevin model and PUCK model
Why Random Walk ?Buy and Sell orders occur randomly
This is roughly correct, but the mechanism is not that simpleDeeper understanding necessary
A naïve question :Does the randomness comes from out‐side or is it an intrinsic property of the market ?Exogenous? Endogenous?
Exogenous effects exist, of course, also an endogenous mechanism of chaos exists due to the market rule
9
Every dealer hopes to increase the money by suitable exchange
buy
sell
gain
1: Try to buy at a low price
and to sell at a high price
pricebid ask2: Follow the trend
x spread (greediness)
expectationTrend?
Common strategies …
Yamada et al, Phys. Rev. E 79 (2009), 051120
The dealer model H.Takayasu, H.Miura, T.Hirabayashi, and K.Hamada [Physica A, 184 (1992),127‐134](The brain of Abe‐Cabinet) An oldest agent‐based model
of artificial market
10
Dealer model (artificial markets, agent models)Each dealer has both buying and selling pricesA trade occurs when buying and selling prices meet
price
Buy Sell
bids asks
x(t) market price
Min{Ask(j)}≦Max{Bid(i)}Trades have attractive interactions for the dealers in the price space
PlatinumCO
O
OCO
Similar to catalytic reaction
O O OCO
CO
Or matter‐antimatter reaction
Bid(i) < Ask(i) for all i
11
Trend‐following action?
?
time
price
Appearance of trend ・・・
depends on dealers
A trend continues or turns ・・・depends on dealers strategy
We can assume a term proportional to latest price change( )
nd x t
Coefficient: positive means trend‐follow
Negative means anti‐trend action (contrarian)
12
Compromise:A seller (or buyer) gradually decreases (or increases) the price
In the simplest deterministic dealer model, we assume that after the deal, the seller becomes a buyer, the buyer becomes a seller
price
Ask (seller)
Bid (buyer)
time
Trade occurs
A hasty dealer has a steeper slope
A slow dealer has a gentle slope
(these values are given initially)
nnsa s=1:buyers=‐1:seller
13
( ) ( ) ( ) ( )n n n n nx t t x t a s t d x t
Evolution of dealers (deterministic, no randomness added)
n‐th dealer’s buying price
(selling price is given by adding the spread L )
A market simulation with 3 deterministic dealers
The mechanism of chaos makes seemingly random fluctuations
Trend‐followBuyer or seller
What is the chaos in this model ?Chaos is the mechanism of enhancing small difference by nonlinear effect
price
Dealer 0Dealer 1Dealer 2
Dealer 1 deals Dealer 2 deals
Deals are highly nonlinear (step‐function type) irreversible interaction
Dealer 1 and 2 are very different
All dealers are quite similar
“Randomness” is produced by this market mechanism intrinsically
Without the trend‐follow (d=0), the market price fluctuates around an equilibrium price.
Trend‐follow (d≠0) is essential for realization of random walk
( )x t
( )x tTrend‐follow (d≠0) is also essential for volatility clustering
Volatility clustering occurs automatically by this model
t
( 1) ( )x x t x tD º + -
Real Market Simulated MarketxD
( ) ( )X
P X p x d x
X a
¥
D-
³D = D D
µD
ò
Price difference distribution follows a power law just like the real market
K.Yamada, H.Takayasu, T.Ito, M.Takayasu, Phys. Rev. E79, 051120 (2009)
( ) ( )x x t dt x tD º + -
Our latest data analysis clarified that there exist real dealers who behave quite similar to this model, pair‐wise ordering (buy‐sell) and trend‐follow behaviors (preparing a paper)
17
Theoretically we can show that the motion of mass center is approximated by Langevin eq. in the case d>0
2
2
1( ) ( ) ( ) , ( ) ( )
nn
d dx t x t F t x t x t
dt Ndt
T.Hirabayashi, H.Takayasu, H.Miura, K.Hamada, Fractals, 1(1993), 29‐
Langevin eq. is the key concept for Brownian motion
Therefore, the dealer model produces price motion similar to Brownian motion
In 1908 Langevin introduced the equation as a model of
Brownian motion of a fine particle suspended in water.
Open Access
Recently, market price dynamics in very short time scale is shown to follow Langevin equation with non‐white noise by analyzing high quality market data, "order‐book data".
( ) ( ) ( ) ( )dv t t v t G t
dtm=- +
In the case of a colloidal particle motion in water molecules, we expect the following relationsAssume that the colloidal particle is moving to the rightThe density of molecules will changes around the particle
Financial Brownian particleCore size = bid‐ask spreadCenter at the mid price=
(best‐bid + best‐ask)/2
In material:Water molecules are invisible, however, In market: Financial molecules are visible.
- + - +
Inner layer
Density changes
outer layer outer layer
Inner layer diameter = core size + 2 γc
Density change profile when price goes up (right)
Push by moleculedensity increase
Pull by moleculedensity decrease
ɤc =18 pips (0.001 yen)No correlation for ɤ>100
No correlation for ɤ>100
:[ , ] :inner
( ) { ( ) ( ) ( ( ) ( ))}i x x x x
s t t t x
F t c s a s c s a s- - + +
+D
= - - -å åNumber of particles increased in the inner layer bid side
Number of particles increased in the inner layer ask side
Summing up all changes in the inner layer for a time interval
ɤ : the depth from the best bid (ask)Density change
( )/slope L t t= D D
Velocity of the colloidal particle is approximated by the following quantity
( )( ) ( ) ( )
i
L tv t F t t
th
D= +
D
( )L tD : Price change per order change→ mean free path
( )th : Noise term
: Time window 100 deals ≒160 sec
tD
( ) :iF t
The total particle number change in the inner layer (anti‐particles are counted with negative sign) is highly correlated with market price velocity.
Layered order flow
The correlation is much higher than the conventional order flow
( ) ( )i
v t F t< >Correlation becomes higher by decomposition of the inner layer
Layered order flow vs. conventional order flow( )iF t
( ) { ( , ) ( , ) ( , ) ( , )}x
F t c x t c x t a x t a x t¥
- + - +
=-¥
= - - +å( )F t
(sum of increment of all buy orders – sum of increment of all sell orders)
( ) ( )v t F t< >Correlation is lower because opposite contributions are added
The inner layer orders work as driving force accelerating price motion,and the outer layer work oppositely braking the movement
Conventionally, it has been believed that
Price change ∝ demand - supply(all buy orders) (all sell orders)
The fact is Price change ∝ change of (buy-sell orders)
only in the inner layer
Outer layer molecules work in an opposite way
driving force
drag force (resistive force)
Space‐time composition of particlesParticle created in the inner layer
Annihilation in the inner, outer
Number of particles coming from outer and annihilated in the inner (origin of drag force)
Particle created in the outer layer
Creation in the inner and outer
Particle annihilated in the inner layer
Particle annihilated in the outer layer
Number of particles created in the inner and annihilated in the inner, never being in the outer (driving force)
( )( ) ( ) ( )
i
L tv t F t t
th
D= +
D
( ) ( ) ( ) ( ) ( )i i i i if t c t c t a t a t- + - += - - +
( ) ( ) ( ) ( )oia others
v t v t v t th= + +
0
( )( ) { ( ) ( )}
oi
t
a oi ois
L tv t a t s a t s
tD - +=
D= - + + +
D å
0
( )( ) { ( ) ( ) ( ) ( )}
t
others i i ii iis
L tv t c t s c t s a t s a t s
tD - + - +=
D= + - + - + + +
D å
( ) ( ) ( )oi
t
av t t s v s dsf
-¥
= -ò
Molecules coming from outer to inner
Linear Response relation between and
Decomposition of velocity
2 2 2| ( ) | | ( ) | | ( ) |oiav vw f w w=
2( ) ( )oi oi
i ta av v t e dtp ww
¥-
-¥
= òFourier Transform
( )oiav t ( )v t
Response function
Velocity can be decomposed into originated and the others.( )oia t
Estimation of the response function
2
22
| ( ) || ( ) |
| ( ) |oiav
v
wf w
w< >=< >
2| ( ) |oiav w< >
2
0.00725
0.0764 w+
Average is taken over 200 realizations for window size 28.
0( ) tt e df f -=
00.08, 0.27f d= =
From the data we calculate the power spectrum of and
2| ( ) |v w< >
( )v t ( )oiav t
By taking their ratio, the spectrum of response function is estimated
The functional form of
is approximated by a Lorentzian. This means that the response function is an exponential function:
2
2
| ( ) |
| ( ) |oiav
v
w
w< >
2
22
| ( ) || ( ) |
| ( ) |oiav
v
wf w
w< >=< >
( )0
( ) ( )oi
tt s
av t e v s dsdf - -
-¥
= ò
Derivation of Langevin Equation and resistivity
( ) ( ) ( ) ( )oia others
v t v t v t th= + +
( ) ( ) ( ) ( )oia others
d d d dv t v t v t t
dt dt dt dth= + +
0
0
( ) ( ) ( )
{ ( ) ( )} ( )oi oia a
F
dv t v t v t
dtv t v t v t
d f
d h f
=- +
=- - - +
0( ) ( ) ( ){ ( ) ( )}
( ) ( )others
dv t v t t
dtv t G t
d f d h
m
=- - + + +
=- +
00.27 0.08 0.19m d f= - = - =
Drag coefficient (viscosity):
In the PRL, time is normalized by φ0 and 0 0' ( ) / 2.2m d f f= - =
contributes as the resistivity force( )
oiav t
( ) ( ) ( )d
m v t v t f tdt
n=- +
Langevin equation is naturally accepted for colloid in physics
Conventional finance theory neglects the inertia term
0 ( ) ( )v t f tn=- + ( ) ( ) /v t f t n=
mass resistivity random force
We proved existence of the inertia term for price motion
( ) ( ) ( ) ( )dv t t v t G t
dtm=- +
Resistivity is time‐dependentNoise term is not white but highly correlated(to vanish the autocorrelation of v)
( ) /t mm n=
Brownian motion of a colloid particle
Random walk of Market prices
time
trajectory
Surrounding molecules are observable
“Particle” is not directly observable
From motion of surrounding molecules the dynamics of “Particle” is estimated
Price
Water molecules are invisible“Colloid particle” is observableFrom trajectory of a colloid particleThermal motion of water molecules are estimated.
price
The biggest difference is that
Molecule number 1023
Ingenhousz 1785Brown 1827Enstein 1905Perin 1907
Bachelier 1900
A financial molecule costs 1 million USD!
A lot of similarities are confirmed
( ) ( ) ( ) ( )dv t t v t G t
dtm=- +
Langevin equation with random resistivity produces power law distributions (known as Kestin‐process)
( ) (1 ( ) ) ( ) ( )v t t t t v t f tm+D = - D +
( ) ( ) ( ) ( )v t t b t v t f t+D = +
( | |) | |P v v a-³ µ ( ) 1b t a< >=
Power law distributions of financial markets can be explained by temporal fluctuation of resistivity of markets
PUCK model
Deviations from RWDealer model
Potential forceTrend‐follow
Time dependent Langevin eq.
Validity of continuum description
ARCH・GARCH model
Viewing only variance
Inflation Eq.
Macro limit
Crash alarm
Change‐point detection
New PhysicsFinancial Technology Macro Economy
Practical application
Time series analysisArtificial Markets
Our data analyses and models of financial markets
Order‐Book analysis
Layered OB model
Statistical properties of OB and correlation between changes in number of order‐book and velocity
Slow response of outer‐layer contributes as "viscosity"
Knudsen number of markets
1992‐2014‐
1997‐
2014‐
2015‐
2005‐
2002‐
2009‐
2009‐
2012‐
2009‐
2013‐
All empirical laws fulfilled
PUCK model bridges physics and finance
2
( )( ) ( 1) ( ) ( ) ( ) ( )
1( )
( ) ( ) ( )2( 1)
M
M
b tv t x t x t x t x t f t
Mb t
x t x t f tx M
1
0
1( ) ( )
M
Mk
x t x t kM
Potential of Unbalanced Complex Kinetics Misako Takayasu et alPhysica A370 (2006) , 91PRE 79(2009), 051120PRE 80 (2009), 056110
Moving average of price
( ) ( ) Mx t x t
Non‐trivial relation
Random walk
: Moving average of market price
( 1) ( )x t x t
potential
Price chart
Market Potential
Potential coefficient
An example of PUCK analysis for a typical day
b>0b<0
stable unstable
Microscopic anomalies are reproduced by PUCK model
Real data
PUCK model using measured {b(t)} with random noise f(t)
Time series Diffusion
Auto-correlation
PUCK potential
Market potential force in the dealer model
Prediction termContrarian (d<0)
Stable potential (b>0)
Trend‐follower (d>0)
Unstable potential(b<0)
Dealers’ strategy is gradually changed from contrarian to trend‐follower(d<0)
(d>0)
K. Yamada et al, EPJ B 63(2008), 529, PRE 79(2009)051120
The first attack
The second attack
News Effects in Real Markets 9.11.2001
Market potential changed drastically by the news
The market was stable. It became very unstable. It gradually stabilized.
President Bush appeared
Pentagon attacked
A WTC collapsed
22.0
0
‐2.0
1.0
‐1.0
Stable
Unstable
( )b t
Observation of asymmetric potentials before large crash (1998 for a month) Yen‐Dollar Physical Review E 80 (2009), 074911
Stable attractive potential
Asymmetric potentialSymptom of a crash
Unstable but symmetric random walk
Asymmetric potential: onset of crash
3
1
( 1) ( ) (( ) ) ( ) ( )kM
kkP t P t P t Pb t tt f
PUCK model can be used by financial practitioners via Bloomberg App Portal:
“Order Book Tracer”
Derivation of time evolution of variance by PUCK model( )( ) ( ) { ( ) ( )} ( )2
b tx t t x t x t x t t f t
In the continuum limit, PUCK model becomes Langevin Eq.
( ) ( ) ( ) ( )dv t t v t F t
dt
( ) ( )
( )x t x t t
v tt
1 ( ) / 2( )
b tt
t
( )( )
f tF t
t
2
2( , ) { ( ) ( , )} ( , ) 0vw v t t vw v t D w v t
t v v
2 2( ) 2 ( ) ( ) 2v
dt t t D
dt
2 ( ') ( ) ( ')vD t t f t f t
2 2( ) 2 {1 2 ( ) } ( )v
t t D t t t t Time evolution of varianceARCH‐GARCH model
1. The mass is normalized to be unity2. Viscosity changes with time
Fokker‐Planck equation holds for the probability density of velocity
2M
2 2( ) ( ) ( )t v t w t dt
( , )w v t : Probability density of velocity
By applying the Particle filter version of PUCK model, PUCK parameters, b(t) (the potential coefficient) and M (the size of moving average) are estimated in very short time
Particle filter method:104 PUCK models with different parameters are run in parallel, and they evolve like species of life
A test: sudden change of parameters
Particle filter version
Conventional method
Particle filter PUCK is suitable for real time data analysis
Y.Yura, H.Takayasu, K.Nakamura, M.TakayasuJour. Stat. Compu. and Simu. Vo.84, 2073‐, 2014
stable
unstable
10 sec
M: scale of moving average shortens
In this example the change of market condition is detected within 10 seconds
Example: USD-JPY for a week (Mar 2011)
Intervention
Flash Crash
zooming
1 hourFlash Crash takes more than 10 minutes continuous unstable states with exponential growth
( ) * ( ) ( )dv t v t G t
dtm=+ +
If the market is too much unstable, the viscosity becomes negative, and price difference grow exponentially !
( ) exp( * )v t tmµ
11 years dayly data of Australia Dollar / Japanese YenNon‐stationary time series analysis・change‐point detection
By human eye we may segment the time series
A‐h. Sato and H.Takayasu, arXiv:1309.0602
Here, there is bias from left down to right topWe apply this method for price changes
0 left s right t
x
There are many methods for change point detection, here, weIntroduce a most basic non‐parametric method (not assuming specific functional form of statistical model)
We apply Fisher’s exact probability test for the change‐point detection.Non‐stationary time series is segmented into stationary periods by the p‐value ( the probability that steady independent random fluctuation can realize the bias)(Fisher's test is very popular in medicine)
Left Right
UP a b a+bDown c d c+d
a+c b+d n
Assuming that the whole interval is statistically stationary,then this level of deviation (a,b,c,d) occurs with probability less than10‐25
We consider that the minimum point is the strongest change‐point
The time series is segmented automatically by the statistical analysisFor p‐value smaller than 10 ‐ 4
Price difference
p=10‐10
p=3x10‐27
p=6x10‐5 p=9x10‐9
p=3x10‐8
PUCK model
Deviations from RWDealer model
Potential forceTrend‐follow
Time dependent Langevin eq.
Validity of continuum description
ARCH・GARCH model
Viewing only variance
Inflation Eq.
Macro limit
Crash alarm
Change‐point detection
New PhysicsFinancial Technology Macro Economy
Practical application
Time series analysisArtificial Markets
Spending about 25 years, we are ready for the next step
Order‐Book analysis
Layered OB model
Statistical properties of OB and correlation between changes in number of order‐book and velocity
Slow response of outer‐layer contributes as "viscosity"
Knudsen number of markets
1992‐2014‐
1997‐
2014‐
2015‐
2005‐
2002‐
2009‐
2009‐
2012‐
2009‐
2013‐
All empirical laws fulfilled
Financial institutions
General public
Firms
StorageAnalysis Simulation
Countries
Observatory
bondscurrencies stocks
Markets
Investors
Change‐pointsPUCK StabilityMalfunctions・・・
StabilizationRule changesInterventions・・・
“Observing the markets as it is” is the first step for establishment of science of financial markets
The world situation
The natural environmentcommodities
derivatives
Financial Observatory of Real‐time Markets