Post on 11-Jan-2017
All of Virtual Bidding: A Data-Driven Approach
Wenyuan Tang1,2
joint work withRam Rajagopal2 Kameshwar Poolla1 Pravin Varaiya1
1University of California, Berkeley
2Stanford University
November 1, 2016
Wenyuan Tang Virtual Bidding 1 / 41
Two-Settlement Wholesale Electricity Market
Locational marginal prices (LMPs) reflect the value (price) of powerat different locations, and the LMP at a load-zone or a hub is theweighted average of the nodal LMPs
The day-ahead (DA) market lets market participants commit to buyor sell power one day before the operating day, and establishes 24hourly DA LMPs
The real-time (RT) market balances the differences between DAcommitments and the actual demand and supply during the course ofthe operating day, and establishes the 5-minute RT LMPs
Systematic nonzero spreads are routinely observed, which indicatessome market inefficiency
hourly spread = hourly DA LMP− hourly average RT LMP
Wenyuan Tang Virtual Bidding 2 / 41
Virtual Bidding
Virtual bids are included in DA dispatch, settled at DA LMPs,liquidated at RT LMPs
Allows participants to take financial positions in DA withoutdelivering or consuming physical power in RT
Hedging tools for physical entities; arbitrage tools for financialentities; adding liquidity; mitigating market power
Goals: enhancing market efficiency through DA/RT price convergence(financial efficiency) and dispatch convergence (economic efficiency)
Virtual supply (INC): generation bid in DA to be closed in RT
Virtual demand (DEC): demand bid in DA to be closed in RT
DA RT
spread INC
DA RT
spread =INC profit
Wenyuan Tang Virtual Bidding 3 / 41
Outline
Part I: Exploratory Data Analysis
Part II: Virtual Bidding and Financial Efficiency
Part III: Virtual Bidding and Economic Efficiency
DataAnalytics
Micro-economics
GameTheory
EmpiricalAnalysis
Virtual Bidding
Theory
Two-Settlement Market
Model
FinancialEfficiency
EconomicEfficiency
Wenyuan Tang Virtual Bidding 4 / 41
Part I: Exploratory Data Analysis
CAISO
Independent System Operator (ISO)
DA market: Apr 2009
Virtual bidding: Feb 2011
Data (NP15, 2010/2012): DA/RT LMP
Peak load ≈ 50 GW
PJM
Regional transmission organization (RTO)
DA market and virtual bidding: Jun 2000
Data (RTO, 2012–2015): DA/RT LMP,DA/RT/forecast load, INC/DEC
Peak load ≈ 150 GW
Wenyuan Tang Virtual Bidding 5 / 41
DA and RT LMP
●● ● ● ●
●
●● ● ● ● ● ● ● ●
●●
●
●● ●
●
●●
● ● ● ● ● ●
● ●
● ● ● ●●
●
●
●●
●
●
●●
● ● ●
Mean
Standard Deviation
0
10
20
30
40
50
0
10
20
30
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour
LMP
($/
MW
h)
Market● DA
RT
Statistics of DA and RT LMP, PJM, 2012
spread = DA LMP− RT LMP ≈ 0
RT LMP is more volatile than DA LMP
Wenyuan Tang Virtual Bidding 6 / 41
Spread
−200
−100
0
100
200
Mar Jun Sep Dec
Spr
ead
($/M
Wh)
Hour
4
18
Hourly Spread Time Series, PJM, 2012
0.00
0.05
0.10
−200 −100 0 100 200Spread ($/MWh)
Den
sity Hour
4
18
Hourly Spread Histogram, PJM, 2012
The distribution of the spread is heavy-tailed and left-skewed
Wenyuan Tang Virtual Bidding 7 / 41
Spread
−500
−250
0
250
Mar Jun Sep Dec
Spr
ead
($/M
Wh)
Hour
4
18
Hourly Spread Time Series, PJM, 2014
The polar vortex triggered two extreme weather events in Jan 2014
Recent data do not support classical models, e.g., [Bessembinder &Lemmon 2002], which states that spread is negatively related toVar(RT LMP), and positively related to Skew(RT LMP)
Wenyuan Tang Virtual Bidding 8 / 41
Virtual Bids
●
●
●●●
●
●●
●
●
●●●●
●
●
●
●●
●
●●●●●●
●
●
●
●●●●
●●●
●
●●●●●
●●
●●
●
●●●
●
●●●●
●●
●●●●
●●
●●
●
●
●
●
●●
●●
●
●●●●●●
●
●●●
●●
●
●
●
●
●●●
●●●
●●
●●
●
●
●●●●●
●●
●●●●
●●●
●
●●
●●
●
●●●●
●●●●●●
●
●
●●
●
●
●●●
●●
●
●
●●
●●●●●●
●●
●●●●●
●●●●
●●●
●●
●
●
●
●●
●●●●●
●
●●
●● ●●●●
●●●●
●●●●
●
●●●
●●●
●●
●
●●●
●
●●●
●
●●
●
●
●●
●
●
●●
●●●●
●●
●●
●●●●●●●●●
●●
●●●●
●●●
●●
●●●●●●●
●
●●●●●
●
●
●
●●●●●
●●●●●●
●●●
●●●●●
●
●●●●●●
●
●
●●
●
●●
●●
●●●●
●
●
●●●
●●●●●
●
●●
●●●●●●●●
●●
●
●●●
●
●●
●●
●●
●●●
●
●
●
●
●
●
●●●●●● ●●●●
●●
●●●
●●●●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●●●●
●●
●●●●
●●●●●●
●●
●●
●
●●●
●●
●
●●●
●
●●
●
●●●●
●
●●●
●
●●
●●
●●
●●●
●●●●
●
●●●●
●
●●
●●●
●●
●●
●
●●●●
●●●●
●
●●●
●
●●
●●●●●
●
●
●
●●●
●
●●●
●
●●●●●●●
●
●
●
●●
●
●
● ●●●●●● ●
●●
●
●●
●●●
●●●
●
●
●
●
●
●● ●
●●
●
●
●●
●
●●
●
●●●●●●
●
●●
●
●
●●
●●
●
●
●
●●
●●●●
●●
●
●●●
●
●
●
●●
●
●
●●●●●●●●●
●
●
●
●●
●●
●●
●●
●●●●●●●●●●●●●
●●●●
●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●●
●●
●●
●●
●
●●●●●
●
●●
●●●●
●
●
●●●●
●
●●
●●
●●●●
●●
●
●
●
●
●●
●●●●●
●●●●
●●●●
●●
●
●●●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
●
●●
●●
●●
●
●
●●
●●●●
●
●
●●
●
●
●●●
●
●
●
●●●●
●●
●●●●●●●●
●
●
●●
●●
●●●●●
●●●●●
●●●●●●
●
●
●●●
●
●●
●●●●●
●
●
●●
●●●●●
●●●
●●
●●●●
●
●●●
●
●
●●
●
●●
●●
●●
●
●
●●●●●
●
●●●
●
●
●●
●
●●
●
●●●●●
●●●
●
●●
●●
●●●●●●
●●
●
●
●●
●●●
●●●
●
●
●●
●●●
●
●●
●
●●
●
●●●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●●
●●
●●●●
●●
●●●●●●
●
●
●●●●●●
●●
●●
●
●●●
●●●●
●
●●
●●●
●●
●●●●●
●
●
●
●
●
●●●●
●●
●●
●
●
●
●
●
●
● ●
●
●●●●
●●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●
●
●
●●
●●●
●
●●●
●
●●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●●●
●
●
●●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●●
●
●
●
● ●
●
●
●
●
●
●
●●●
●
●
●
●
●●●●
●
●
●
●●●
●●
●
●●●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●●●●●●
●
●
●●●
●●●
●
●●●●
●●
●●●
●●●
●●
●
●
●
●●●●
●●
●
●●
●
●●●●
●
●●
●
●
●
●
●
●
●●
●
●
●●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●● ●
●●
●
●●
●
●
●
●
●●
●●
●
●
●
●●
●
● ●
●
●
●●●
●●
●
●
●●
●●● ●
●●
●●
●
●
●
●
●
●●
●
●●
● ●●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●●
●●
●●
●
●● ●
●
●●
● ●●
●
●
●
●
●
●
●
●●●●
●
●●●
●
●
●●
●
●
●
● ●●●●
●
●●
●●
●
●
●●
●
●
●
●
●
●●
●
●●
●●●
●●
●
●
●●
●●
●
●
●
●
●
●●●
●●
●●
●●●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●●●●● ●●●
●
●●
●
●
●
●
●
●●●●●
●
●●●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●●●●
●
●
●●
●
●
●
●
●●
●●●●●●●●
●
●
●●●●
●●
●●●●
●
●
●
●●
●
●
●
●
●
●●●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
● ●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●●
● ●
●
●●●
●●
●
●●
●
●
●
●
●●●●●●
●
●●●
●●●●●
●
●●
●●
●●●●
●●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●● ●
●
●●
●
●●●●
●
●
●
●
●●●
●●
● ●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●●●●●●
● ●
●
●
●
●●
●
●●●
●●
●●●●●
●●
●●●
●●
●
●●●●
●●●
●●●
●
●
●●
●
●●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●●
●●
●●
●●
●
●●
●
●
●●●●●
●
●●●
●
●
●●●
●●●
●
●●
●●●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●●●
● ●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●●●
●
●
●●●●●
●●
●
●●
●●●●●●
●●●●
●●
●●●
●●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●●●●●
●
●
●
●●
●
●
●
●
●
●
● ●
●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●●●●
●●
●
●●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●● ●●
●
●
●
●
●
●●
●
●●●●●
●
●●●
●
●
●
●
●
●
●●
●
●
● ●●
●
●●
●●
●●
●
●
●
●●●●●●
●●●●
●●
●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●●
●
●
●
● ●
●
●●
● ●●
● ●
●
●
●
●
●●●
●
●
●
●●
●
● ●●
● ●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●
●●
●●
●
●
●●●
●●
●
●
●●
●
●
●●●
●
●
●
●●
● ●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●
●
●●
●
● ●●●
● ●●
●●
●
●●
●● ●●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
● ●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●●●●●●
●
●●
●
●●
●●●
●●●
●●
●
●●
●
●
●●
●
●●
●●
●
●●●
●●●
●
●
●
●
●●
●
●
●
●●● ●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●
●●
●●
●
●●
●
●● ●
●
● ●
●●
●
●
● ● ●
●
●●●●●
●
●
●
●
●
●
●●
●
●
●
●●●●
●●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●●●●
●
●
●●
●●
●
●
●
●●
●
●●●
●●
●●●
●
●●
●●
●●● ●
●●
●
●
●●
●
●
●●
●●●●●
●●
●●
●
●
●●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●●●
●
●
●●
●●●●
●●
●
●
●
●● ●●●●
●
●
●
●
●●
●●●
●●
●●
●
●
●●●●
●
●
●●
●
●
●●●
●●●
●
●●●●●●
●
●
●
●●●
●
●●
●
●●●●
●
●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●
●
●●●●●
●●
● ●
●
●
●●
●
●
●●●●●●●
●
●●●●
●●●
●●
●●●
●
●
●●●
●
●●
●●●●●●●
●●
●●
●
●●
●●●
●
●●
●
●
●●
●●●●
●
●●
●●●●
●●●
●
●
●●●●●
●●
●●
●●
●●
●● ●●●●●
●●
●●●
●●●●●
●●
●
●●●
●
●●
●●●●●●
●
●
●●●
●●●
●
●
●
●●
●
●●●●●
−15000
−10000
−5000
0
5000
−400 −200 0 200Spread ($/MWh)
Net
INC
Vol
ume
(MW
)
● Profit
Loss
Hourly Net INC Volume versus Spread, PJM, 2012
profit = spread× (INC− DEC) = spread× net INC
Virtual bidders expect negative spreads, or RT LMP spikes
Wenyuan Tang Virtual Bidding 9 / 41
Virtual Bids
2012 2013
2014 2015
($20)
$0
$20
$40
($20)
$0
$20
$40
Mar Jun Sep Dec Mar Jun Sep Dec
Mar Jun Sep Dec Mar Jun Sep Dec
Cum
ulat
ive
Pro
fit (
Mill
ions
)
Hours
All
Normal
Abnormal
Profit of Virtual Bids, PJM
Normal hours (98% of all): spread between 1st and 99th percentile
Abnormal hours (2% of all): otherwise
Wenyuan Tang Virtual Bidding 10 / 41
Performance Metric: Sharpe Ratio
N: number of days, R: daily profit
Sharpe ratio =√N
E[R]√Var(R)
Four-year (2012–2015) Sharpe ratio of S&P 500 is 1.68
Sharpe ratios of the PJM virtual bids
Year All Hours Normal Hours Abnormal Hours
2012 1.80 0.40 1.99
2013 0.96 −2.09 2.77
2014 0.39 −1.42 0.93
2015 4.31 1.60 4.82
Total 1.79 −1.29 2.56
Virtual bidders speculate on extreme events
Wenyuan Tang Virtual Bidding 11 / 41
Part II: Virtual Bidding and Financial Efficiency
We define financial efficiency as DA/RT price convergence
DA LMP = E[RT LMP|DA LMP]
orE[spread|DA LMP] = 0
which implies
ρ(spread,DA LMP) = 0 and E[spread] = 0
It is not clear that virtual bidding improves the financial efficiency
●
● ●
●
●
●
●
● ●●
● ●
●
●
●
● ● ●●
● ● ●
●
●
−0.2
−0.1
0.0
0.1
0.2
0.3
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour
Cor
rela
tion Year
● 2010
2012
Correlation between Spread and DA LMP, CAISO
Wenyuan Tang Virtual Bidding 12 / 41
Part II: Virtual Bidding and Financial Efficiency
●●
●
●
● ●
●
●
● ●
● ●
●●
●
●●
●●
● ●
●
●
●
●
●
●
● ● ●
● ● ●
●
●●
●●
●
● ●
●●
● ●
●
●●
● ●●
● ● ●● ●
● ●●
●● ●
●
● ●
● ● ●●
●
●●
Mean
Standard Deviation
Mean Absolute Value
−10
−5
0
5
10
0
20
40
60
80
0
10
20
30
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour
Spr
ead
($/M
Wh)
Year● 2010
2012
Statistics of Spread, CAISO
Mean is closer to zero after the implementation of virtual bidding
But standard deviation and mean absolute value remain high
Wenyuan Tang Virtual Bidding 13 / 41
Part II: Virtual Bidding and Financial Efficiency
Alternative approach: testing whether profitable bidding strategiesexist before and after the implementation of virtual bidding [Li,Svoboda & Oren 2015], [Jha & Wolak 2015]
We propose a measure that tests the randomness of the sequence ofthe spread: more random spread leaves less room for arbitrageopportunities
We examine the autocorrelation of the sequence of the spread andpropose a benchmark bidding strategy that is only based on theup-to-date price information
We employ machine learning methods to design more sophisticatedbidding strategies that utilize other data such as load
Note that our definition of financial efficiency does not depend ontransaction costs, which are therefore not considered
Wenyuan Tang Virtual Bidding 14 / 41
Wald-Wolfowitz Runs Test on sgn(spread)
●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●●●
●●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●
●●
−400
−300
−200
−100
0
100
Mar Jun Sep Dec
Spr
ead
($/M
Wh)
Spread● > 0
< 0
Runs Test on Sign of Spread, CAISO, 2010, Hour 18
A run is a segment of consecutive “+”s or “−”s
N = 365 = N+ + N− = 268 + 97. Under the null hypothesis that thesequence is i.i.d.
µ = (2N+N−/N) + 1 = 143, σ = (µ− 1)(µ− 2)/(N − 1) = 7.4
110 runs, few enough to reject the null hypothesis (at α = 0.05)
Wenyuan Tang Virtual Bidding 15 / 41
Runs Test Results (p-values ≤ 0.05 in bold)
Hour 2010 2012 Hour 2010 2012
1 0.016 0.331 13 0.000 0.074
2 0.026 0.028 14 0.000 0.123
3 0.161 0.020 15 0.002 0.002
4 0.002 0.625 16 0.000 0.002
5 0.063 0.352 17 0.002 0.002
6 0.050 0.998 18 0.000 0.001
7 0.000 0.561 19 0.000 0.033
8 0.021 0.730 20 0.000 0.000
9 0.016 0.501 21 0.006 0.323
10 0.524 0.066 22 0.000 0.000
11 0.000 0.001 23 0.000 0.005
12 0.000 0.001 24 0.002 0.703
The spread is more random after virtual bidding
Wenyuan Tang Virtual Bidding 16 / 41
Exploring Intertemporal Correlation
95% CI
95% CI
2010
2012
−0.1
0.0
0.1
0.2
0.3
−0.1
0.0
0.1
0.2
0.3
0 5 10 15 20 25Lag (Day)
PAC
F
Partial Autocorrelation of Daily Average Spread, CAISO
The spread is more random after virtual bidding
The strong lag-1 autocorrelation motivates the benchmark strategy
Wenyuan Tang Virtual Bidding 17 / 41
Lag-1.5 Algorithm
sgn(∑
spread) sgn(∑
spread)forecast
deadline
a.m. p.m. a.m. p.m. a.m. p.m.
day t − 2 day t − 1 day t
Bids for day t should be submitted by noon on day t − 1
Lag-1.5 forecast (sht : spread at hour h on day t)
sgn
(24∑
h=13
sht−2 +12∑h=1
sht−1
)→ sgn
(24∑h=1
sht
)
If “+”, trade 1 MW INC for each hour
If “−”, trade 1 MW DEC for each hour
Wenyuan Tang Virtual Bidding 18 / 41
Lag-1.5 Algorithm
2010
2012
($5,000)
$0
$5,000
$10,000
($5,000)
$0
$5,000
$10,000
Mar Jun Sep Dec
Mar Jun Sep Dec
Cum
ulat
ive
Pro
fit
Hours
All
Normal
Abnormal
Profit of the Lag−1.5 Algorithm, CAISO
Pre-VB: average profit $1.35/MWh, Sharpe ratio 1.93
Post-VB: average profit $1.08/MWh, Sharpe ratio 1.35
Wenyuan Tang Virtual Bidding 19 / 41
Support Vector Machine
Input variables: {spread, DA load, RT load, forecast load} of dayt − 7, t − 6, . . . , t − 2
Output variable: sign of the daily spread of day t
Training data: 365 samples in year y
Test data: 365 samples in year y + 1
Sharpe ratios of the PJM virtual bids and SVM
YearPJM SVM
All Normal Abnormal All Normal Abnormal
2013 0.96 −2.09 2.77 1.66 4.72 −1.26
2014 0.39 −1.42 0.93 1.62 3.19 0.98
2015 4.31 1.60 4.82 1.29 3.75 −0.81
Total 1.19 −1.60 1.97 2.17 6.05 0.63
There is still room for arbitrage opportunities: even a simple machinelearning algorithm works well
Wenyuan Tang Virtual Bidding 20 / 41
Support Vector Machine
2013 PJM
2014 PJM
2015 PJM
2013 SVM
2014 SVM
2015 SVM
($10,000,000)
$0
$10,000,000
$20,000,000
($20,000,000)
$0
$20,000,000
$40,000,000
$0
$10,000,000
$20,000,000
($2,500)
$0
$2,500
$5,000
$7,500
$0
$10,000
$20,000
$30,000
($3,000)
$0
$3,000
$6,000
$9,000
Mar Jun Sep Dec
Mar Jun Sep Dec
Mar Jun Sep Dec
Mar Jun Sep Dec
Mar Jun Sep Dec
Mar Jun Sep Dec
Cum
ulat
ive
Pro
fit
Hours
All
Normal
Abnormal
Profit Comparison between PJM Virtual Bids and SVM
SVM capture the patterns: profitable in normal hours
Actual virtual bidders speculate on the extreme events
Wenyuan Tang Virtual Bidding 21 / 41
Part III: Virtual Bidding on Economic Efficiency
We define economic efficiency as generation cost minimization
Why is load forecast important?
Why is DA load close to RT load?
Why is RT supply curve steeper than DA?
How is economic efficiency related to price convergence?
How is economic efficiency related to dispatch convergence?
How to explain the phenomena: negative RT LMP; DA load close toRT load, but DA LMP far apart from RT LMP; etc.
How does virtual bidding affect DA/RT dispatch?
How to estimate the generation cost with and without virtual bids?
Wenyuan Tang Virtual Bidding 22 / 41
DA/RT Load
●
●● ● ●
●
●
●
●
●
●● ● ● ● ●
●
● ● ● ●
●
●
●
●
●● ●
●
●
●
●
●
●● ● ● ● ● ●
●
● ● ● ●
●
●
●
●
●● ●
●
●
●
●
●
●● ● ● ● ● ●
●● ● ● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
● ● ● ● ●●
● ● ● ●
●
●
●
2012 2013
2014 2015
70
80
90
100
70
80
90
100
0 6 12 18 24 0 6 12 18 24Hour
Load
(G
W)
Market● DA
RT
Mean of DA and RT Load, PJM
DA load is close to RT load
Wenyuan Tang Virtual Bidding 23 / 41
DA/RT Generation
●
●● ● ●
●
●
●
●
●
●● ● ● ● ●
●
● ● ● ●
●
●
●
●
●● ●
●
●
●
●●
●
● ● ● ● ● ●●
● ● ● ●
●
●
●
●
●● ●
●
●
●
●
●
●● ● ● ● ● ●
●
● ● ● ●
●
●
●
●●
● ●●
●
●
●
●
●
● ● ● ● ● ●●
● ● ● ●
●
●
●
2012 2013
2014 2015
70
80
90
100
70
80
90
100
0 6 12 18 24 0 6 12 18 24Hour
Gen
erat
ion
(GW
)
Market● DA
RT
Mean of DA and RT Generation, PJM
DA generation is close to RT generation
DA generation may not equal DA load due to virtual bids
Wenyuan Tang Virtual Bidding 24 / 41
DA/RT LMP and Generation
● ● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●● ● ● ●●●● ●●●● ● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●●● ●●●●●●● ●●● ●●●● ●● ●●●●● ● ●●●●● ● ●●●●●● ● ●●●●●●●● ● ●●●●● ●●●●●● ● ● ●●●●● ● ●● ● ● ●● ● ●●●●●● ●●●● ●●● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●●● ●●● ● ● ●●●●● ● ●● ● ●●● ● ● ●● ●●●●● ●●●●●● ●●●●●● ● ●●● ●●●● ●●● ● ●●● ● ●●●●●●●● ●●●● ● ●●● ●●● ●● ● ●●●●● ● ●●●●●● ●● ●●●●●● ●● ●●●● ●●● ●●● ●● ●●●●● ● ●● ●●●●● ● ●●●●●● ●●●● ● ●●● ●●●●●●●● ●●●●● ● ● ●●●●● ● ● ●● ●●●● ● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●● ● ● ●●●● ●●● ●● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●●● ●● ●●●●● ● ● ●●●● ● ●●●●●● ● ●●●● ●●●● ● ●●●●● ●●●●●● ● ● ●● ●●● ● ●● ● ● ●●● ●●● ●●● ●●●● ●●● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●●●●● ● ●●●●●●● ●● ● ● ● ●●●●● ● ●● ● ●●●● ●●● ●●●●● ●●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●●● ● ●●●●●● ●● ●●●● ● ●●●●●● ● ● ● ●●●●● ● ●●●●● ● ●●● ●●●● ● ●● ●●● ● ●●●●●● ●● ●●●●● ● ●● ●●●●● ● ●●●●●● ●●●● ● ●●● ●●●●●●● ● ●●●●● ● ● ●●●●● ● ● ●● ●●●● ●● ●●●●● ● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●● ● ● ●●●● ●●● ● ● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●●● ●● ●●●●● ●● ●●●● ● ●●●●●● ● ●●●● ●●●● ● ●●●●● ●●●●●● ● ● ●●●●● ●●●● ●●● ● ●●● ●●● ●●●● ●●● ●●●●●● ● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ● ●● ● ● ● ●●●●● ● ●● ● ●●● ● ●●● ●●●●● ● ●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●● ● ● ●●●●●● ●● ●●●● ● ●●●●●● ● ● ●●●●● ● ● ●●●●● ● ●● ● ●●●● ● ●● ●●● ●●●●●●● ●● ●●●●● ● ●● ●●●● ● ● ●●●●● ● ●●●● ● ●●● ●●●●●●● ● ●●●●● ● ● ●●●●●● ● ●● ●●●● ● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●● ● ●●● ● ● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●●● ●● ●●●●● ● ● ●●●● ● ●● ●●●● ● ●●●● ●●●● ● ●●●●● ●●●●●● ● ● ●● ●●● ●●● ● ●●●● ●●● ●●● ●●●● ●●● ●●●●●● ● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ● ●● ● ● ● ●●●●● ● ●● ● ●●● ● ●●● ●●●●● ● ●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●● ● ● ●●●●●● ●● ●●●● ● ●●●●●● ●●●●●●●● ● ●●●●● ● ●● ● ●●● ● ● ●● ●●● ●●●●●●● ●● ●●●●● ● ●● ●●●● ● ● ●●●●● ● ●●●● ● ● ●● ●●●●●●● ● ●●●●● ● ● ●●●●● ● ● ●● ●●●● ●● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●● ● ●●● ●● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●● ● ●● ●●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●● ●● ● ●●●● ● ●●●●●● ●●●●●●● ●●●● ●●● ● ●●● ●●● ●●●● ●●● ●●●●●● ● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ●●●●● ● ●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●● ● ● ●●●●●● ●● ●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ● ●● ● ●●● ● ● ●● ●●● ●●●●●●● ●● ●●●●● ● ●● ●●●● ● ● ●●●●● ●●●●●● ● ●● ●●●●●●● ● ●●●● ● ● ● ●●●● ● ● ●●● ●●●● ●● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●● ● ●●● ● ● ●●● ● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ● ●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●● ●●●● ● ●●● ●●● ● ●● ●●●● ● ●● ●●●● ● ●●●●●● ● ●●●●●● ●●● ●●●● ● ●●●●●● ●● ●●●●● ●●●● ●●● ● ●●● ●●● ●●●● ●● ●●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ●●●● ● ● ●●●●●
● ●● ● ●●● ● ●●● ●●● ●● ●●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ●●●● ● ●●●●● ● ●● ●●●● ● ●●●●●● ●● ● ●●●● ● ● ●●●●● ● ●● ●●●● ● ● ●● ●●● ●●●●●●● ●● ● ●●●● ● ●● ●●●● ● ● ●●●●● ●●●●●● ● ●● ● ●●●● ●● ● ●●●● ● ● ● ●●●● ●● ●●●●●●● ●● ●●● ●● ●● ●
●●●● ● ● ●● ●●● ●● ● ●●●● ● ● ●●●●●
●●● ● ●●●●● ● ●●● ●
●●●●●●● ● ●●●●● ●
●●●● ●● ●● ●●
● ●●● ●●●●●●● ● ●●●●●● ●●
●● ●●● ● ●● ●●●● ● ● ● ●●●● ● ●● ●●●● ● ●●●●●● ●●● ●●●● ● ●●●●●● ●● ●●●●● ●● ●●●●● ● ●●● ●●● ●●●● ●● ●●●●●●● ●
● ● ● ●●●●●●● ● ●●● ●●● ● ●●● ●●● ●●●● ● ● ●●●●●
● ●● ● ●●●● ●●● ●●● ●● ●●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ●●● ● ● ● ●●●● ●●● ●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●●
● ●● ● ●●● ● ● ● ●●●●●●●● ●●●
●●●● ●
●●● ●● ●●
●● ● ● ●●●●● ●● ●● ●● ●
●● ●●●●● ●● ● ●●●● ● ● ●● ●●● ●● ●●●●●●● ● ● ●●● ●● ●●●
●●●● ●● ●● ●●● ●● ●● ●
●● ● ●●
●●●●●●●● ●●●
●●● ●●● ●●
●●●●●● ● ●●●●● ●●●●● ●● ●
● ●●
● ●●● ●●●●●●● ● ●●●●●● ●●
●● ●●● ● ●●●●●● ● ● ● ● ●●● ●●●●●●●
● ●●●●●● ●● ● ●●●● ● ●●●●●● ●● ●●●●● ●● ●● ●●●● ●●● ●●● ●●●● ●●● ●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●●● ●●●● ●●● ●●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ●●● ●●● ●●●● ●● ●●●●● ●●● ●●●● ● ●●●●●●● ● ● ●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●●● ●● ● ●●● ●● ● ●●●●
●●●●●●●●●
●●●●●
● ● ●●●●●
● ●●●●
●● ●●●● ●● ●
●● ● ●●●● ●●●●●●● ● ● ● ●●●● ●● ● ● ●●●●● ● ● ●●● ●● ●
● ●●●●● ●● ●● ●●● ●● ● ●●●● ● ● ●●●●● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●●●● ●●●● ●● ● ● ●●● ●●● ●●●●●●● ● ●●●●●● ● ●●● ●●● ● ●●●●●● ●● ●● ●●● ●●●●●●● ●●●●●●● ● ● ● ●●●● ● ●●●●●● ●● ●●●●● ●● ●●●●●● ●●● ●●● ●●●● ●●● ●●●● ●●● ● ● ● ●●●● ●●● ● ●●● ●●● ●●●● ●● ● ●●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ●●● ●●● ●●●● ●● ●●●●● ●●● ● ●●● ●●●● ●●● ● ● ●●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●●● ●● ●●●● ●●●●●●● ●●●●●●● ●●
●●●●●● ●●●●●● ● ● ●●●●● ●●●● ● ● ●
●● ●●●●● ●● ● ●●●● ● ● ●● ●●● ●●● ● ●●●●● ● ●●●● ●● ●●●●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●●● ●●● ●● ●●●● ● ●●●●●●● ● ●●● ●●● ●●●●●● ● ● ●●● ●●● ●●●●●●● ● ●●● ●●● ● ●●●●●● ● ●●●●●● ●● ●●●●● ●●●●●●● ●●●●●●● ● ● ●●●●● ● ●●●●●● ●● ●●●●● ●● ●● ●●●● ●●● ●●● ●●●● ●●● ●●●● ●●● ● ● ●
●●●● ●●● ● ●●● ●●● ●●●●●●● ●●●● ● ● ●●●●● ● ●● ●●●● ●●● ● ●●● ● ●● ●●●● ●●●●●●● ●● ● ● ●●● ● ●● ● ●●● ● ● ●●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ● ●●●● ● ●● ●●●● ●●●● ●●● ●●●●●●● ●●
●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ● ● ●●●●●●● ●● ● ●●●● ● ● ●●●●● ●●● ● ●●●●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ● ●●●●● ●●●● ●●● ●● ●●● ●● ●●●●●●● ● ●●● ●●● ●● ●●●●● ● ●●● ●●● ●●● ●●●● ● ●●● ●●● ● ●●●●●● ●●●●●●● ●● ●●●●● ●●●●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ●● ●●●●● ●● ● ● ●●●● ●●● ●●● ●●● ● ●●● ●●●● ●●● ● ● ●●
●●● ●● ● ● ●●● ●●● ●●●●●● ● ●●●● ● ● ●●●●●
● ●● ●●●● ●●● ● ●●● ● ●● ●●●● ●●● ●●●● ●● ● ● ●●● ● ●● ● ●●● ● ● ●●●●● ●● ●●●●● ● ●●● ●●● ●● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ●●●●●●● ●●●●●●● ● ● ●●●●● ● ●●●●●●
●●●● ● ●● ●●●●●●● ● ●● ●●●● ● ● ●●●●● ●●● ● ●●● ●● ● ● ●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ● ●●●●● ●●●● ●●● ●● ●●● ●● ●●●●●● ● ● ●●● ●●● ●● ●●●●● ● ●●● ●●● ●●● ●●●● ● ●●● ●●● ●●●●●●● ●●●●●●● ●● ●●●●●●●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●●●●● ●● ● ● ●●●● ●●● ●●● ●●● ● ●●● ●●●● ●●● ● ●
●●
●●● ●● ● ● ●●● ●●● ●●●●●● ● ●●●● ● ●●
●●● ●● ●●
●●●● ●●● ● ●●● ● ●● ●●●● ●●● ●●●● ●● ● ● ●●● ● ●● ● ●●● ● ● ● ●●●● ●● ● ●●●● ● ●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●● ●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ●● ●●●● ● ● ●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ●●●●●● ●●●● ●●● ●● ●●● ●● ●●● ●●● ● ● ●●● ●●● ●● ●●●●● ● ●●● ●●● ●●● ●●●● ● ●●● ●●● ●●●●●●● ●●●●●●● ●● ●●●● ●●●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●●● ●● ● ● ●●●● ●●● ●●● ●●● ● ●●● ●●●● ●●● ● ●
●
●
●●● ●● ●● ●●● ●●●
●●●●●● ● ●●●● ●●
●
●●● ●●●●
●●
●● ●●● ● ●●● ● ● ● ●●●● ●●● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●●● ●● ●●●●● ● ●●● ●●● ●● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ● ●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●● ●●●● ● ●● ●●●● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ●●●●●● ●●●● ●●● ●● ●●● ●● ●●● ●●● ● ● ●●● ●●● ●● ●●●●● ●●●● ●●● ●●● ●●●● ● ● ●● ●●● ●●●●●●● ●●●●●●● ●● ●●●● ●
●●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ●● ●● ●●● ●● ● ● ●● ●● ●●● ●●● ●●● ● ● ●● ●●●● ●●● ● ●
●
●
●●● ●● ●
●●
●● ●●●●
●●
●●● ● ●●●●●
●
●
●●● ●
●●●
●●
●● ●●● ●●
●● ● ● ● ●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ●●●●
●● ●● ●●●●● ● ●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ●● ●●●● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ● ●●●●●● ●●●● ●●● ●● ●●● ●● ●●● ●●● ●● ●●● ●●● ●● ●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●● ●●● ●●●●●●● ●●●●●●● ●● ●●●● ●●
●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●● ● ●● ● ● ● ● ● ● ●●● ●●● ●●● ● ●● ● ●●●● ●●● ● ●
●
●
●●● ●● ●
●●
●● ●●●
●●●
●●● ● ●●● ●●
●●
●●● ●
●●●
●
●●● ●●● ●
●
●● ● ●● ●●● ● ●● ● ●●
●● ● ● ● ● ●●● ● ●● ●●
●● ●●● ●●
●● ● ● ●●●●● ● ●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ● ●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ● ● ●●●● ● ●●●●● ● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ● ●●●●● ● ●●●●●●● ● ● ●●● ●● ●●● ●●● ●● ●●● ●●● ●● ●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●● ●●● ●●●●●●● ●●●●●●● ●●●●●● ●●
●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●● ● ●● ● ●● ●●● ●
●● ●●● ●●● ● ● ● ● ●●●● ●●● ●●
●
●
●●● ●● ●
●
●
●● ●●●
●●●
●●● ● ●●● ●
●
●●
●●● ●
●●●
●
●●●
●●● ●
●
●● ● ●● ●
●● ●●● ● ●●
●● ● ● ● ● ●●●● ●● ●
●●
● ●●●
●●
●● ● ● ●●●●● ● ●●● ●●
● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ● ●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ●●● ● ●●●●●● ● ● ● ●●● ● ● ●●●●● ● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●●● ●●●●● ● ●●●●●●● ● ● ●●● ●● ●●● ●●● ●● ●●● ●●● ●● ●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●● ●●● ●● ●●●●● ●●●●●●● ●● ●●●● ●●
●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●● ● ●● ● ● ● ●● ● ●●● ●●● ●●● ● ● ● ● ●●●● ●●●
●●
●
●
●●● ●●●
●
●
●● ●●●
●●●
●●● ● ●●
● ●
●
●●
●●● ●
●●●
●
●●● ●●●
●●
●● ●
●● ●●● ● ●● ● ●●
●● ● ● ● ● ●●●
●●●●
●●
● ●●● ●
●
●● ● ● ●●●●● ●●●● ●●
● ● ● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●● ●●●● ● ● ● ●●●● ● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ● ● ●●● ● ● ●●●●● ● ●●● ● ●●● ●● ●●●●● ●●●
●●●●● ● ●● ●●
●●●●● ●●●● ●● ●●●●●● ●●●●●● ● ● ● ●●● ● ●●●● ●●● ●●●●● ●●● ●● ●●●● ● ●●●● ●●● ●●● ●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ●●
●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ●●● ●●● ●●● ● ● ● ● ●●● ● ●●●●
●
●
●
●● ● ●●●
●●
●●●●●
●●●
●●● ● ●●● ●●
●
●
●●● ●●●●
●
●●● ●●●●●
●● ● ●● ●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●●●
●
●● ● ●● ●●
●● ● ● ●●●●● ●●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●●●●● ● ● ●●●● ● ●● ●●●●●
● ● ●●●● ● ● ●●●●●●●●●
● ● ● ●●●●●●●● ●●
● ●●● ● ● ● ●●●●● ●●● ● ●●●●● ●●●●● ●●
●●
●●●● ● ●● ●●●● ●●● ●●●● ●● ●●●●● ● ●●●●●●● ● ● ●●●
●● ●●●●●● ● ●●●● ●●● ●● ●●●● ● ●●●● ●●● ●●● ●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ● ●●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ●●● ●● ● ●●● ● ● ● ● ●●● ● ●●● ● ●
●
●
●● ●●● ●● ●
●●●●●
●●●●●● ● ●●● ●
●
●
●
●●● ●●●●
●●●● ●●● ●●
●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●●● ●
●● ● ● ● ●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ● ●●●●●● ●●● ●●● ● ●● ●●●● ● ●●●●●● ● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ●● ● ●●● ● ● ● ●●●●● ● ●● ● ●●●●● ● ●●●●●●
●●
●●●● ● ●● ●● ●● ●●● ●●●●●● ●●●●● ● ●●● ●●●● ●● ●●● ● ●●●●●●● ● ●●●● ●●● ●●●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ● ●●● ●●●● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ●●● ●● ● ●●● ● ● ● ● ●●● ● ●●● ● ●
●●
●●●●● ● ●●●● ●●●●●●●●● ● ●●● ● ●
●●●●● ● ●●●
●●●● ● ●● ● ●●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●
●● ● ● ●●●● ● ●●●● ●●● ●● ●●●●● ● ● ●●●●●●
●●●●●● ●●● ●●●● ●● ●●●● ● ●● ●●●● ● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ●● ● ●●● ● ● ● ●●●●● ● ●● ● ●●●●● ● ●●● ●●●●
●●●●● ● ●● ● ● ●● ●●● ●●●●●● ●●●●● ● ●●● ●●●● ● ● ●●● ● ●●●●●●● ● ●●●● ●●● ●●●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●●
●●● ● ●●●●●● ●●●●●●● ●● ●●●● ● ●●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ●● ●● ●● ● ●● ● ●● ●● ●●●● ●● ● ●●● ● ● ● ● ●●●● ●●● ● ●
● ●●● ● ●● ● ●●●● ● ●● ●● ●●●● ● ●●● ● ●
●●●●● ● ●● ● ●
●●● ●●● ●●●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●●● ●● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ● ●● ●●●● ●●●●●●● ●●●●●● ● ●● ●●●● ● ● ● ●●●● ● ● ●●●●●● ●●●● ● ●● ●●●●●●● ● ● ● ●●● ● ● ●●●●●● ● ●● ● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●●● ●●●●● ● ●●● ●●● ● ● ● ●●● ● ●●●●●●● ● ●●●● ● ●● ●●●●●● ● ●●●● ●●● ●● ●●●● ● ● ● ● ●●●● ●●●●●●● ●●●●●●● ●●●●●● ● ●●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ●● ● ● ●●●● ●● ● ●●● ● ● ● ● ●●● ● ●●● ● ● ●●
●● ●●● ● ● ●●●● ●● ●●●●●● ● ●●● ● ● ●●●●● ● ●● ● ●●● ● ●●● ●●●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ●●●● ● ● ● ●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ● ●● ●●●● ●●●●●●● ●●●●●●● ●● ●●●● ● ● ● ●●●● ● ● ●●●●●● ●●●● ● ●● ●●●●●●● ● ● ● ●●● ● ● ●●●●●● ● ●● ● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●●● ●●●●● ● ●●● ●●● ● ● ● ●●● ●●●●●●●● ●●●●● ● ●●●●●●●● ● ●●●● ●●● ●●● ●●●● ● ● ●●●●● ●●● ●●●● ●●●●●●● ●●●●●● ● ●●● ●●●● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ●●●● ●● ● ●● ● ● ● ●● ●●●● ●●● ●●● ● ● ● ● ●●●● ●●● ● ● ● ●●●●●● ● ● ●●●●●● ●● ●●●● ● ●●● ● ● ●●●●● ● ●● ● ●●●● ●●● ●●●● ● ●● ●●● ● ●● ●●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●●● ● ● ●●●●● ●●●●●●● ●● ●●●●● ● ● ●●●● ● ●●● ●●●● ● ●●●●●● ●●●●●●● ●● ●●●● ● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ● ● ● ●●● ● ● ●● ●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●●●● ● ●●● ●●● ●●● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●●● ●●● ●●●●●●● ●●● ●●●● ●●● ●●●● ●●● ●●●● ● ●●●●● ● ●●● ●●●● ●●●●●●●● ● ● ●●●● ● ●●●●● ● ●●●●●● ● ●● ● ● ● ●● ●●●●●● ● ●●● ● ●● ● ●●●● ●●● ● ● ●●●●●●● ● ● ●●● ●●● ●●●●●●● ●●● ● ● ●●●●● ● ●●● ●●●● ●●● ● ●●●● ●●●●● ● ●● ●●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ●●●●● ● ●●●●●● ●●●●●●● ●● ●●●●● ● ● ●●●● ● ●● ●●●●● ●●●●●●● ●●●●●●● ●●●●●● ● ●● ●●●●● ● ●●●●●● ●●●● ● ●● ● ● ●●●●● ●● ● ●●● ● ● ●● ●●●● ● ●●● ●●●●● ● ●●● ●●
99%
●●● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●●● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●●●●●● ● ●●●●●● ●●● ●●●● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●● ● ●●● ● ●●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ● ●●●● ●●● ● ●● ●● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●●● ●●● ● ●●●● ●●●●●● ● ●●●●●● ●● ● ● ●●● ●●● ● ●●● ● ●●●●●●●
● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ●●●●●
●● ● ●●●●●● ●● ●●●● ●●● ●●●● ● ●●●●●●● ● ● ●●●●● ●●●● ● ●●● ● ●●●●●●● ●●●●●
● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●●
● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●●●●● ● ●●●●●●
●
●●● ●●●● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●●● ●●● ● ●●● ● ●●●●●● ●●●● ● ● ● ●●●● ●●●
●● ● ●●●● ●●● ●
●● ●● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●●● ●●●●●●● ●● ● ●●●● ●●● ● ●●● ● ●●●●●●●● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ●●●●●●● ● ●●●●●● ●● ●●●● ●● ●
●●●● ● ●●●●●●● ● ● ●●●●● ●●●● ● ●●● ● ●●●●●●●
●●●●● ● ● ● ●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●●
●● ●●●● ●●● ●● ● ●●●●
●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●●●●● ● ●●●●●●
●
●●● ●●●● ●●
●
●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ●● ●●
●●●
●●●●●● ● ● ●●●●● ●●● ●●●● ●
●●●●●● ●●●●● ● ● ●●
●●
●
●●
●
● ● ●●●● ●●● ● ●● ●● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●●●
●●●●●● ●
●● ● ●●● ● ●●● ●
●●● ● ●●●●●●●
● ●●●● ● ●●●●●●● ● ● ●●●● ● ●
●●●●● ● ●●●●●●● ● ●●●●● ●●●●●●● ●● ● ●●●● ● ●●●●●●● ● ●●●●●●●●●● ● ●●● ● ●●●●●●● ●●●●● ● ● ● ●●●●
● ●●● ●●●●● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●● ● ●●●
●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●
●●●● ● ●●●●●
●
●
●●● ●●●● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●●●● ●●
●● ● ●●●●●● ● ●●●●●● ●●● ● ●●● ● ●●●●●● ●●●●●● ● ●●●●●
●● ●● ● ●●●● ●●● ●
●● ● ● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●● ● ●●●●●● ●
●●● ●●● ● ●●● ●
●●● ● ●●●●●●●
● ●●●● ● ●●
●●●●● ●● ●●●● ● ● ●●●●● ● ●● ●●●●● ● ●●●●● ●●●●●●● ●● ●●●●● ● ●●●●●●● ● ●●●●●●●●●●● ● ●● ● ●●●●●
●● ●●●● ●● ● ● ●●●● ● ●●● ●●●● ● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●● ● ●●●●●●●●● ●●●●●●● ● ●●● ●●● ● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●●●● ● ●●● ● ●●●●●
●
● ●●●●●●
●●● ●●●● ● ●●●●●● ●●●●●
●● ●●●●●●
● ● ● ● ●●●● ●●● ●●
● ● ● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●●
● ●●● ●●●●● ●●●●●
● ●●●●●● ●●● ● ●●●
● ●●● ● ●●● ● ●●●●● ●●
● ●●●● ● ●●●●●● ● ●● ●●●● ● ● ●●●●● ● ●● ●●●●● ● ●●●●● ●●●●●●● ●● ●●●●● ● ●●●●●●● ● ●●●●● ●●●●●● ● ●● ● ●●●●●●● ●●●● ● ● ● ● ●●● ● ● ●●● ●●●● ● ● ●●● ●● ● ● ●●●●●● ●●● ● ●● ●●● ●●●●●● ●●●●● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ● ●●●● ●●● ● ●
●● ● ●●●●●●●●● ●●●●●●● ● ●●● ●●● ● ●●● ●●● ● ● ● ●●●● ● ●●●●●● ● ●●●●●● ●●●●●
●● ● ●●●●●
●● ●●●●●● ●●● ● ●●● ●
●●●●●●
●●●●●●● ●●●●●●●
●
● ● ●●●●
●●● ●●
● ●● ●● ●●●●●●●● ●●● ● ● ●●●●
●● ●● ●●●● ● ●●● ●●● ●● ●●●●
●● ●●●●●● ●●● ● ●●
●
● ●●●
●●●● ● ●●●●● ●●● ●●●● ● ●●●●●● ●●● ●●●● ● ● ●●●●● ● ●● ●●●● ● ● ●●●●● ●●●●●●● ●● ●●●●● ●
●●●
●●● ● ● ●●●●● ●●●●●● ● ●● ● ●●●● ●●● ●●●● ● ● ● ● ●●● ● ● ●●●●●●● ● ● ●●● ●● ● ● ●●●●● ●
●●
●
●
●● ●● ●
●
●●●●
● ●●●●● ●●●● ●●●
●● ● ●●● ●
●●●●●●● ●
●
●●
●
● ● ●●●● ●●●
●●
●● ●●●●
●●●●●●●●●●●●● ● ●●● ●●● ● ●●●●●●
● ● ●●
●●●
● ●●●●●●
● ●●●●●● ●●● ●●●
●
● ●●●●●
●
● ●●●●●●
●●● ● ●●● ●
●●●●●●
●●●●●
●●●●●●●●● ● ● ● ●●●
●●●● ● ●
●●● ●● ●●●● ●●●● ●●● ● ● ●●●●●
● ●● ●●●●
● ●●● ●●● ●● ●●●●● ● ●●●●●● ●●● ● ●●● ● ●●● ●●●●
●●
●●●●●●● ●●●●
● ●●●●●● ●●●
●●
●●
●● ●
●●●●● ●● ●●●● ●
●
●
●●
●●●●●
●●●● ●
●
●● ●●●
●●
●●●
●● ●●
●
●●●● ●
●●●●● ● ●● ●
●●
●● ●● ● ●●●● ● ● ● ● ●●● ● ● ●●●●●●
● ● ● ●●● ●● ● ●●
●●●●
●
●●●
●
●● ●● ●
●
●●●●
● ●●●●● ●●●● ●●●●● ● ●●● ●
●
●●●●●●●
●
●●●●● ●●●● ●●●● ●●● ●●●
●●●●●●●
●●●●●●● ● ●●● ●●● ● ●●●●●● ● ● ●
●
●●●
● ●●●●●● ● ●●●●●● ●●● ●●●
●
●●●●●●●●●●●●
●●●●● ● ●●● ●
●●●●●● ●●●●●●● ●●●●●●● ● ● ● ●●●●●●● ● ●●● ● ●●
●●●● ●●●●●●● ● ● ●●●●●● ●● ●●●
●● ●●● ●●● ●● ●●●●●
● ●●●●●●
●●● ● ●●●● ●●● ●●●● ● ●●●●
● ●●● ●●●●● ●●●●●● ●●● ●●
●●●
●
●●●●●● ●
● ●●●
● ●● ●●
●
●●●
●●●●●● ●●
●● ●●● ● ●●●
●
●● ● ●
●
●●●● ●●●●●● ● ●● ●
●
●
●● ●● ● ●●●● ● ● ● ●●
●● ●● ●●●●●●● ● ● ●●● ●● ● ● ●●●●● ●●●● ●●● ●● ●
●●●●
●● ●●●●● ●●● ● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●●●● ●●●● ●●●
●●●● ●●● ●●●●●●● ●●●●●●● ● ●●● ●●● ● ●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●●● ●●●
●●●●
●●●●●●
●●●●● ●●● ● ●●● ●
●
●●●●● ●●● ●●●● ●●●● ●●● ● ● ● ●●●●●
● ● ● ●● ● ● ●●●●●● ●●●●●●● ● ●
●
●●●● ●●
● ●●●● ●●●● ●●● ●● ●●●●● ● ●●●●●● ●●● ● ●●● ● ●●● ●●●●● ●●●●● ●●● ●●●● ● ●
●●●●● ●● ● ●●●● ● ● ●●●●●● ●● ●●●
● ● ●●●
●
●● ●●●●●●●●
●
●● ●
●● ●
●●●●
●●
● ● ●●●●● ●●●●● ● ● ●● ● ●●●● ●● ● ●●
●● ● ● ● ●●●● ●● ● ●●●●●● ●● ●●● ●● ● ● ●●●●● ●●●●●
●● ●● ● ●●●●
●● ●●●●● ●●● ● ●●● ●● ●●●● ● ●●●●●●● ● ●●● ●●● ●●●● ●●●
●
●●● ●●●●
●●●●●●●●●●●●● ● ●●● ●●● ●●●●●●● ● ●
●
●
●●● ●●●●●●●
●
●●●●●● ●
●
● ●●●
●●●●●●●● ● ●●●●●● ●● ● ● ●●● ● ●●●●●● ●●● ● ●●● ●●●● ●●● ● ● ● ●
●●● ●● ● ● ●● ● ● ●●
●●●
● ●●●●●●● ●
●
●●●●● ● ●● ●●●● ●●● ●
●●● ●● ●●●●● ● ●●●●●● ●● ● ● ●●● ● ●
●● ●●●●●
●●●●● ●● ● ●●●● ●●
●●●●● ●● ● ●●●● ● ● ●●●●●
●
●● ●●●● ●●●●●
●●●●●●●●●
●●●● ●
●
● ●●●●●●●
● ● ●●●
●● ●●●●● ●● ●●
● ●●●● ●● ● ●●●● ● ● ●●●●● ●● ● ●●●● ●●
●●●●● ●● ● ● ●
●●●●
●
● ●●
●
●●●
● ● ●●
●
●●
● ●●●●● ●●● ● ●●● ●● ●●●● ● ●●● ●●●● ●●●● ●●● ●● ●● ●●●●●
●● ●●● ●●●●●
●●●●●●●●● ● ●●● ●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●●
●● ● ●●●●
●
●●●●●● ●●●●●●● ●● ● ●
●●● ●
●●●
●●● ●●● ● ●● ● ●●●● ●●● ● ● ●●
●●● ●● ● ● ●● ● ● ●●
●
●●●●●●●●● ● ●
●●
●●●●
●●● ●●●● ●●● ●
●●●
●
● ●●●●● ●● ●●●●● ●● ● ● ●●●●●
●● ●●● ● ●●●●●● ●● ● ●●●● ● ●●● ●●● ●● ● ●●●● ● ●
●
●
●●●●
●●●●●● ●●●● ●●●●●●●●●●
●
●
●
● ●●●
●●●●
●
●
●● ● ●
●
●
●● ●●●●● ●● ● ●●●●●● ●● ● ●●●● ● ● ●●●●● ●● ● ● ●●● ●●
●
●●●● ●● ● ● ●●●●● ●● ●● ●●● ●● ● ●●●
●
●
● ●●●●● ●●● ● ●●● ●● ●●●● ● ●●● ●●●● ●●●● ●●● ●● ●●●●●
●
●●● ●●● ●●●●●●●●
●●
●
●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ● ● ●●●●
●
●●●●●● ● ●●●●●● ●● ●●
●●● ●
●
●●●●● ●●● ● ●● ● ●●●● ●●● ● ●●
●●●● ●● ● ● ●● ● ● ●●
●●●
●●●
● ●●● ● ● ●
●
●●● ●●
●● ●●
●
●●●● ● ●
●● ●
● ●●●●
● ●●●●●●● ●● ● ● ●●●
● ●●●●
●● ●●●
● ●
●
● ●● ● ●●●● ● ●●● ●●● ●● ●
●
●●● ● ● ●●●
●● ●●●●●●● ●●●● ●●● ●●●●●●● ●●
●●●●● ●● ●●●●●
● ● ●●●●● ●●●●● ●● ●●●●●●● ●● ● ●●●● ● ● ●●●●● ●●● ● ●●● ●●
●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●●●● ● ●●● ●●● ● ● ●●● ●●● ●● ●●●●● ●●●● ●●● ●●●●●●●
●●●●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●● ●●●●
● ● ● ●●●●●
●●●●●●●
●
●●●● ● ●● ● ● ●● ● ●
●
●●
●●● ●● ● ● ●● ● ●●●● ●●● ● ● ● ●●●●●
● ● ● ●● ● ● ●●●●
●●
●●●
●●●
● ●
●
●
●●● ●
●●● ●●●● ●●● ●
●●● ●● ●●●●● ●● ●●●●● ● ● ● ● ●●●
● ●●● ●●●
● ●●
● ●
●
● ●● ● ●●
●●● ●●● ●●● ●● ●
●●●● ● ● ●●
●●●
●
●●●●●●●
●●●●
●● ●●●●●●● ●● ●●●●● ●●●
●●●● ● ●●●●●● ●●●●● ●● ● ●●●●●● ●● ● ●●●● ● ●●●●●● ●●● ● ●●● ●●●
●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●●●●● ●●● ●●● ● ● ●●●●
●● ●● ●●●●● ●●●● ●●● ●●●●●●● ● ●●●●●● ● ●●●●●● ●●●●●●●● ● ●
●●●● ●●●●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●●● ●●
●●●● ● ●● ●
●
●
● ● ●
●
●●●●● ●● ● ● ●● ● ●●●● ●●
● ● ● ●●●●●●
● ● ● ●● ● ●
●●
●
●
●●●
●● ●●●
●
●
●
●
●●● ●
●●● ●●●
● ●●● ● ●
●●●● ●●●●● ●
●●●●
●● ● ● ●●
●●● ● ●● ● ●●
● ●
●
●●●
●
● ●● ● ●●●● ● ●●● ●●● ● ● ●
●
●●● ● ●
●●●●●
●●●●●●● ●●●●
●●● ● ●●●●●● ●● ●●●●● ●● ●
●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ●●● ● ●● ● ●●●●●●● ●●● ● ●●● ●●
●
●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●● ●●● ●●● ●●● ● ● ●●● ●●● ●● ●● ●● ● ●●●●●●● ●●●●●●● ● ●● ●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●●
●●
●●●●●
●●●●●●● ● ● ●●●● ● ●●
● ●●● ● ●
●
●●●●● ●● ● ● ●● ● ●●● ●
●
●● ● ● ●●●●
●●● ●
●●
● ●● ●●
●
●
●●●
●● ●●● ●
● ●●
●●● ●
●
●● ●●●
●
●●● ● ●
●
●
●● ●●●●●
●
●
●●●
●● ● ● ●
●
●
●●
●●● ●
●
●
● ● ●
●● ●
●● ●● ● ●●●● ● ●●● ●●● ● ● ●●
●●● ● ● ●●
●●● ●●●●●●● ●●●● ●●●
● ● ●●●● ● ●●●●●●● ●● ●●●●●● ●●●●●● ●●●● ● ●● ●
●
●●●●● ●●● ● ●● ● ●●●●●●● ●●● ● ●●● ●●●●●●● ●● ● ●●●●● ● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●● ●●●●●● ●●● ● ● ●●● ●●● ●● ●● ●
● ● ●●● ●●●● ●●●●●●●● ●●
●
●●● ●●●●●●● ●●●●●●●
● ● ●●●● ●●●●●●●● ●●●●●●●
●● ● ●
●●● ●●●●●●● ●●●
●●● ● ●● ● ●●
● ●●
●
●●●●● ●● ● ● ●● ● ●●● ● ●●● ●
●
●●
●●
●
●● ●● ●
● ● ● ●
●
●●
●●●● ● ●
●●● ●
●
●●
●● ●●
●●
●●●●
●●● ●●
●
●
●● ●●●●●
●●●●●
●● ●●
●● ●●● ●
●● ●
●
●● ●
●
●
●
●
●● ●● ●●●●● ●●●● ●●
● ● ● ●●
●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ●
●●●● ● ●
●●●●●● ●● ●●●●● ● ●●●●●● ●●●● ● ●● ●
●●●●●● ●●● ●●● ● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●● ● ●● ●● ●●●●
● ● ●●●●
●● ● ●●● ● ●●● ●●●● ● ● ●● ●●●●●● ●●● ● ● ●●● ●●● ●● ●● ●● ● ●●● ●●●● ●●●●●●● ● ●●●
●●● ●●●●●●● ● ●●●●●● ● ●●●●● ●●
●●●●●● ●●●●●●●● ●
●●
●●● ●●●●●●●●
●●
●●
●
● ●● ●
● ●
● ● ●
●
●●●● ● ●● ● ● ●●
●●●● ●
●
●● ●
●●●
●●●●●
●●
●
●● ● ●
● ●
●
●●●
●●
●
●● ●
●
●
●●●●●
●
●
●●
●●●●
●●
● ●●
●●● ●●●●●
●
● ● ●●
●● ●
●●
●
●●●
● ●● ●
●
●●●
●●●
●
●● ● ●
●
●●●● ●●●● ●●
● ● ● ●
●●
●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●● ●●●●● ●●
●
●●●
● ● ●●●
●●● ●●●●
● ●● ●
●
●●●●● ●● ● ●●● ● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●●
●●
●
●●● ● ●● ●●
●
●●
●●
●
●●
●
● ●●●●●● ● ●●● ●●●● ● ●
●●
●●●●●● ●●● ● ● ●●●
●●● ●●
●● ●● ● ●●● ●●●● ●● ●●●●● ● ●●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●●●
●●●● ●●●●●●●●
●●
●●●
● ●●●●●●●●
●
●●●● ● ●● ●
● ●
● ● ●
●
●●●● ● ●● ● ● ●●
●
●●● ●
●
●● ●
●
●
●●●
●
●● ● ●
●
● ● ●●●
●
●
●
●
●
● ● ●●● ●
●
●
●●
●● ●
●
● ●
●
●●● ●
●
●●●
●● ●●●●●●●●
● ● ●●●● ● ●● ●●
●●
●
●● ●
●
●●
●●●
●●●● ● ● ●●●●● ●●●● ●●● ● ● ●
●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●●●●●●
●
●
●
●
●
●
●● ●
● ●●●
●● ●●●● ● ●● ●
●
●
●
●●● ●●●
●●● ●
● ●●●●●● ●●● ● ●●● ●● ●
●●●
● ●●● ●
●●●● ● ●● ●●●
●● ●●●●●
●● ●● ●●●● ● ●●● ●●●● ●● ●●● ●●●●●●●● ● ●●
●● ●
●
● ●●●●
●
●
●●
●●●●●● ●● ● ●●●● ● ●●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●●●●●●● ●●● ●●●●● ● ●
●●●● ●●●●●●● ● ●●●●● ● ●● ● ● ●
● ●●
●●●●● ● ●● ● ● ● ● ●●●● ● ●
●● ●
●
●
●●● ●●●
● ●●
● ●●●
● ●
●
●● ●● ● ●●●
● ●
●
●●●●●
●
●● ●●●● ●●● ●●●● ● ●●●●● ● ●● ● ●●●● ● ● ●
●●●●
●
●● ● ●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●●● ●
●●
●●●●●
●●●
●
●● ●
●●●
●●●
●
●
● ●●● ●
●
●
●
●●●● ●●●●●●● ●●●● ● ●● ●
●●
●
●●● ●●●
●●● ● ● ●●●●●●● ●● ● ●●●●● ●●●●●●●
● ●●●●● ● ●●●● ●● ●●
●
●●●●●●●●●●● ● ●●● ●●●● ●● ●●● ●●●●●●●● ● ●●●●● ●● ●●●●●●
● ●●●● ●●● ●
●
● ●●●
●● ●
●●
●●●●●●●●●● ●●●●●●● ● ● ●
●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ●● ●●●● ● ● ●● ● ● ●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ●
●●
●●
●
●● ● ●●● ● ●● ●●
●
●●●● ● ●●●● ●
●
●●●● ●
●
● ● ●●●● ●●● ●●●
● ● ●●●●● ● ●●●●●●● ● ● ● ●●●●●
●● ● ●●●
●●
●●●●● ● ●
●
●●●●
●
●●
●
●●● ● ● ●●●● ● ● ●●
●●●●
●
●●
●●●●
●●
●●
●
●● ●● ●●●● ●
●●● ●●● ● ●●
●
●●●● ●●●●●●● ●●●● ● ●
●● ●●●●●● ●● ● ●●● ● ● ●●●●●● ● ●● ● ●●●●● ●●●●●●● ● ●●●●● ● ●●●●
●● ●●●
●●●●●●●●●●● ● ●●● ●●●● ●● ●●●●●●●●●●● ● ●●●●● ●● ●●●●●●
●●
●●● ●●●
●●
●●●● ●●
●
●●
●● ●●
●●●●●● ●●●●●●● ●● ●●
●●● ●
●●●
●● ● ●●●●●●●● ●
●●
●●● ●●●●●● ●● ●●●● ● ●●● ●
●●● ●
●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ●●●
●● ●●● ● ●●● ● ●●
●●
●
●●●● ● ●●● ● ●
●
●●●● ●
●
● ● ●●●● ●●● ●●●
● ● ●●●●● ● ●
●
●●●●● ● ● ●●
●●●
●
●● ●●
●●
●
●
●
●●●● ● ● ●●●● ●
●
●●●●●● ● ● ●●●● ● ● ●●
●●●●●
●●●●● ● ●●●●●● ● ●● ●●●● ●●
●●●●● ●
●●●
●●●● ● ●●
●●●● ●●●● ● ●●
●●●●●●● ●●
● ●●● ● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●●●●
● ●●● ●●●●●●●●●●● ● ●●● ●●● ● ●● ●●● ●●●●●●●● ● ●●●● ● ●● ●●●●●●● ●
●●● ●●● ●●●●●● ●●
●●●
●● ● ●●●●●●● ●●●●●●● ● ●●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ●
● ●●●●● ●●● ●
●●● ●
●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ●
●
●●●●
●● ● ●●● ●●●
● ●
●
●●●● ● ●●● ● ●
●
●●●● ●
●
●●●●● ● ●●● ●●●
● ● ●●●●● ● ●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●
●
●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●●
●●
● ●●●●● ●
●
●●●
●● ● ● ●●●●● ●●●● ● ●● ● ●●
●●●● ●● ● ●●● ● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●●●●●●● ● ●●● ●●● ● ●● ●●● ●●●●●●●● ● ●●●● ● ●●●●●●●● ● ●●●● ●●● ●●●●●● ● ● ●●●●●● ●●● ●●●● ●●●●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●● ● ●●● ● ●●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●● ● ●● ● ● ●●●●● ● ●●● ●● ●
●●●● ●●
● ●●●● ● ●●● ●●●● ● ●●●●●● ●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ●
●● ●●●●● ●●●●●●● ● ● ●●●●● ●●●● ● ●● ● ● ●●●●● ●● ● ●●●● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●●
● ●●●●● ● ●●● ● ●● ●●● ●●●●●●●●●●● ●●●● ●●● ●●● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ●●
●●● ●●● ●●●●●●● ● ●●●●●● ●●● ●●●● ●●●●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●● ● ●●● ● ●●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●●● ● ● ● ● ●●●●●● ●●● ● ●●
●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●●● ● ●●●●●● ● ● ● ●●●● ●●● ● ●●● ● ●●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ●●●●●●● ● ●●●●●● ●●●●●● ●●● ● ●●●● ●●●●●●● ● ●●●●●● ●●●● ● ●●● ● ●●●●● ●●
●●●●● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●
99%
DA
RT
−100
0
100
200
300
400
−100
0
100
200
300
400
50 75 100 125 150Generation (GW)
LMP
($/
MW
h)
LMP versus Generation, PJM, 2012
Supply curves are approximately linear, and steeper in RT
Wenyuan Tang Virtual Bidding 25 / 41
Two-Settlement Market Model: Nomenclature
financial physical
demand
supply
DA LMP p(x)
v+ + x = v− + y
INC v+ DA gen x
DEC v− DA load y
RT LMP p(x)
x = y
RT gen x
RT load y
Observations from the market data
x ≈ x , y ≈ y , p′′ ≈ 0, p′′ ≈ 0, p′ > p′
Next we show how p depends on x
Wenyuan Tang Virtual Bidding 26 / 41
RT Supply Curve Conditioned on DA Generation
x
p
DA
x
p
RT
x
Inflexible Flexible
RT supply curve steeper: dispatchable generation
Intercept of RT supply curve depends on x
Wenyuan Tang Virtual Bidding 27 / 41
Modeling DA/RT Supply Curves
x , x
p, p
p(x) = ax
ηx
p(x) = ax + b
x
Inflexible Flexible
Inflexible generation uniformly distributed (proportion η)
Representing b and η in terms of a and a{p(ηx) = 0
p(x) = p(x)=⇒
{b = (a− a)x
η = 1− (a/a)
Wenyuan Tang Virtual Bidding 28 / 41
Generation Cost Depends on DA Generation
x x , x
p, p
p(x) = ax
ηx
p(x) = ax + b
x
Inflexible Flexible
Generation cost = + = DA gen cost + flexible gen cost
c(x) = η
∫ x
0azdz +
∫ x
ηx(az + b)dz
= η
∫ x
0azdz +
∫ x−ηx
0azdz
Wenyuan Tang Virtual Bidding 29 / 41
Cost Min. ⇐⇒ Dispatch Conv. ⇐⇒ Price Conv.
x
p > pif x < x
p = p if x = x
x , x
p, p
p(x) = ax
ηx
p(x) = ax + b
x
Inflexible Flexible
x > x : cost of generation dispatched in RT =
Generation cost = +
p > p
If we had x = x , cost would be minimized: savings = , p = p
Wenyuan Tang Virtual Bidding 30 / 41
Cost Min. ⇐⇒ Dispatch Conv. ⇐⇒ Price Conv.
x
p < p if x > x
p = pif x = x
x , x
p, p
p(x) = ax
ηx
p(x) = ax + b
x
Inflexible Flexible
x < x : cost reduction of generation descheduled in RT =
Generation cost = +
p < p
If we had x = x , cost would be minimized: savings = , p = p
Wenyuan Tang Virtual Bidding 31 / 41
Two-Settlement Market Model: Complete Specification
DA supply curve: p(x) = ax
RT supply curve: p(x) = ax + b
RT load L is fixed: x = L
a and a are fixed with a < a, and so η = 1− (a/a)
b is subject to an indepedent disturbance δ
b = b + δ = (a− a)x + δ
Empirical estimation
Yearp = ax p = ax + b
ηa (×10−3) R2 a (×10−3) b R2
2012 0.37 0.93 0.68 −27.5 0.30 45.4%
2013 0.42 0.91 0.76 −31.8 0.34 44.6%
2014 0.56 0.55 1.72 −108.2 0.18 67.3%
2015 0.41 0.78 0.86 −44.7 0.28 53.0%
Wenyuan Tang Virtual Bidding 32 / 41
Two-Settlement Market Model: Complete Specification
L0: market’s forecast about L, the RT load
δ0: market’s forecast about δ, the RT supply function disturbance
Market minimizes the forecast generation cost over x
minx
c(x) = η
∫ x
0azdz +
∫ L0
ηx(az + (a− a)x + δ0)dz
DA generationx0 = L0 + (δ0/a)
Spread measures the forecast accuracy
s0 = p0 − p0 = a(L0 − L) + (δ0 − δ)
Financial efficiency and economic efficiency are aligned
price convergence ⇐⇒ cost minimization
δ=0⇐⇒ dispatch convergence
Wenyuan Tang Virtual Bidding 33 / 41
Theory of Virtual Bidding
While virtual bids do not affect RT generation, they affect DAgeneration and therefore DA LMP, RT LMP and generation cost
We formulate a game with virtual bidders as strategic players, basedon the two-settlement market model
N: number of virtual bidders
Li : virtual bidder i ’s forecast about L
δi : virtual bidder i ’s forecast about δ
vi : virtual bidder i ’s quantity bid (INC if vi > 0, DEC if vi < 0)
DA generation: x = x0 −∑
i vi
DA LMP: p = ax
Virtual bidder i ’s forecast RT LMP: pi = aLi + b + δi
Virtual bidder i ’s forecast profit: πi = (p − pi )vi
Wenyuan Tang Virtual Bidding 34 / 41
Theory of Virtual Bidding
Solving the simultaneous FOCs
∂πi (vi , v−i )
∂vi= 0, i = 1, . . . ,N
yields ∑i
vi =N(aL0 + δ0)− (a
∑i Li +
∑δi )
(N + 1)a
Equilibrium spread measures the average forecast accuracy
s∗ =a(L0 − L) + (δ0 − δ) +
∑i (a(Li − L) + (δi − δ))
N + 1
=s0 +
∑i (a(Li − L) + (δi − δ))
N + 1
Wenyuan Tang Virtual Bidding 35 / 41
Theory of Virtual Bidding
Sufficient Condition of Price Convergence
If1
N
∣∣∣∣∣∑i
(a(Li − L) + (δi − δ))
∣∣∣∣∣ < a(L0 − L) + (δ0 − δ),
then|s∗| < |s0|
Cournot Theorem for Virtual Bidding
If1
N
∣∣∣∣∣∑i
(a(Li − L) + (δi − δ))
∣∣∣∣∣→ 0 as N →∞,
then|s∗| → 0 as N →∞
Wenyuan Tang Virtual Bidding 36 / 41
Theory of Virtual Bidding
Profitability and Price Convergence
Virtual bidder i makes a positive profit if and only if its participationdrives the spread toward zero:
s∗vi > 0 ⇐⇒
{0 < s∗ < s∗−i , s∗−i > 0
s∗−i < s∗ < 0, s∗−i < 0,
where s∗−i is the equilibrium spread without the participation of i
Screening out unqualified virtual bidders with poor forecast accuracy
Introducing more qualified virtual bidders into the market
The virtual bidding mechanism is self-incentivizing: a virtual biddercan make more profit by improving its forecast accuracy, which is alsofavorable to the market
Wenyuan Tang Virtual Bidding 37 / 41
Profit and Cost Savings of Virtual Bids
Market’s DA generation: x0 = L0 + (δ0/a)
Optimal DA generation: x∗ = L + (δ/a)
Net INC of the virtual bids: v
v∗ = x0 − x∗ minimizes the generation cost and induces zero spread
Profit of the virtual bids
f (v) = −av2 + av∗v
Cost savings of the virtual bids
g(v) = −(aη/2)v2 + aηv∗v
Wenyuan Tang Virtual Bidding 38 / 41
Profit and Cost Savings of Virtual Bids
v
f(v)
g(v)v∗
2v∗
η = 0.25
v
f(v)
g(v)
v∗
2v∗
η = 0.5
v
f(v)g(v)
v∗
2v∗
η = 0.75
profit > cost savings profit < cost savings
The generation cost savings may not recover the profit of the virtualbids when v is small
As η increases, the cost savings are more likely to recover the profit
The more competitive the virtual bidders, the more likely that thecost savings recover the profit
Wenyuan Tang Virtual Bidding 39 / 41
Effectiveness of Virtual Bidding: Empirical Estimation
●
●
●
●
−5.0
−2.5
0.0
2.5
5.0
2012 2013 2014 2015Year
Spr
ead
($/M
Wh)
Virtual Bids● No (Estimate)
Yes (Market Data)
Mean of Spread with and without Virtual Bids, PJM
YearCost (Billions) Savings (pct. Profit (pct.
No VB VB of No VB Cost) of Savings)
2012 22.14 21.71 1.95% 8.03%
2013 24.03 23.66 1.53% 2.86%
2014 29.58 28.48 3.73% 1.59%
2015 20.29 20.24 0.23% 60.21%
Wenyuan Tang Virtual Bidding 40 / 41
Conclusion
A successful fusion of data and theory: market data → market model→ theory and implications → methodology of estimation → empiricalevidence by market data
We propose a two-settlement market model which explains variousphenomena in the market, and provides a methodology of estimatingthe generation cost
The proposed model aligns financial efficiency and economicefficiency, which serves as the basis of the theory of virtual bidding
Virtual bidding has improved the market efficiency: (i) comparativeanalysis before and after virtual bidding in CAISO; (ii) estimation ofcost savings and price convergence by virtual bids in PJM
There still exist substantial profitable opportunities: (i) market virtualbidders make profits; (ii) simple machine learning algorithms can beprofitable on top of the market virtual bids
Wenyuan Tang Virtual Bidding 41 / 41