All of Virtual Bidding: A Data-Driven Approach

All of Virtual Bidding: A Data-Driven Approach

Wenyuan Tang1,2

joint work withRam Rajagopal2 Kameshwar Poolla1 Pravin Varaiya1

1University of California, Berkeley

2Stanford University

November 1, 2016

Wenyuan Tang Virtual Bidding 1 / 41

Two-Settlement Wholesale Electricity Market

Locational marginal prices (LMPs) reflect the value (price) of powerat different locations, and the LMP at a load-zone or a hub is theweighted average of the nodal LMPs

The day-ahead (DA) market lets market participants commit to buyor sell power one day before the operating day, and establishes 24hourly DA LMPs

The real-time (RT) market balances the differences between DAcommitments and the actual demand and supply during the course ofthe operating day, and establishes the 5-minute RT LMPs

Systematic nonzero spreads are routinely observed, which indicatessome market inefficiency

hourly spread = hourly DA LMP− hourly average RT LMP


Virtual Bidding

Virtual bids are included in DA dispatch, settled at DA LMPs,liquidated at RT LMPs

Allows participants to take financial positions in DA withoutdelivering or consuming physical power in RT

Hedging tools for physical entities; arbitrage tools for financialentities; adding liquidity; mitigating market power

Goals: enhancing market efficiency through DA/RT price convergence(financial efficiency) and dispatch convergence (economic efficiency)

Virtual supply (INC): generation bid in DA to be closed in RT

Virtual demand (DEC): demand bid in DA to be closed in RT

DA RT

spread INC

DA RT

spread =INC profit


Outline

Part I: Exploratory Data Analysis

Part II: Virtual Bidding and Financial Efficiency

Part III: Virtual Bidding and Economic Efficiency

DataAnalytics

Micro-economics

GameTheory

EmpiricalAnalysis

Virtual Bidding

Theory

Two-Settlement Market

Model

FinancialEfficiency

EconomicEfficiency


Part I: Exploratory Data Analysis

CAISO

Independent System Operator (ISO)

DA market: Apr 2009

Virtual bidding: Feb 2011

Data (NP15, 2010/2012): DA/RT LMP

Peak load ≈ 50 GW

PJM

Regional transmission organization (RTO)

DA market and virtual bidding: Jun 2000

Data (RTO, 2012–2015): DA/RT LMP,DA/RT/forecast load, INC/DEC

Peak load ≈ 150 GW


DA and RT LMP

●● ● ● ●

●

●● ● ● ● ● ● ● ●

●●

●

●● ●

●

●●

● ● ● ● ● ●

● ●

● ● ● ●●

●

●

●●

●

●

●●

● ● ●

Mean

Standard Deviation

0

10

20

30

40

50

0

10

20

30

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

LMP

($/

MW

h)

Market● DA

RT

Statistics of DA and RT LMP, PJM, 2012

spread = DA LMP− RT LMP ≈ 0

RT LMP is more volatile than DA LMP


Spread

−200

−100

0

100

200

Mar Jun Sep Dec

Spr

ead

($/M

Wh)

Hour

4

18

Hourly Spread Time Series, PJM, 2012

0.00

0.05

0.10

−200 −100 0 100 200Spread ($/MWh)

Den

sity Hour

4

18

Hourly Spread Histogram, PJM, 2012

The distribution of the spread is heavy-tailed and left-skewed


Spread

−500

−250

0

250

Mar Jun Sep Dec

Spr

ead

($/M

Wh)

Hour

4

18

Hourly Spread Time Series, PJM, 2014

The polar vortex triggered two extreme weather events in Jan 2014

Recent data do not support classical models, e.g., [Bessembinder &Lemmon 2002], which states that spread is negatively related toVar(RT LMP), and positively related to Skew(RT LMP)


Virtual Bids

●

●

●●●

●

●●

●

●

●●●●

●

●

●

●●

●

●●●●●●

●

●

●

●●●●

●●●

●

●●●●●

●●

●●

●

●●●

●

●●●●

●●

●●●●

●●

●●

●

●

●

●

●●

●●

●

●●●●●●

●

●●●

●●

●

●

●

●

●●●

●●●

●●

●●

●

●

●●●●●

●●

●●●●

●●●

●

●●

●●

●

●●●●

●●●●●●

●

●

●●

●

●

●●●

●●

●

●

●●

●●●●●●

●●

●●●●●

●●●●

●●●

●●

●

●

●

●●

●●●●●

●

●●

●● ●●●●

●●●●

●●●●

●

●●●

●●●

●●

●

●●●

●

●●●

●

●●

●

●

●●

●

●

●●

●●●●

●●

●●

●●●●●●●●●

●●

●●●●

●●●

●●

●●●●●●●

●

●●●●●

●

●

●

●●●●●

●●●●●●

●●●

●●●●●

●

●●●●●●

●

●

●●

●

●●

●●

●●●●

●

●

●●●

●●●●●

●

●●

●●●●●●●●

●●

●

●●●

●

●●

●●

●●

●●●

●

●

●

●

●

●

●●●●●● ●●●●

●●

●●●

●●●●

●

●

●

●

●

●●

●●

●

●●

●●

●

●

●

●●●

●●

●●

●

●

●●

●

●

●●●●

●●

●●●●

●●●●●●

●●

●●

●

●●●

●●

●

●●●

●

●●

●

●●●●

●

●●●

●

●●

●●

●●

●●●

●●●●

●

●●●●

●

●●

●●●

●●

●●

●

●●●●

●●●●

●

●●●

●

●●

●●●●●

●

●

●

●●●

●

●●●

●

●●●●●●●

●

●

●

●●

●

●

● ●●●●●● ●

●●

●

●●

●●●

●●●

●

●

●

●

●

●● ●

●●

●

●

●●

●

●●

●

●●●●●●

●

●●

●

●

●●

●●

●

●

●

●●

●●●●

●●

●

●●●

●

●

●

●●

●

●

●●●●●●●●●

●

●

●

●●

●●

●●

●●

●●●●●●●●●●●●●

●●●●

●

●

●●

●

●●

●

●

●

●

●●●

●

●●

●●

●●

●●

●●

●

●●●●●

●

●●

●●●●

●

●

●●●●

●

●●

●●

●●●●

●●

●

●

●

●

●●

●●●●●

●●●●

●●●●

●●

●

●●●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●●●

●

●

●

●

●●

●●

●●

●

●

●●

●●●●

●

●

●●

●

●

●●●

●

●

●

●●●●

●●

●●●●●●●●

●

●

●●

●●

●●●●●

●●●●●

●●●●●●

●

●

●●●

●

●●

●●●●●

●

●

●●

●●●●●

●●●

●●

●●●●

●

●●●

●

●

●●

●

●●

●●

●●

●

●

●●●●●

●

●●●

●

●

●●

●

●●

●

●●●●●

●●●

●

●●

●●

●●●●●●

●●

●

●

●●

●●●

●●●

●

●

●●

●●●

●

●●

●

●●

●

●●●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●●

●●●

●●

●

●

●

●

●

●●

●●

●●●●

●●

●●●●●●

●

●

●●●●●●

●●

●●

●

●●●

●●●●

●

●●

●●●

●●

●●●●●

●

●

●

●

●

●●●●

●●

●●

●

●

●

●

●

●

● ●

●

●●●●

●●●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●

●

●

●

●

●●

●

●

●

●

●

●

● ●●●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●●●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●●●●

●

●

●●

●

●

●

●●

●●●

●

●●●

●

●●

●

●

●

●●●●

●●●

●

●●

●

●

●

●

●

●●●

●

●

●●●

●

●

●

●●

●

●

●●

● ●

●

●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●●●

●●●●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●●

●●

●●●

●

●

●●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●●

●

●

●

● ●

●

●

●

●

●

●

●●●

●

●

●

●

●●●●

●

●

●

●●●

●●

●

●●●

●

●

●

●●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●●●●

●

●●

●●

●●

●

●

●

●

●●

●

●

●

●●●●●●

●

●

●●●

●●●

●

●●●●

●●

●●●

●●●

●●

●

●

●

●●●●

●●

●

●●

●

●●●●

●

●●

●

●

●

●

●

●

●●

●

●

●●●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●● ●

●●

●

●●

●

●

●

●

●●

●●

●

●

●

●●

●

● ●

●

●

●●●

●●

●

●

●●

●●● ●

●●

●●

●

●

●

●

●

●●

●

●●

● ●●

●●

●●

●

●

●

●●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●●

●●

●●

●

●● ●

●

●●

● ●●

●

●

●

●

●

●

●

●●●●

●

●●●

●

●

●●

●

●

●

● ●●●●

●

●●

●●

●

●

●●

●

●

●

●

●

●●

●

●●

●●●

●●

●

●

●●

●●

●

●

●

●

●

●●●

●●

●●

●●●

●●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●●●●

●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●●

●

●

●

●●●

●

●

● ●●

●

●

●●●●● ●●●

●

●●

●

●

●

●

●

●●●●●

●

●●●

●●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●●

●

●

●

●●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●●●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●●●●●

●

●

●●

●

●

●

●

●●

●●●●●●●●

●

●

●●●●

●●

●●●●

●

●

●

●●

●

●

●

●

●

●●●●

●

●●

●

●●

●

●

●

●

●

●

●

●●

● ●

●●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●●

●●

●

●●

●

●

●●

●

●

●

●●

●

●●

●●

●

●

●●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●●●●

● ●

●

●●●

●●

●

●●

●

●

●

●

●●●●●●

●

●●●

●●●●●

●

●●

●●

●●●●

●●●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●● ●

●

●●

●

●●●●

●

●

●

●

●●●

●●

● ●

●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●●

●

●

●

●

●●●●●●

● ●

●

●

●

●●

●

●●●

●●

●●●●●

●●

●●●

●●

●

●●●●

●●●

●●●

●

●

●●

●

●●

●●●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●●

●

●

●●

●

●●

●●

●●

●●

●

●●

●

●

●●●●●

●

●●●

●

●

●●●

●●●

●

●●

●●●

●

● ●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●●

●

●

●●

●●●

● ●

●

●

●

●

●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●●

●

●●●

●

●

●●●●●

●●

●

●●

●●●●●●

●●●●

●●

●●●

●●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●●

●

●●

●●

●

●●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●●●

●●●●●

●

●

●

●●

●

●

●

●

●

●

● ●

●●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●●●

●●●●

●●

●

●●●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●●●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●● ●●

●

●

●

●

●

●●

●

●●●●●

●

●●●

●

●

●

●

●

●

●●

●

●

● ●●

●

●●

●●

●●

●

●

●

●●●●●●

●●●●

●●

●●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●●●●

●

●

●

●●

●

●

●

● ●

●

●●

● ●●

● ●

●

●

●

●

●●●

●

●

●

●●

●

● ●●

● ●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●●

●●

●●

●●

●

●

●●●

●●

●

●

●●

●

●

●●●

●

●

●

●●

● ●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●●●

●●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●●

●

●●

●

● ●●●

● ●●

●●

●

●●

●● ●●

●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

● ●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●●●●●●

●

●●

●

●●

●●●

●●●

●●

●

●●

●

●

●●

●

●●

●●

●

●●●

●●●

●

●

●

●

●●

●

●

●

●●● ●

●

●

●

●●●

●●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●●

●

●●

●●

●

●●

●

●● ●

●

● ●

●●

●

●

● ● ●

●

●●●●●

●

●

●

●

●

●

●●

●

●

●

●●●●

●●

●

●●●

●

●

●

●●

●●

●

●

●

●

●

●●●

●

●

●

●●●●

●

●

●●

●●

●

●

●

●●

●

●●●

●●

●●●

●

●●

●●

●●● ●

●●

●

●

●●

●

●

●●

●●●●●

●●

●●

●

●

●●

●

●

●

●

●●

●

●● ●

●

●

●

●●

●

●

●●●

●

●●

●

●

●

●●

●

●

●

●

●●

●

●●●

●

●

●

●●●

●

●

●●

●●●●

●●

●

●

●

●● ●●●●

●

●

●

●

●●

●●●

●●

●●

●

●

●●●●

●

●

●●

●

●

●●●

●●●

●

●●●●●●

●

●

●

●●●

●

●●

●

●●●●

●

●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●●●

●

●●●●●

●●

● ●

●

●

●●

●

●

●●●●●●●

●

●●●●

●●●

●●

●●●

●

●

●●●

●

●●

●●●●●●●

●●

●●

●

●●

●●●

●

●●

●

●

●●

●●●●

●

●●

●●●●

●●●

●

●

●●●●●

●●

●●

●●

●●

●● ●●●●●

●●

●●●

●●●●●

●●

●

●●●

●

●●

●●●●●●

●

●

●●●

●●●

●

●

●

●●

●

●●●●●

−15000

−10000

−5000

0

5000

−400 −200 0 200Spread ($/MWh)

Net

INC

Vol

ume

(MW

)

● Profit

Loss

Hourly Net INC Volume versus Spread, PJM, 2012

profit = spread× (INC− DEC) = spread× net INC

Virtual bidders expect negative spreads, or RT LMP spikes


Virtual Bids

2012 2013

2014 2015

($20)

$0

$20

$40

($20)

$0

$20

$40

Mar Jun Sep Dec Mar Jun Sep Dec

Mar Jun Sep Dec Mar Jun Sep Dec

Cum

ulat

ive

Pro

fit (

Mill

ions

)

Hours

All

Normal

Abnormal

Profit of Virtual Bids, PJM

Normal hours (98% of all): spread between 1st and 99th percentile

Abnormal hours (2% of all): otherwise


Performance Metric: Sharpe Ratio

N: number of days, R: daily profit

Sharpe ratio =√N

E[R]√Var(R)

Four-year (2012–2015) Sharpe ratio of S&P 500 is 1.68

Sharpe ratios of the PJM virtual bids

Year All Hours Normal Hours Abnormal Hours

2012 1.80 0.40 1.99

2013 0.96 −2.09 2.77

2014 0.39 −1.42 0.93

2015 4.31 1.60 4.82

Total 1.79 −1.29 2.56

Virtual bidders speculate on extreme events



We define financial efficiency as DA/RT price convergence

DA LMP = E[RT LMP|DA LMP]

orE[spread|DA LMP] = 0

which implies

ρ(spread,DA LMP) = 0 and E[spread] = 0

It is not clear that virtual bidding improves the financial efficiency

●

● ●

●

●

●

●

● ●●

● ●

●

●

●

● ● ●●

● ● ●

●

●

−0.2

−0.1

0.0

0.1

0.2

0.3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

Cor

rela

tion Year

● 2010

2012

Correlation between Spread and DA LMP, CAISO



●●

●

●

● ●

●

●

● ●

● ●

●●

●

●●

●●

● ●

●

●

●

●

●

●

● ● ●

● ● ●

●

●●

●●

●

● ●

●●

● ●

●

●●

● ●●

● ● ●● ●

● ●●

●● ●

●

● ●

● ● ●●

●

●●

Mean

Standard Deviation

Mean Absolute Value

−10

−5

0

5

10

0

20

40

60

80

0

10

20

30

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

Spr

ead

($/M

Wh)

Year● 2010

2012

Statistics of Spread, CAISO

Mean is closer to zero after the implementation of virtual bidding

But standard deviation and mean absolute value remain high



Alternative approach: testing whether profitable bidding strategiesexist before and after the implementation of virtual bidding [Li,Svoboda & Oren 2015], [Jha & Wolak 2015]

We propose a measure that tests the randomness of the sequence ofthe spread: more random spread leaves less room for arbitrageopportunities

We examine the autocorrelation of the sequence of the spread andpropose a benchmark bidding strategy that is only based on theup-to-date price information

We employ machine learning methods to design more sophisticatedbidding strategies that utilize other data such as load

Note that our definition of financial efficiency does not depend ontransaction costs, which are therefore not considered


Wald-Wolfowitz Runs Test on sgn(spread)

●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●

●●●●●●●●●●●●●●●

●●●●●●●●●

●●●●●●

●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●

●●

−400

−300

−200

−100

0

100

Mar Jun Sep Dec

Spr

ead

($/M

Wh)

Spread● > 0

< 0

Runs Test on Sign of Spread, CAISO, 2010, Hour 18

A run is a segment of consecutive “+”s or “−”s

N = 365 = N+ + N− = 268 + 97. Under the null hypothesis that thesequence is i.i.d.

µ = (2N+N−/N) + 1 = 143, σ = (µ− 1)(µ− 2)/(N − 1) = 7.4

110 runs, few enough to reject the null hypothesis (at α = 0.05)


Runs Test Results (p-values ≤ 0.05 in bold)

Hour 2010 2012 Hour 2010 2012

1 0.016 0.331 13 0.000 0.074

2 0.026 0.028 14 0.000 0.123

3 0.161 0.020 15 0.002 0.002

4 0.002 0.625 16 0.000 0.002

5 0.063 0.352 17 0.002 0.002

6 0.050 0.998 18 0.000 0.001

7 0.000 0.561 19 0.000 0.033

8 0.021 0.730 20 0.000 0.000

9 0.016 0.501 21 0.006 0.323

10 0.524 0.066 22 0.000 0.000

11 0.000 0.001 23 0.000 0.005

12 0.000 0.001 24 0.002 0.703

The spread is more random after virtual bidding


Exploring Intertemporal Correlation

95% CI

95% CI

2010

2012

−0.1

0.0

0.1

0.2

0.3

−0.1

0.0

0.1

0.2

0.3

0 5 10 15 20 25Lag (Day)

PAC

F

Partial Autocorrelation of Daily Average Spread, CAISO

The spread is more random after virtual bidding

The strong lag-1 autocorrelation motivates the benchmark strategy


Lag-1.5 Algorithm

sgn(∑

spread) sgn(∑

spread)forecast

deadline

a.m. p.m. a.m. p.m. a.m. p.m.

day t − 2 day t − 1 day t

Bids for day t should be submitted by noon on day t − 1

Lag-1.5 forecast (sht : spread at hour h on day t)

sgn

(24∑

h=13

sht−2 +12∑h=1

sht−1

)→ sgn

(24∑h=1

sht

)

If “+”, trade 1 MW INC for each hour

If “−”, trade 1 MW DEC for each hour


Lag-1.5 Algorithm

2010

2012

($5,000)

$0

$5,000

$10,000

($5,000)

$0

$5,000

$10,000

Mar Jun Sep Dec

Mar Jun Sep Dec

Cum

ulat

ive

Pro

fit

Hours

All

Normal

Abnormal

Profit of the Lag−1.5 Algorithm, CAISO

Pre-VB: average profit $1.35/MWh, Sharpe ratio 1.93

Post-VB: average profit $1.08/MWh, Sharpe ratio 1.35


Support Vector Machine

Input variables: {spread, DA load, RT load, forecast load} of dayt − 7, t − 6, . . . , t − 2

Output variable: sign of the daily spread of day t

Training data: 365 samples in year y

Test data: 365 samples in year y + 1

Sharpe ratios of the PJM virtual bids and SVM

YearPJM SVM

All Normal Abnormal All Normal Abnormal

2013 0.96 −2.09 2.77 1.66 4.72 −1.26

2014 0.39 −1.42 0.93 1.62 3.19 0.98

2015 4.31 1.60 4.82 1.29 3.75 −0.81

Total 1.19 −1.60 1.97 2.17 6.05 0.63

There is still room for arbitrage opportunities: even a simple machinelearning algorithm works well


Support Vector Machine

2013 PJM

2014 PJM

2015 PJM

2013 SVM

2014 SVM

2015 SVM

($10,000,000)

$0

$10,000,000

$20,000,000

($20,000,000)

$0

$20,000,000

$40,000,000

$0

$10,000,000

$20,000,000

($2,500)

$0

$2,500

$5,000

$7,500

$0

$10,000

$20,000

$30,000

($3,000)

$0

$3,000

$6,000

$9,000

Mar Jun Sep Dec

Mar Jun Sep Dec

Mar Jun Sep Dec

Mar Jun Sep Dec

Mar Jun Sep Dec

Mar Jun Sep Dec

Cum

ulat

ive

Pro

fit

Hours

All

Normal

Abnormal

Profit Comparison between PJM Virtual Bids and SVM

SVM capture the patterns: profitable in normal hours

Actual virtual bidders speculate on the extreme events


Part III: Virtual Bidding on Economic Efficiency

We define economic efficiency as generation cost minimization

Why is load forecast important?

Why is DA load close to RT load?

Why is RT supply curve steeper than DA?

How is economic efficiency related to price convergence?

How is economic efficiency related to dispatch convergence?

How to explain the phenomena: negative RT LMP; DA load close toRT load, but DA LMP far apart from RT LMP; etc.

How does virtual bidding affect DA/RT dispatch?

How to estimate the generation cost with and without virtual bids?


DA/RT Load

●

●● ● ●

●

●

●

●

●

●● ● ● ● ●

●

● ● ● ●

●

●

●

●

●● ●

●

●

●

●

●

●● ● ● ● ● ●

●

● ● ● ●

●

●

●

●

●● ●

●

●

●

●

●

●● ● ● ● ● ●

●● ● ● ●

●

●

●

●

●● ●

●

●

●

●

●

●●

● ● ● ● ●●

● ● ● ●

●

●

●

2012 2013

2014 2015

70

80

90

100

70

80

90

100

0 6 12 18 24 0 6 12 18 24Hour

Load

(G

W)

Market● DA

RT

Mean of DA and RT Load, PJM

DA load is close to RT load


DA/RT Generation

●

●● ● ●

●

●

●

●

●

●● ● ● ● ●

●

● ● ● ●

●

●

●

●

●● ●

●

●

●

●●

●

● ● ● ● ● ●●

● ● ● ●

●

●

●

●

●● ●

●

●

●

●

●

●● ● ● ● ● ●

●

● ● ● ●

●

●

●

●●

● ●●

●

●

●

●

●

● ● ● ● ● ●●

● ● ● ●

●

●

●

2012 2013

2014 2015

70

80

90

100

70

80

90

100

0 6 12 18 24 0 6 12 18 24Hour

Gen

erat

ion

(GW

)

Market● DA

RT

Mean of DA and RT Generation, PJM

DA generation is close to RT generation

DA generation may not equal DA load due to virtual bids


DA/RT LMP and Generation

● ● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●● ● ● ●●●● ●●●● ● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●●● ●●●●●●● ●●● ●●●● ●● ●●●●● ● ●●●●● ● ●●●●●● ● ●●●●●●●● ● ●●●●● ●●●●●● ● ● ●●●●● ● ●● ● ● ●● ● ●●●●●● ●●●● ●●● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●●● ●●● ● ● ●●●●● ● ●● ● ●●● ● ● ●● ●●●●● ●●●●●● ●●●●●● ● ●●● ●●●● ●●● ● ●●● ● ●●●●●●●● ●●●● ● ●●● ●●● ●● ● ●●●●● ● ●●●●●● ●● ●●●●●● ●● ●●●● ●●● ●●● ●● ●●●●● ● ●● ●●●●● ● ●●●●●● ●●●● ● ●●● ●●●●●●●● ●●●●● ● ● ●●●●● ● ● ●● ●●●● ● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●● ● ● ●●●● ●●● ●● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●●● ●● ●●●●● ● ● ●●●● ● ●●●●●● ● ●●●● ●●●● ● ●●●●● ●●●●●● ● ● ●● ●●● ● ●● ● ● ●●● ●●● ●●● ●●●● ●●● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●●●●● ● ●●●●●●● ●● ● ● ● ●●●●● ● ●● ● ●●●● ●●● ●●●●● ●●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●●● ● ●●●●●● ●● ●●●● ● ●●●●●● ● ● ● ●●●●● ● ●●●●● ● ●●● ●●●● ● ●● ●●● ● ●●●●●● ●● ●●●●● ● ●● ●●●●● ● ●●●●●● ●●●● ● ●●● ●●●●●●● ● ●●●●● ● ● ●●●●● ● ● ●● ●●●● ●● ●●●●● ● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●● ● ● ●●●● ●●● ● ● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●●● ●● ●●●●● ●● ●●●● ● ●●●●●● ● ●●●● ●●●● ● ●●●●● ●●●●●● ● ● ●●●●● ●●●● ●●● ● ●●● ●●● ●●●● ●●● ●●●●●● ● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ● ●● ● ● ● ●●●●● ● ●● ● ●●● ● ●●● ●●●●● ● ●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●● ● ● ●●●●●● ●● ●●●● ● ●●●●●● ● ● ●●●●● ● ● ●●●●● ● ●● ● ●●●● ● ●● ●●● ●●●●●●● ●● ●●●●● ● ●● ●●●● ● ● ●●●●● ● ●●●● ● ●●● ●●●●●●● ● ●●●●● ● ● ●●●●●● ● ●● ●●●● ● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●● ● ●●● ● ● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●●● ●● ●●●●● ● ● ●●●● ● ●● ●●●● ● ●●●● ●●●● ● ●●●●● ●●●●●● ● ● ●● ●●● ●●● ● ●●●● ●●● ●●● ●●●● ●●● ●●●●●● ● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ● ●● ● ● ● ●●●●● ● ●● ● ●●● ● ●●● ●●●●● ● ●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●● ● ● ●●●●●● ●● ●●●● ● ●●●●●● ●●●●●●●● ● ●●●●● ● ●● ● ●●● ● ● ●● ●●● ●●●●●●● ●● ●●●●● ● ●● ●●●● ● ● ●●●●● ● ●●●● ● ● ●● ●●●●●●● ● ●●●●● ● ● ●●●●● ● ● ●● ●●●● ●● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●● ● ●●● ●● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●●●●●● ● ●●● ●●● ● ●● ●●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●● ●● ● ●●●● ● ●●●●●● ●●●●●●● ●●●● ●●● ● ●●● ●●● ●●●● ●●● ●●●●●● ● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ●●●●● ● ●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ● ●● ● ● ●●●●●● ●● ●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ● ●● ● ●●● ● ● ●● ●●● ●●●●●●● ●● ●●●●● ● ●● ●●●● ● ● ●●●●● ●●●●●● ● ●● ●●●●●●● ● ●●●● ● ● ● ●●●● ● ● ●●● ●●●● ●● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●● ● ●●● ● ● ●●● ● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ● ●●●● ●● ● ● ●●● ● ●●●●●●●●● ● ●● ●●●● ● ●●● ●●● ● ●● ●●●● ● ●● ●●●● ● ●●●●●● ● ●●●●●● ●●● ●●●● ● ●●●●●● ●● ●●●●● ●●●● ●●● ● ●●● ●●● ●●●● ●● ●●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●●● ● ●●●●●● ●●●● ● ● ●●●●●

● ●● ● ●●● ● ●●● ●●● ●● ●●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ●●●● ● ●●●●● ● ●● ●●●● ● ●●●●●● ●● ● ●●●● ● ● ●●●●● ● ●● ●●●● ● ● ●● ●●● ●●●●●●● ●● ● ●●●● ● ●● ●●●● ● ● ●●●●● ●●●●●● ● ●● ● ●●●● ●● ● ●●●● ● ● ● ●●●● ●● ●●●●●●● ●● ●●● ●● ●● ●

●●●● ● ● ●● ●●● ●● ● ●●●● ● ● ●●●●●

●●● ● ●●●●● ● ●●● ●

●●●●●●● ● ●●●●● ●

●●●● ●● ●● ●●

● ●●● ●●●●●●● ● ●●●●●● ●●

●● ●●● ● ●● ●●●● ● ● ● ●●●● ● ●● ●●●● ● ●●●●●● ●●● ●●●● ● ●●●●●● ●● ●●●●● ●● ●●●●● ● ●●● ●●● ●●●● ●● ●●●●●●● ●

● ● ● ●●●●●●● ● ●●● ●●● ● ●●● ●●● ●●●● ● ● ●●●●●

● ●● ● ●●●● ●●● ●●● ●● ●●●●● ● ●●●●●● ● ●● ● ●●● ● ●●● ●●● ● ● ● ●●●● ●●● ●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●●

● ●● ● ●●● ● ● ● ●●●●●●●● ●●●

●●●● ●

●●● ●● ●●

●● ● ● ●●●●● ●● ●● ●● ●

●● ●●●●● ●● ● ●●●● ● ● ●● ●●● ●● ●●●●●●● ● ● ●●● ●● ●●●

●●●● ●● ●● ●●● ●● ●● ●

●● ● ●●

●●●●●●●● ●●●

●●● ●●● ●●

●●●●●● ● ●●●●● ●●●●● ●● ●

● ●●

● ●●● ●●●●●●● ● ●●●●●● ●●

●● ●●● ● ●●●●●● ● ● ● ● ●●● ●●●●●●●

● ●●●●●● ●● ● ●●●● ● ●●●●●● ●● ●●●●● ●● ●● ●●●● ●●● ●●● ●●●● ●●● ●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●●● ●●●● ●●● ●●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ●●● ●●● ●●●● ●● ●●●●● ●●● ●●●● ● ●●●●●●● ● ● ●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●●● ●● ● ●●● ●● ● ●●●●

●●●●●●●●●

●●●●●

● ● ●●●●●

● ●●●●

●● ●●●● ●● ●

●● ● ●●●● ●●●●●●● ● ● ● ●●●● ●● ● ● ●●●●● ● ● ●●● ●● ●

● ●●●●● ●● ●● ●●● ●● ● ●●●● ● ● ●●●●● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●●●● ●●●● ●● ● ● ●●● ●●● ●●●●●●● ● ●●●●●● ● ●●● ●●● ● ●●●●●● ●● ●● ●●● ●●●●●●● ●●●●●●● ● ● ● ●●●● ● ●●●●●● ●● ●●●●● ●● ●●●●●● ●●● ●●● ●●●● ●●● ●●●● ●●● ● ● ● ●●●● ●●● ● ●●● ●●● ●●●● ●● ● ●●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ●●● ●●● ●●●● ●● ●●●●● ●●● ● ●●● ●●●● ●●● ● ● ●●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●●● ●● ●●●● ●●●●●●● ●●●●●●● ●●

●●●●●● ●●●●●● ● ● ●●●●● ●●●● ● ● ●

●● ●●●●● ●● ● ●●●● ● ● ●● ●●● ●●● ● ●●●●● ● ●●●● ●● ●●●●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●●● ●●● ●● ●●●● ● ●●●●●●● ● ●●● ●●● ●●●●●● ● ● ●●● ●●● ●●●●●●● ● ●●● ●●● ● ●●●●●● ● ●●●●●● ●● ●●●●● ●●●●●●● ●●●●●●● ● ● ●●●●● ● ●●●●●● ●● ●●●●● ●● ●● ●●●● ●●● ●●● ●●●● ●●● ●●●● ●●● ● ● ●

●●●● ●●● ● ●●● ●●● ●●●●●●● ●●●● ● ● ●●●●● ● ●● ●●●● ●●● ● ●●● ● ●● ●●●● ●●●●●●● ●● ● ● ●●● ● ●● ● ●●● ● ● ●●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ● ●●●● ● ●● ●●●● ●●●● ●●● ●●●●●●● ●●

●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ● ● ●●●●●●● ●● ● ●●●● ● ● ●●●●● ●●● ● ●●●●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ● ●●●●● ●●●● ●●● ●● ●●● ●● ●●●●●●● ● ●●● ●●● ●● ●●●●● ● ●●● ●●● ●●● ●●●● ● ●●● ●●● ● ●●●●●● ●●●●●●● ●● ●●●●● ●●●●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ●● ●●●●● ●● ● ● ●●●● ●●● ●●● ●●● ● ●●● ●●●● ●●● ● ● ●●

●●● ●● ● ● ●●● ●●● ●●●●●● ● ●●●● ● ● ●●●●●

● ●● ●●●● ●●● ● ●●● ● ●● ●●●● ●●● ●●●● ●● ● ● ●●● ● ●● ● ●●● ● ● ●●●●● ●● ●●●●● ● ●●● ●●● ●● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ●●●●●●● ●●●●●●● ● ● ●●●●● ● ●●●●●●

●●●● ● ●● ●●●●●●● ● ●● ●●●● ● ● ●●●●● ●●● ● ●●● ●● ● ● ●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ● ●●●●● ●●●● ●●● ●● ●●● ●● ●●●●●● ● ● ●●● ●●● ●● ●●●●● ● ●●● ●●● ●●● ●●●● ● ●●● ●●● ●●●●●●● ●●●●●●● ●● ●●●●●●●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●●●●● ●● ● ● ●●●● ●●● ●●● ●●● ● ●●● ●●●● ●●● ● ●

●●

●●● ●● ● ● ●●● ●●● ●●●●●● ● ●●●● ● ●●

●●● ●● ●●

●●●● ●●● ● ●●● ● ●● ●●●● ●●● ●●●● ●● ● ● ●●● ● ●● ● ●●● ● ● ● ●●●● ●● ● ●●●● ● ●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●● ●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ●● ●●●● ● ● ●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ●●●●●● ●●●● ●●● ●● ●●● ●● ●●● ●●● ● ● ●●● ●●● ●● ●●●●● ● ●●● ●●● ●●● ●●●● ● ●●● ●●● ●●●●●●● ●●●●●●● ●● ●●●● ●●●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●●● ●● ● ● ●●●● ●●● ●●● ●●● ● ●●● ●●●● ●●● ● ●

●

●

●●● ●● ●● ●●● ●●●

●●●●●● ● ●●●● ●●

●

●●● ●●●●

●●

●● ●●● ● ●●● ● ● ● ●●●● ●●● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●●● ●● ●●●●● ● ●●● ●●● ●● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ● ●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●● ●●●● ● ●● ●●●● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●●● ●●●● ● ●●●●●● ●●●● ●●● ●● ●●● ●● ●●● ●●● ● ● ●●● ●●● ●● ●●●●● ●●●● ●●● ●●● ●●●● ● ● ●● ●●● ●●●●●●● ●●●●●●● ●● ●●●● ●

●●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ●● ●● ●●● ●● ● ● ●● ●● ●●● ●●● ●●● ● ● ●● ●●●● ●●● ● ●

●

●

●●● ●● ●

●●

●● ●●●●

●●

●●● ● ●●●●●

●

●

●●● ●

●●●

●●

●● ●●● ●●

●● ● ● ● ●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ●●●●

●● ●● ●●●●● ● ●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ●●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ●● ●●●● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ● ●●●●●● ●●●● ●●● ●● ●●● ●● ●●● ●●● ●● ●●● ●●● ●● ●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●● ●●● ●●●●●●● ●●●●●●● ●● ●●●● ●●

●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●● ● ●● ● ● ● ● ● ● ●●● ●●● ●●● ● ●● ● ●●●● ●●● ● ●

●

●

●●● ●● ●

●●

●● ●●●

●●●

●●● ● ●●● ●●

●●

●●● ●

●●●

●

●●● ●●● ●

●

●● ● ●● ●●● ● ●● ● ●●

●● ● ● ● ● ●●● ● ●● ●●

●● ●●● ●●

●● ● ● ●●●●● ● ●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ● ●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ● ● ●●●● ● ●●●●● ● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ● ●●●●● ● ●●●●●●● ● ● ●●● ●● ●●● ●●● ●● ●●● ●●● ●● ●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●● ●●● ●●●●●●● ●●●●●●● ●●●●●● ●●

●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●● ● ●● ● ●● ●●● ●

●● ●●● ●●● ● ● ● ● ●●●● ●●● ●●

●

●

●●● ●● ●

●

●

●● ●●●

●●●

●●● ● ●●● ●

●

●●

●●● ●

●●●

●

●●●

●●● ●

●

●● ● ●● ●

●● ●●● ● ●●

●● ● ● ● ● ●●●● ●● ●

●●

● ●●●

●●

●● ● ● ●●●●● ● ●●● ●●

● ● ● ●●●●● ● ● ●●●●● ● ●● ●●●● ●●●●●●● ● ● ●●●●● ●● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●● ●●● ● ●●●●●● ● ● ● ●●● ● ● ●●●●● ● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●●● ●●●●● ● ●●●●●●● ● ● ●●● ●● ●●● ●●● ●● ●●● ●●● ●● ●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●● ●●● ●● ●●●●● ●●●●●●● ●● ●●●● ●●

●● ●●●● ●●● ●●●● ● ● ● ●●●● ● ●●●●●● ● ● ●● ●● ● ●● ● ● ● ●● ● ●●● ●●● ●●● ● ● ● ● ●●●● ●●●

●●

●

●

●●● ●●●

●

●

●● ●●●

●●●

●●● ● ●●

● ●

●

●●

●●● ●

●●●

●

●●● ●●●

●●

●● ●

●● ●●● ● ●● ● ●●

●● ● ● ● ● ●●●

●●●●

●●

● ●●● ●

●

●● ● ● ●●●●● ●●●● ●●

● ● ● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●● ●●●● ● ● ● ●●●● ● ● ●●●●●● ●●●● ● ●● ● ●●●●●● ● ● ● ●●● ● ● ●●●●● ● ●●● ● ●●● ●● ●●●●● ●●●

●●●●● ● ●● ●●

●●●●● ●●●● ●● ●●●●●● ●●●●●● ● ● ● ●●● ● ●●●● ●●● ●●●●● ●●● ●● ●●●● ● ●●●● ●●● ●●● ●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ●●

●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ●●● ●●● ●●● ● ● ● ● ●●● ● ●●●●

●

●

●

●● ● ●●●

●●

●●●●●

●●●

●●● ● ●●● ●●

●

●

●●● ●●●●

●

●●● ●●●●●

●● ● ●● ●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●●●

●

●● ● ●● ●●

●● ● ● ●●●●● ●●●● ●●● ● ● ●●●●● ● ● ●●●●● ● ●●●●●● ●●●●●●● ● ● ●●●● ● ●● ●●●●●

● ● ●●●● ● ● ●●●●●●●●●

● ● ● ●●●●●●●● ●●

● ●●● ● ● ● ●●●●● ●●● ● ●●●●● ●●●●● ●●

●●

●●●● ● ●● ●●●● ●●● ●●●● ●● ●●●●● ● ●●●●●●● ● ● ●●●

●● ●●●●●● ● ●●●● ●●● ●● ●●●● ● ●●●● ●●● ●●● ●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ● ●●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ●●● ●● ● ●●● ● ● ● ● ●●● ● ●●● ● ●

●

●

●● ●●● ●● ●

●●●●●

●●●●●● ● ●●● ●

●

●

●

●●● ●●●●

●●●● ●●● ●●

●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●●● ●

●● ● ● ● ●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ● ●●●●●● ●●● ●●● ● ●● ●●●● ● ●●●●●● ● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ●● ● ●●● ● ● ● ●●●●● ● ●● ● ●●●●● ● ●●●●●●

●●

●●●● ● ●● ●● ●● ●●● ●●●●●● ●●●●● ● ●●● ●●●● ●● ●●● ● ●●●●●●● ● ●●●● ●●● ●●●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ● ●●● ●●●● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ●●● ●● ● ●●● ● ● ● ● ●●● ● ●●● ● ●

●●

●●●●● ● ●●●● ●●●●●●●●● ● ●●● ● ●

●●●●● ● ●●●

●●●● ● ●● ● ●●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●

●● ● ● ●●●● ● ●●●● ●●● ●● ●●●●● ● ● ●●●●●●

●●●●●● ●●● ●●●● ●● ●●●● ● ●● ●●●● ● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ●● ● ●●● ● ● ● ●●●●● ● ●● ● ●●●●● ● ●●● ●●●●

●●●●● ● ●● ● ● ●● ●●● ●●●●●● ●●●●● ● ●●● ●●●● ● ● ●●● ● ●●●●●●● ● ●●●● ●●● ●●●●●● ● ●●●● ●●● ●● ● ●●●● ● ● ●●

●●● ● ●●●●●● ●●●●●●● ●● ●●●● ● ●●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ●● ●● ●● ● ●● ● ●● ●● ●●●● ●● ● ●●● ● ● ● ● ●●●● ●●● ● ●

● ●●● ● ●● ● ●●●● ● ●● ●● ●●●● ● ●●● ● ●

●●●●● ● ●● ● ●

●●● ●●● ●●●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●●● ●● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ● ●● ●●●● ●●●●●●● ●●●●●● ● ●● ●●●● ● ● ● ●●●● ● ● ●●●●●● ●●●● ● ●● ●●●●●●● ● ● ● ●●● ● ● ●●●●●● ● ●● ● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●●● ●●●●● ● ●●● ●●● ● ● ● ●●● ● ●●●●●●● ● ●●●● ● ●● ●●●●●● ● ●●●● ●●● ●● ●●●● ● ● ● ● ●●●● ●●●●●●● ●●●●●●● ●●●●●● ● ●●● ●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ● ●● ● ● ●●●● ●● ● ●●● ● ● ● ● ●●● ● ●●● ● ● ●●

●● ●●● ● ● ●●●● ●● ●●●●●● ● ●●● ● ● ●●●●● ● ●● ● ●●● ● ●●● ●●●● ● ●●●●● ● ●● ● ●●●● ● ● ● ● ●●● ●●● ●●●● ● ● ● ●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ● ●● ●●●● ●●●●●●● ●●●●●●● ●● ●●●● ● ● ● ●●●● ● ● ●●●●●● ●●●● ● ●● ●●●●●●● ● ● ● ●●● ● ● ●●●●●● ● ●● ● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●●● ●●●●● ● ●●● ●●● ● ● ● ●●● ●●●●●●●● ●●●●● ● ●●●●●●●● ● ●●●● ●●● ●●● ●●●● ● ● ●●●●● ●●● ●●●● ●●●●●●● ●●●●●● ● ●●● ●●●● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ●●●● ●● ● ●● ● ● ● ●● ●●●● ●●● ●●● ● ● ● ● ●●●● ●●● ● ● ● ●●●●●● ● ● ●●●●●● ●● ●●●● ● ●●● ● ● ●●●●● ● ●● ● ●●●● ●●● ●●●● ● ●● ●●● ● ●● ●●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ●●●● ● ● ●●●●● ●●●●●●● ●● ●●●●● ● ● ●●●● ● ●●● ●●●● ● ●●●●●● ●●●●●●● ●● ●●●● ● ● ● ●●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ● ● ● ●●● ● ● ●● ●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●● ● ●●●●● ● ●●● ●●● ●●● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●● ● ● ●●●● ●●● ●●●●●●● ●●● ●●●● ●●● ●●●● ●●● ●●●● ● ●●●●● ● ●●● ●●●● ●●●●●●●● ● ● ●●●● ● ●●●●● ● ●●●●●● ● ●● ● ● ● ●● ●●●●●● ● ●●● ● ●● ● ●●●● ●●● ● ● ●●●●●●● ● ● ●●● ●●● ●●●●●●● ●●● ● ● ●●●●● ● ●●● ●●●● ●●● ● ●●●● ●●●●● ● ●● ●●●●● ● ● ● ● ●●● ●●● ● ●●● ● ● ●●●●● ● ●●●●●● ●●●●●●● ●● ●●●●● ● ● ●●●● ● ●● ●●●●● ●●●●●●● ●●●●●●● ●●●●●● ● ●● ●●●●● ● ●●●●●● ●●●● ● ●● ● ● ●●●●● ●● ● ●●● ● ● ●● ●●●● ● ●●● ●●●●● ● ●●● ●●

99%

●●● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●●● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●●●●●● ● ●●●●●● ●●● ●●●● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●● ● ●●● ● ●●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ● ●●●● ●●● ● ●● ●● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●●● ●●● ● ●●●● ●●●●●● ● ●●●●●● ●● ● ● ●●● ●●● ● ●●● ● ●●●●●●●

● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ●●●●●

●● ● ●●●●●● ●● ●●●● ●●● ●●●● ● ●●●●●●● ● ● ●●●●● ●●●● ● ●●● ● ●●●●●●● ●●●●●

● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●●

● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●●●●● ● ●●●●●●

●

●●● ●●●● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●●● ●●● ● ●●● ● ●●●●●● ●●●● ● ● ● ●●●● ●●●

●● ● ●●●● ●●● ●

●● ●● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●●● ●●●●●●● ●● ● ●●●● ●●● ● ●●● ● ●●●●●●●● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ●●●●●●● ● ●●●●●● ●● ●●●● ●● ●

●●●● ● ●●●●●●● ● ● ●●●●● ●●●● ● ●●● ● ●●●●●●●

●●●●● ● ● ● ●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●●

●● ●●●● ●●● ●● ● ●●●●

●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●●●●● ● ●●●●●●

●

●●● ●●●● ●●

●

●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ●● ●●

●●●

●●●●●● ● ● ●●●●● ●●● ●●●● ●

●●●●●● ●●●●● ● ● ●●

●●

●

●●

●

● ● ●●●● ●●● ● ●● ●● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●●●

●●●●●● ●

●● ● ●●● ● ●●● ●

●●● ● ●●●●●●●

● ●●●● ● ●●●●●●● ● ● ●●●● ● ●

●●●●● ● ●●●●●●● ● ●●●●● ●●●●●●● ●● ● ●●●● ● ●●●●●●● ● ●●●●●●●●●● ● ●●● ● ●●●●●●● ●●●●● ● ● ● ●●●●

● ●●● ●●●●● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●● ● ●●●

●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●●● ●●●●●

●●●● ● ●●●●●

●

●

●●● ●●●● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●●●● ●●

●● ● ●●●●●● ● ●●●●●● ●●● ● ●●● ● ●●●●●● ●●●●●● ● ●●●●●

●● ●● ● ●●●● ●●● ●

●● ● ● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●● ● ●●●●●● ●

●●● ●●● ● ●●● ●

●●● ● ●●●●●●●

● ●●●● ● ●●

●●●●● ●● ●●●● ● ● ●●●●● ● ●● ●●●●● ● ●●●●● ●●●●●●● ●● ●●●●● ● ●●●●●●● ● ●●●●●●●●●●● ● ●● ● ●●●●●

●● ●●●● ●● ● ● ●●●● ● ●●● ●●●● ● ● ●●● ●● ● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●● ●●● ●● ●●●● ●●● ●● ● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ● ●●● ● ●●●●●●●●● ●●●●●●● ● ●●● ●●● ● ●●● ●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●●●● ● ●●● ● ●●●●●

●

● ●●●●●●

●●● ●●●● ● ●●●●●● ●●●●●

●● ●●●●●●

● ● ● ● ●●●● ●●● ●●

● ● ● ● ● ●● ●●●●●● ●●● ● ● ●●●●● ● ●● ●●●●

● ●●● ●●●●● ●●●●●

● ●●●●●● ●●● ● ●●●

● ●●● ● ●●● ● ●●●●● ●●

● ●●●● ● ●●●●●● ● ●● ●●●● ● ● ●●●●● ● ●● ●●●●● ● ●●●●● ●●●●●●● ●● ●●●●● ● ●●●●●●● ● ●●●●● ●●●●●● ● ●● ● ●●●●●●● ●●●● ● ● ● ● ●●● ● ● ●●● ●●●● ● ● ●●● ●● ● ● ●●●●●● ●●● ● ●● ●●● ●●●●●● ●●●●● ●●●● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●● ● ● ●●●● ●●● ● ●

●● ● ●●●●●●●●● ●●●●●●● ● ●●● ●●● ● ●●● ●●● ● ● ● ●●●● ● ●●●●●● ● ●●●●●● ●●●●●

●● ● ●●●●●

●● ●●●●●● ●●● ● ●●● ●

●●●●●●

●●●●●●● ●●●●●●●

●

● ● ●●●●

●●● ●●

● ●● ●● ●●●●●●●● ●●● ● ● ●●●●

●● ●● ●●●● ● ●●● ●●● ●● ●●●●

●● ●●●●●● ●●● ● ●●

●

● ●●●

●●●● ● ●●●●● ●●● ●●●● ● ●●●●●● ●●● ●●●● ● ● ●●●●● ● ●● ●●●● ● ● ●●●●● ●●●●●●● ●● ●●●●● ●

●●●

●●● ● ● ●●●●● ●●●●●● ● ●● ● ●●●● ●●● ●●●● ● ● ● ● ●●● ● ● ●●●●●●● ● ● ●●● ●● ● ● ●●●●● ●

●●

●

●

●● ●● ●

●

●●●●

● ●●●●● ●●●● ●●●

●● ● ●●● ●

●●●●●●● ●

●

●●

●

● ● ●●●● ●●●

●●

●● ●●●●

●●●●●●●●●●●●● ● ●●● ●●● ● ●●●●●●

● ● ●●

●●●

● ●●●●●●

● ●●●●●● ●●● ●●●

●

● ●●●●●

●

● ●●●●●●

●●● ● ●●● ●

●●●●●●

●●●●●

●●●●●●●●● ● ● ● ●●●

●●●● ● ●

●●● ●● ●●●● ●●●● ●●● ● ● ●●●●●

● ●● ●●●●

● ●●● ●●● ●● ●●●●● ● ●●●●●● ●●● ● ●●● ● ●●● ●●●●

●●

●●●●●●● ●●●●

● ●●●●●● ●●●

●●

●●

●● ●

●●●●● ●● ●●●● ●

●

●

●●

●●●●●

●●●● ●

●

●● ●●●

●●

●●●

●● ●●

●

●●●● ●

●●●●● ● ●● ●

●●

●● ●● ● ●●●● ● ● ● ● ●●● ● ● ●●●●●●

● ● ● ●●● ●● ● ●●

●●●●

●

●●●

●

●● ●● ●

●

●●●●

● ●●●●● ●●●● ●●●●● ● ●●● ●

●

●●●●●●●

●

●●●●● ●●●● ●●●● ●●● ●●●

●●●●●●●

●●●●●●● ● ●●● ●●● ● ●●●●●● ● ● ●

●

●●●

● ●●●●●● ● ●●●●●● ●●● ●●●

●

●●●●●●●●●●●●

●●●●● ● ●●● ●

●●●●●● ●●●●●●● ●●●●●●● ● ● ● ●●●●●●● ● ●●● ● ●●

●●●● ●●●●●●● ● ● ●●●●●● ●● ●●●

●● ●●● ●●● ●● ●●●●●

● ●●●●●●

●●● ● ●●●● ●●● ●●●● ● ●●●●

● ●●● ●●●●● ●●●●●● ●●● ●●

●●●

●

●●●●●● ●

● ●●●

● ●● ●●

●

●●●

●●●●●● ●●

●● ●●● ● ●●●

●

●● ● ●

●

●●●● ●●●●●● ● ●● ●

●

●

●● ●● ● ●●●● ● ● ● ●●

●● ●● ●●●●●●● ● ● ●●● ●● ● ● ●●●●● ●●●● ●●● ●● ●

●●●●

●● ●●●●● ●●● ● ●●● ●● ● ●●● ● ●●●●●●● ● ●●●●●● ●●●● ●●●

●●●● ●●● ●●●●●●● ●●●●●●● ● ●●● ●●● ● ●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●●● ●●●

●●●●

●●●●●●

●●●●● ●●● ● ●●● ●

●

●●●●● ●●● ●●●● ●●●● ●●● ● ● ● ●●●●●

● ● ● ●● ● ● ●●●●●● ●●●●●●● ● ●

●

●●●● ●●

● ●●●● ●●●● ●●● ●● ●●●●● ● ●●●●●● ●●● ● ●●● ● ●●● ●●●●● ●●●●● ●●● ●●●● ● ●

●●●●● ●● ● ●●●● ● ● ●●●●●● ●● ●●●

● ● ●●●

●

●● ●●●●●●●●

●

●● ●

●● ●

●●●●

●●

● ● ●●●●● ●●●●● ● ● ●● ● ●●●● ●● ● ●●

●● ● ● ● ●●●● ●● ● ●●●●●● ●● ●●● ●● ● ● ●●●●● ●●●●●

●● ●● ● ●●●●

●● ●●●●● ●●● ● ●●● ●● ●●●● ● ●●●●●●● ● ●●● ●●● ●●●● ●●●

●

●●● ●●●●

●●●●●●●●●●●●● ● ●●● ●●● ●●●●●●● ● ●

●

●

●●● ●●●●●●●

●

●●●●●● ●

●

● ●●●

●●●●●●●● ● ●●●●●● ●● ● ● ●●● ● ●●●●●● ●●● ● ●●● ●●●● ●●● ● ● ● ●

●●● ●● ● ● ●● ● ● ●●

●●●

● ●●●●●●● ●

●

●●●●● ● ●● ●●●● ●●● ●

●●● ●● ●●●●● ● ●●●●●● ●● ● ● ●●● ● ●

●● ●●●●●

●●●●● ●● ● ●●●● ●●

●●●●● ●● ● ●●●● ● ● ●●●●●

●

●● ●●●● ●●●●●

●●●●●●●●●

●●●● ●

●

● ●●●●●●●

● ● ●●●

●● ●●●●● ●● ●●

● ●●●● ●● ● ●●●● ● ● ●●●●● ●● ● ●●●● ●●

●●●●● ●● ● ● ●

●●●●

●

● ●●

●

●●●

● ● ●●

●

●●

● ●●●●● ●●● ● ●●● ●● ●●●● ● ●●● ●●●● ●●●● ●●● ●● ●● ●●●●●

●● ●●● ●●●●●

●●●●●●●●● ● ●●● ●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●●

●● ● ●●●●

●

●●●●●● ●●●●●●● ●● ● ●

●●● ●

●●●

●●● ●●● ● ●● ● ●●●● ●●● ● ● ●●

●●● ●● ● ● ●● ● ● ●●

●

●●●●●●●●● ● ●

●●

●●●●

●●● ●●●● ●●● ●

●●●

●

● ●●●●● ●● ●●●●● ●● ● ● ●●●●●

●● ●●● ● ●●●●●● ●● ● ●●●● ● ●●● ●●● ●● ● ●●●● ● ●

●

●

●●●●

●●●●●● ●●●● ●●●●●●●●●●

●

●

●

● ●●●

●●●●

●

●

●● ● ●

●

●

●● ●●●●● ●● ● ●●●●●● ●● ● ●●●● ● ● ●●●●● ●● ● ● ●●● ●●

●

●●●● ●● ● ● ●●●●● ●● ●● ●●● ●● ● ●●●

●

●

● ●●●●● ●●● ● ●●● ●● ●●●● ● ●●● ●●●● ●●●● ●●● ●● ●●●●●

●

●●● ●●● ●●●●●●●●

●●

●

●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ● ● ●●●●

●

●●●●●● ● ●●●●●● ●● ●●

●●● ●

●

●●●●● ●●● ● ●● ● ●●●● ●●● ● ●●

●●●● ●● ● ● ●● ● ● ●●

●●●

●●●

● ●●● ● ● ●

●

●●● ●●

●● ●●

●

●●●● ● ●

●● ●

● ●●●●

● ●●●●●●● ●● ● ● ●●●

● ●●●●

●● ●●●

● ●

●

● ●● ● ●●●● ● ●●● ●●● ●● ●

●

●●● ● ● ●●●

●● ●●●●●●● ●●●● ●●● ●●●●●●● ●●

●●●●● ●● ●●●●●

● ● ●●●●● ●●●●● ●● ●●●●●●● ●● ● ●●●● ● ● ●●●●● ●●● ● ●●● ●●

●●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●●●● ● ●●● ●●● ● ● ●●● ●●● ●● ●●●●● ●●●● ●●● ●●●●●●●

●●●●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●● ●●●●

● ● ● ●●●●●

●●●●●●●

●

●●●● ● ●● ● ● ●● ● ●

●

●●

●●● ●● ● ● ●● ● ●●●● ●●● ● ● ● ●●●●●

● ● ● ●● ● ● ●●●●

●●

●●●

●●●

● ●

●

●

●●● ●

●●● ●●●● ●●● ●

●●● ●● ●●●●● ●● ●●●●● ● ● ● ● ●●●

● ●●● ●●●

● ●●

● ●

●

● ●● ● ●●

●●● ●●● ●●● ●● ●

●●●● ● ● ●●

●●●

●

●●●●●●●

●●●●

●● ●●●●●●● ●● ●●●●● ●●●

●●●● ● ●●●●●● ●●●●● ●● ● ●●●●●● ●● ● ●●●● ● ●●●●●● ●●● ● ●●● ●●●

●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●●●●● ●●● ●●● ● ● ●●●●

●● ●● ●●●●● ●●●● ●●● ●●●●●●● ● ●●●●●● ● ●●●●●● ●●●●●●●● ● ●

●●●● ●●●●●●● ●●● ●●●● ● ● ● ●●●● ●●●●●●● ●●

●●●● ● ●● ●

●

●

● ● ●

●

●●●●● ●● ● ● ●● ● ●●●● ●●

● ● ● ●●●●●●

● ● ● ●● ● ●

●●

●

●

●●●

●● ●●●

●

●

●

●

●●● ●

●●● ●●●

● ●●● ● ●

●●●● ●●●●● ●

●●●●

●● ● ● ●●

●●● ● ●● ● ●●

● ●

●

●●●

●

● ●● ● ●●●● ● ●●● ●●● ● ● ●

●

●●● ● ●

●●●●●

●●●●●●● ●●●●

●●● ● ●●●●●● ●● ●●●●● ●● ●

●●●● ● ●●●●●● ●●●● ● ●● ●●●●●●● ●●● ● ●● ● ●●●●●●● ●●● ● ●●● ●●

●

●●●● ●● ● ●●●●●● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●● ●●● ●●● ●●● ● ● ●●● ●●● ●● ●● ●● ● ●●●●●●● ●●●●●●● ● ●● ●●●● ● ●●●●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●●

●●

●●●●●

●●●●●●● ● ● ●●●● ● ●●

● ●●● ● ●

●

●●●●● ●● ● ● ●● ● ●●● ●

●

●● ● ● ●●●●

●●● ●

●●

● ●● ●●

●

●

●●●

●● ●●● ●

● ●●

●●● ●

●

●● ●●●

●

●●● ● ●

●

●

●● ●●●●●

●

●

●●●

●● ● ● ●

●

●

●●

●●● ●

●

●

● ● ●

●● ●

●● ●● ● ●●●● ● ●●● ●●● ● ● ●●

●●● ● ● ●●

●●● ●●●●●●● ●●●● ●●●

● ● ●●●● ● ●●●●●●● ●● ●●●●●● ●●●●●● ●●●● ● ●● ●

●

●●●●● ●●● ● ●● ● ●●●●●●● ●●● ● ●●● ●●●●●●● ●● ● ●●●●● ● ●● ●● ●●● ●● ● ●●●● ●●● ●●●● ●●● ●●●● ●● ●● ●●●●●● ●●● ● ● ●●● ●●● ●● ●● ●

● ● ●●● ●●●● ●●●●●●●● ●●

●

●●● ●●●●●●● ●●●●●●●

● ● ●●●● ●●●●●●●● ●●●●●●●

●● ● ●

●●● ●●●●●●● ●●●

●●● ● ●● ● ●●

● ●●

●

●●●●● ●● ● ● ●● ● ●●● ● ●●● ●

●

●●

●●

●

●● ●● ●

● ● ● ●

●

●●

●●●● ● ●

●●● ●

●

●●

●● ●●

●●

●●●●

●●● ●●

●

●

●● ●●●●●

●●●●●

●● ●●

●● ●●● ●

●● ●

●

●● ●

●

●

●

●

●● ●● ●●●●● ●●●● ●●

● ● ● ●●

●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ●

●●●● ● ●

●●●●●● ●● ●●●●● ● ●●●●●● ●●●● ● ●● ●

●●●●●● ●●● ●●● ● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●● ● ●●●●● ● ●● ●● ●●●●

● ● ●●●●

●● ● ●●● ● ●●● ●●●● ● ● ●● ●●●●●● ●●● ● ● ●●● ●●● ●● ●● ●● ● ●●● ●●●● ●●●●●●● ● ●●●

●●● ●●●●●●● ● ●●●●●● ● ●●●●● ●●

●●●●●● ●●●●●●●● ●

●●

●●● ●●●●●●●●

●●

●●

●

● ●● ●

● ●

● ● ●

●

●●●● ● ●● ● ● ●●

●●●● ●

●

●● ●

●●●

●●●●●

●●

●

●● ● ●

● ●

●

●●●

●●

●

●● ●

●

●

●●●●●

●

●

●●

●●●●

●●

● ●●

●●● ●●●●●

●

● ● ●●

●● ●

●●

●

●●●

● ●● ●

●

●●●

●●●

●

●● ● ●

●

●●●● ●●●● ●●

● ● ● ●

●●

●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●● ●●●●● ●●

●

●●●

● ● ●●●

●●● ●●●●

● ●● ●

●

●●●●● ●● ● ●●● ● ● ●●●●●● ●●● ● ●●● ●● ●●●●● ●●

●●

●

●●● ● ●● ●●

●

●●

●●

●

●●

●

● ●●●●●● ● ●●● ●●●● ● ●

●●

●●●●●● ●●● ● ● ●●●

●●● ●●

●● ●● ● ●●● ●●●● ●● ●●●●● ● ●●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●●●

●●●● ●●●●●●●●

●●

●●●

● ●●●●●●●●

●

●●●● ● ●● ●

● ●

● ● ●

●

●●●● ● ●● ● ● ●●

●

●●● ●

●

●● ●

●

●

●●●

●

●● ● ●

●

● ● ●●●

●

●

●

●

●

● ● ●●● ●

●

●

●●

●● ●

●

● ●

●

●●● ●

●

●●●

●● ●●●●●●●●

● ● ●●●● ● ●● ●●

●●

●

●● ●

●

●●

●●●

●●●● ● ● ●●●●● ●●●● ●●● ● ● ●

●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●●●●●●

●

●

●

●

●

●

●● ●

● ●●●

●● ●●●● ● ●● ●

●

●

●

●●● ●●●

●●● ●

● ●●●●●● ●●● ● ●●● ●● ●

●●●

● ●●● ●

●●●● ● ●● ●●●

●● ●●●●●

●● ●● ●●●● ● ●●● ●●●● ●● ●●● ●●●●●●●● ● ●●

●● ●

●

● ●●●●

●

●

●●

●●●●●● ●● ● ●●●● ● ●●●●●● ●●●●●●● ●●●●●●● ● ● ●●●● ● ●●●●●●● ●●● ●●●●● ● ●

●●●● ●●●●●●● ● ●●●●● ● ●● ● ● ●

● ●●

●●●●● ● ●● ● ● ● ● ●●●● ● ●

●● ●

●

●

●●● ●●●

● ●●

● ●●●

● ●

●

●● ●● ● ●●●

● ●

●

●●●●●

●

●● ●●●● ●●● ●●●● ● ●●●●● ● ●● ● ●●●● ● ● ●

●●●●

●

●● ● ●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●●● ●

●●

●●●●●

●●●

●

●● ●

●●●

●●●

●

●

● ●●● ●

●

●

●

●●●● ●●●●●●● ●●●● ● ●● ●

●●

●

●●● ●●●

●●● ● ● ●●●●●●● ●● ● ●●●●● ●●●●●●●

● ●●●●● ● ●●●● ●● ●●

●

●●●●●●●●●●● ● ●●● ●●●● ●● ●●● ●●●●●●●● ● ●●●●● ●● ●●●●●●

● ●●●● ●●● ●

●

● ●●●

●● ●

●●

●●●●●●●●●● ●●●●●●● ● ● ●

●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ●● ●●●● ● ● ●● ● ● ●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ●

●●

●●

●

●● ● ●●● ● ●● ●●

●

●●●● ● ●●●● ●

●

●●●● ●

●

● ● ●●●● ●●● ●●●

● ● ●●●●● ● ●●●●●●● ● ● ● ●●●●●

●● ● ●●●

●●

●●●●● ● ●

●

●●●●

●

●●

●

●●● ● ● ●●●● ● ● ●●

●●●●

●

●●

●●●●

●●

●●

●

●● ●● ●●●● ●

●●● ●●● ● ●●

●

●●●● ●●●●●●● ●●●● ● ●

●● ●●●●●● ●● ● ●●● ● ● ●●●●●● ● ●● ● ●●●●● ●●●●●●● ● ●●●●● ● ●●●●

●● ●●●

●●●●●●●●●●● ● ●●● ●●●● ●● ●●●●●●●●●●● ● ●●●●● ●● ●●●●●●

●●

●●● ●●●

●●

●●●● ●●

●

●●

●● ●●

●●●●●● ●●●●●●● ●● ●●

●●● ●

●●●

●● ● ●●●●●●●● ●

●●

●●● ●●●●●● ●● ●●●● ● ●●● ●

●●● ●

●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ●●●

●● ●●● ● ●●● ● ●●

●●

●

●●●● ● ●●● ● ●

●

●●●● ●

●

● ● ●●●● ●●● ●●●

● ● ●●●●● ● ●

●

●●●●● ● ● ●●

●●●

●

●● ●●

●●

●

●

●

●●●● ● ● ●●●● ●

●

●●●●●● ● ● ●●●● ● ● ●●

●●●●●

●●●●● ● ●●●●●● ● ●● ●●●● ●●

●●●●● ●

●●●

●●●● ● ●●

●●●● ●●●● ● ●●

●●●●●●● ●●

● ●●● ● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●●●●

● ●●● ●●●●●●●●●●● ● ●●● ●●● ● ●● ●●● ●●●●●●●● ● ●●●● ● ●● ●●●●●●● ●

●●● ●●● ●●●●●● ●●

●●●

●● ● ●●●●●●● ●●●●●●● ● ●●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ●

● ●●●●● ●●● ●

●●● ●

●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ●

●

●●●●

●● ● ●●● ●●●

● ●

●

●●●● ● ●●● ● ●

●

●●●● ●

●

●●●●● ● ●●● ●●●

● ● ●●●●● ● ●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●

●

●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●●

●●

● ●●●●● ●

●

●●●

●● ● ● ●●●●● ●●●● ● ●● ● ●●

●●●● ●● ● ●●● ● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●● ● ●●●●● ● ●●● ● ●● ●●● ●●●●●●●●●●● ● ●●● ●●● ● ●● ●●● ●●●●●●●● ● ●●●● ● ●●●●●●●● ● ●●●● ●●● ●●●●●● ● ● ●●●●●● ●●● ●●●● ●●●●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●● ● ●●● ● ●●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●● ● ●● ● ● ●●●●● ● ●●● ●● ●

●●●● ●●

● ●●●● ● ●●● ●●●● ● ●●●●●● ●●●●●●● ● ● ● ●●●● ●●● ● ●●● ●●●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●●● ●●●●●●● ●●●●●●● ●● ●●●● ●

●● ●●●●● ●●●●●●● ● ● ●●●●● ●●●● ● ●● ● ● ●●●●● ●● ● ●●●● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●●

● ●●●●● ● ●●● ● ●● ●●● ●●●●●●●●●●● ●●●● ●●● ●●● ●●● ●●●●●●●● ● ●●●● ● ●●●●● ●●● ●●

●●● ●●● ●●●●●●● ● ●●●●●● ●●● ●●●● ●●●●●●● ● ● ●●●● ● ●●●●●● ● ●●●●●●● ● ● ● ●●●● ●●●●●● ● ● ●●●● ● ●●● ● ●●● ● ●●●●●● ● ●● ● ● ● ● ●●●● ● ●●● ● ● ●●●● ●●● ● ●●●● ● ● ● ● ●●●●●● ●●● ● ●●

●●●● ● ●● ●●●● ● ●●● ● ●●●● ●●●●●● ● ●●●●●● ● ● ● ●●●● ●●● ● ●●● ● ●●●●●● ● ● ●●●●● ●●●●●●● ● ● ●●●●● ● ● ●●●● ● ●●●●●●● ● ●●●●●● ●●●●●● ●●● ● ●●●● ●●●●●●● ● ●●●●●● ●●●● ● ●●● ● ●●●●● ●●

●●●●● ● ● ●●●●● ● ●●● ●●●●● ● ●●● ●●

99%

DA

RT

−100

0

100

200

300

400

−100

0

100

200

300

400

50 75 100 125 150Generation (GW)

LMP

($/

MW

h)

LMP versus Generation, PJM, 2012

Supply curves are approximately linear, and steeper in RT


Two-Settlement Market Model: Nomenclature

financial physical

demand

supply

DA LMP p(x)

v+ + x = v− + y

INC v+ DA gen x

DEC v− DA load y

RT LMP p(x)

x = y

RT gen x

RT load y

Observations from the market data

x ≈ x , y ≈ y , p′′ ≈ 0, p′′ ≈ 0, p′ > p′

Next we show how p depends on x


RT Supply Curve Conditioned on DA Generation

x

p

DA

x

p

RT

x

Inflexible Flexible

RT supply curve steeper: dispatchable generation

Intercept of RT supply curve depends on x


Modeling DA/RT Supply Curves

x , x

p, p

p(x) = ax

ηx

p(x) = ax + b

x

Inflexible Flexible

Inflexible generation uniformly distributed (proportion η)

Representing b and η in terms of a and a{p(ηx) = 0

p(x) = p(x)=⇒

{b = (a− a)x

η = 1− (a/a)


Generation Cost Depends on DA Generation

x x , x

p, p

p(x) = ax

ηx

p(x) = ax + b

x

Inflexible Flexible

Generation cost = + = DA gen cost + flexible gen cost

c(x) = η

∫ x

0azdz +

∫ x

ηx(az + b)dz

= η

∫ x

0azdz +

∫ x−ηx

0azdz


Cost Min. ⇐⇒ Dispatch Conv. ⇐⇒ Price Conv.

x

p > pif x < x

p = p if x = x

x , x

p, p

p(x) = ax

ηx

p(x) = ax + b

x

Inflexible Flexible

x > x : cost of generation dispatched in RT =

Generation cost = +

p > p

If we had x = x , cost would be minimized: savings = , p = p


Cost Min. ⇐⇒ Dispatch Conv. ⇐⇒ Price Conv.

x

p < p if x > x

p = pif x = x

x , x

p, p

p(x) = ax

ηx

p(x) = ax + b

x

Inflexible Flexible

x < x : cost reduction of generation descheduled in RT =

Generation cost = +

p < p

If we had x = x , cost would be minimized: savings = , p = p


Two-Settlement Market Model: Complete Specification

DA supply curve: p(x) = ax

RT supply curve: p(x) = ax + b

RT load L is fixed: x = L

a and a are fixed with a < a, and so η = 1− (a/a)

b is subject to an indepedent disturbance δ

b = b + δ = (a− a)x + δ

Empirical estimation

Yearp = ax p = ax + b

ηa (×10−3) R2 a (×10−3) b R2

2012 0.37 0.93 0.68 −27.5 0.30 45.4%

2013 0.42 0.91 0.76 −31.8 0.34 44.6%

2014 0.56 0.55 1.72 −108.2 0.18 67.3%

2015 0.41 0.78 0.86 −44.7 0.28 53.0%


Two-Settlement Market Model: Complete Specification

L0: market’s forecast about L, the RT load

δ0: market’s forecast about δ, the RT supply function disturbance

Market minimizes the forecast generation cost over x

minx

c(x) = η

∫ x

0azdz +

∫ L0

ηx(az + (a− a)x + δ0)dz

DA generationx0 = L0 + (δ0/a)

Spread measures the forecast accuracy

s0 = p0 − p0 = a(L0 − L) + (δ0 − δ)

Financial efficiency and economic efficiency are aligned

price convergence ⇐⇒ cost minimization

δ=0⇐⇒ dispatch convergence


Theory of Virtual Bidding

While virtual bids do not affect RT generation, they affect DAgeneration and therefore DA LMP, RT LMP and generation cost

We formulate a game with virtual bidders as strategic players, basedon the two-settlement market model

N: number of virtual bidders

Li : virtual bidder i ’s forecast about L

δi : virtual bidder i ’s forecast about δ

vi : virtual bidder i ’s quantity bid (INC if vi > 0, DEC if vi < 0)

DA generation: x = x0 −∑

i vi

DA LMP: p = ax

Virtual bidder i ’s forecast RT LMP: pi = aLi + b + δi

Virtual bidder i ’s forecast profit: πi = (p − pi )vi



Solving the simultaneous FOCs

∂πi (vi , v−i )

∂vi= 0, i = 1, . . . ,N

yields ∑i

vi =N(aL0 + δ0)− (a

∑i Li +

∑δi )

(N + 1)a

Equilibrium spread measures the average forecast accuracy

s∗ =a(L0 − L) + (δ0 − δ) +

∑i (a(Li − L) + (δi − δ))

N + 1

=s0 +

∑i (a(Li − L) + (δi − δ))

N + 1



Sufficient Condition of Price Convergence

If1

N

∣∣∣∣∣∑i

(a(Li − L) + (δi − δ))

∣∣∣∣∣ < a(L0 − L) + (δ0 − δ),

then|s∗| < |s0|

Cournot Theorem for Virtual Bidding

If1

N

∣∣∣∣∣∑i

(a(Li − L) + (δi − δ))

∣∣∣∣∣→ 0 as N →∞,

then|s∗| → 0 as N →∞



Profitability and Price Convergence

Virtual bidder i makes a positive profit if and only if its participationdrives the spread toward zero:

s∗vi > 0 ⇐⇒

{0 < s∗ < s∗−i , s∗−i > 0

s∗−i < s∗ < 0, s∗−i < 0,

where s∗−i is the equilibrium spread without the participation of i

Screening out unqualified virtual bidders with poor forecast accuracy

Introducing more qualified virtual bidders into the market

The virtual bidding mechanism is self-incentivizing: a virtual biddercan make more profit by improving its forecast accuracy, which is alsofavorable to the market


Profit and Cost Savings of Virtual Bids

Market’s DA generation: x0 = L0 + (δ0/a)

Optimal DA generation: x∗ = L + (δ/a)

Net INC of the virtual bids: v

v∗ = x0 − x∗ minimizes the generation cost and induces zero spread

Profit of the virtual bids

f (v) = −av2 + av∗v

Cost savings of the virtual bids

g(v) = −(aη/2)v2 + aηv∗v


Profit and Cost Savings of Virtual Bids

v

f(v)

g(v)v∗

2v∗

η = 0.25

v

f(v)

g(v)

v∗

2v∗

η = 0.5

v

f(v)g(v)

v∗

2v∗

η = 0.75

profit > cost savings profit < cost savings

The generation cost savings may not recover the profit of the virtualbids when v is small

As η increases, the cost savings are more likely to recover the profit

The more competitive the virtual bidders, the more likely that thecost savings recover the profit


Effectiveness of Virtual Bidding: Empirical Estimation

●

●

●

●

−5.0

−2.5

0.0

2.5

5.0

2012 2013 2014 2015Year

Spr

ead

($/M

Wh)

Virtual Bids● No (Estimate)

Yes (Market Data)

Mean of Spread with and without Virtual Bids, PJM

YearCost (Billions) Savings (pct. Profit (pct.

No VB VB of No VB Cost) of Savings)

2012 22.14 21.71 1.95% 8.03%

2013 24.03 23.66 1.53% 2.86%

2014 29.58 28.48 3.73% 1.59%

2015 20.29 20.24 0.23% 60.21%


Conclusion

A successful fusion of data and theory: market data → market model→ theory and implications → methodology of estimation → empiricalevidence by market data

We propose a two-settlement market model which explains variousphenomena in the market, and provides a methodology of estimatingthe generation cost

The proposed model aligns financial efficiency and economicefficiency, which serves as the basis of the theory of virtual bidding

Virtual bidding has improved the market efficiency: (i) comparativeanalysis before and after virtual bidding in CAISO; (ii) estimation ofcost savings and price convergence by virtual bids in PJM

There still exist substantial profitable opportunities: (i) market virtualbidders make profits; (ii) simple machine learning algorithms can beprofitable on top of the market virtual bids


All of Virtual Bidding: A Data-Driven Approach

Engineering

Transcript of All of Virtual Bidding: A Data-Driven Approach