Selection bias & the value of certainty

Presented by:

Matthew Hendrickson, Sr. Dir. Of Assessment

Renewable Energy

Risk Analysis

Powered by weather science, 3TIER helps the

global energy market manage renewable

energy risk—for the next 5 minutes through

the next 50 years.

3TIER is Risk Management

What to do about Risk?

What is Risk?

» Risk is often described as uncertainty.

» A typical wind consultant will describe uncertainty as the

standard deviation of error around the primary estimate.

A few questions

» Do we know what this means?

» How does knowledge of risk guide our decisions?

σ = ?

US Wind Industry Underperformance

In 2008, reports by three US consultants

show that US Wind Industry is

underperforming estimates by 10-11%.

1. “Understanding and Closing the Gap on Plant Performance” – Eric White, AWS Truewind, AWEA WindPower 2008

2. “Project Underperformance: 2008 Udate” – Steve Jones, DNV-GEC, AWEA WindPower 2008

3. “Validation of Energy Predictions by Comparison to Actual Performance, Clint Johnson, AWEA WindPower 2008

Underperformance explained

» No single factor described as major source. Rather a list of

possibilities has been suggested.

› Wind farm availability

› Inter-annual variability

› Turbine performance

› Wake effects

› Wind flow modeling

› Measurement bias

» Advancements in every one of the these fields has taken

place and US performance gap is closing.

Seems like a bad case of luck

» With so many sources, there is a nagging feeling like much

of what is wrong is luck.

» Could it be luck?

» Or something much more predictable…

» Perhaps it could be Selection Bias

Selection Bias Game

Project Selection Game

» You work for a large wind developer working in an active

market.

» As Chief Investment Officer, you are tasked to manage

your companies wind investments.

» All of your investments come in the form of winning

RFP’s to sell power in a power purchase agreement

(PPA). With a PPA, your company can build the project.

» Each project costs $100 in capital. Your companies goal

is to make a desired return of $20 for each $100.

Game mechanics

» Each Investor must pick a project and bid a price that they

think will both win and yield a desired return.

» Lowest price wins.

» True NCF is hidden.

Windy Hill Desert Breeze Ocean Front Gusty Pass Bent CornTrue NCF (hidden) 36.3% 35.0% 37.7% 32.5% 37.9%

Uncrtainty 8.3% 9.2% 9.4% 7.7% 7.5%

UncertainNCF 34.6% 27.9% 39.6% 35.2% 42.9%

$100 56.10 68.18 47.13 55.13 41.21

$110 60.77 72.85 51.80 59.80 45.88

$120 65.45 77.52 56.47 64.47 50.56

$130 70.12 82.19 61.15 69.15 55.23

$140 74.79 86.87 65.82 73.82 59.90

$150 79.46 91.54 70.49 78.49 64.57

Game mechanics

» Your desired return is $120.

» But your CEO becomes very unhappy with returns less than

$110. Since you can’t be fired, he will take investment

capital away from you for low returns and put it somewhere

else.

» He will take away $4 of capital for every $1 below the

threshold return.

» Each projects return gets added to an investors capital.

» The investor with the most capital at the end of the game

wins!

Game mechanics Windy Hill Desert Breeze Ocean Front Gusty Pass Bent Corn

True NCF (hidden) 36.3% 35.0% 37.7% 32.5% 37.9%

Uncrtainty 8.3% 9.2% 9.4% 7.7% 7.5%

UncertainNCF 34.6% 27.9% 39.6% 35.2% 42.9%

$100 56.10 68.18 47.13 55.13 41.21

$110 60.77 72.85 51.80 59.80 45.88

$120 65.45 77.52 56.47 64.47 50.56

$130 70.12 82.19 61.15 69.15 55.23

$140 74.79 86.87 65.82 73.82 59.90

$150 79.46 91.54 70.49 78.49 64.57

» Watch out for uncertainty!

» It might make a project look better or worse than it really is.

» Each projects true NCF varies. That might be a windy

project or it might be a dud.

Play the game

Launch Game

Selection bias model

» Selection model created that simulated selection

» Factors included

› Energy Estimate Uncertainty (Estimator of Risk)

› Weight of NCF in Selection Decision (How much do you care?)

› Market Pressure (Forces outside of your control)

› Selection Pool Size (How many choices?)

» Model ran iteratively across range of values

» Simulated decision process of investor

Selection Model Results

Assumes 30% NCF Weight & 5 project selection pool

Selection Model Results

Assumes 9% uncertainty, 1.1x market threshold, 30% NCF Weight & 5 project selection pool

How to protect yourself

Conclusion – whenever project return is a consideration

(almost always), selection bias will be present.

» Act with conservatism

› Apply conservative factors where appropriate

› But these factors should be transparent to avoid double-dipping

» System must resist uncertainty

› Investing on the P50 level flawed

› Pick a risk level (PX) that is comfortable and stick with it. (e.g.

Invest at the probability level such that your P95 scenerio is at least

a break even scenerio)