UTC Telecom & Technology 2015 - Battery Backup Workshop

© 2013 Utilities Telecom Council

May 5-‐8, 2015 | Atlanta, GA | www.utctelecom.org

BATTERY BACKUP WORKSHOP MAY 5, 2015

NO POWER=NO SERVICEDAKX TURCOTTE

MULTITEL INC.

DISCLAIMER

From concept to more practical & technical content.

2

3

➡ DISCLAIMER

➡ INTRODUCTION ➡ WHAT IS A BATTERY? ➡ WHAT ARE YOUR CHANCES OF RUNNING ON BATTERIES? ➡ WHY DO BATTERIES FAIL? ➡ WHY DO YOU NEED IT?

➡ BATTERY BACKUP: FIELD OPERATIONS ➡ PREVENTIVE MAINTENANCE ➡ OUTAGE INTERVENTION

➡ BATTERY BACKUP: ENGINEERING PERSPECTIVE ➡ PLANNING NEW PROJECTS ➡ REPLACING OLD OR EXISTING PROJECTS

➡ CONCLUSION

➡ SHAMELESS PLUG & FREEBIE

WHAT’S A BATTERY?

a container consisting of one or more cells, in which chemical energy is converted into electricity and used as a source of power.

bat·ter·y /ˈbatərē/ noun

4

WHAT’S A BATTERY?

Electrolytes allow ions to move between the electrodes, which allows current to flow out of the battery to perform work.

5

WHAT’S A BATTERY?

6

WHAT’S A BATTERY?

7

WHAT’S A BATTERY?

Lower specific energy than common fuels. Only energy type that delivers their energy as electricity.

Meaning they are efficiently converted to mechanical work.

8

WHAT’S A BATTERY?

Wet cell (VLA) jar VRLA jar Battery string/bank

9

10

➡ DISCLAIMER




➡ CONCLUSION


11

CHANCES ARE…The stats from our commercial telecom

customers are, on average:

• Number of outages: 4 outages per site per year • Length of each outage: 1.5 hours • Direct cost of losing a telecom site: $250,0001 • Indirect cost of losing a telecom site: $150,0001 • Direct cost of losing a substation’s protection: TBD2 • Indirect cost of losing a substation’s protection: TBD2

Note: 1: Direct and indirect costs attributed to a battery-related outage when a site was lost in Miami, FL by a major US telecom operator 2: “We don’t even want to image it” is more likely the answer…

MOREOVER…

3 things we know with absolute certainty about batteries: (1) limited CAPACITY

(2) finite and unpredictable LIFETIME (3) always fail EXACTLY when you need them

12

What are your chances of running on batteries? I would say: PRETTY HIGH.

13

14

➡ DISCLAIMER




➡ CONCLUSION


TOP 10 #BATTERY_FAILS

10 - Prolonged period without being used 9 - Battery post corrosion, causing (dis)charge problems

8 - Excessive deep discharges 7 - Bad manufacturing process or material

6- Torquing: too much > break terminal seal, too little > short

15

TOP 10 #BATTERY_FAILS

5 - AC ripple (on UPS batteries) 4 - Lack of electrolyte (wet cell batteries)

3 - ↑25°C > corrosion > self discharge > thermal runaway 2 - Over-charging = creates heat = loss of water = sulfation

1 - Under-charging = electrolyte stratification

16

17

➡ DISCLAIMER




➡ CONCLUSION


WHY DO YOU NEED IT?

18

WHY DO YOU NEED IT?

Because there can be up to 15 pieces of equipment making for the “POWER-TRAIN” within a telecom site.

Because OUTAGES & EQUIPMENT MALFUNCTIONS happen. Because it impacts you and your CUSTOMERS.

And because it’s EXPENSIVE.

20

29

WHY DO YOU NEED IT?A few stats on availability or poor backup management:

• 99.9% (Three Nines) availability = 9 hrs/year of downtime.1 • 99.9999% (Six Nines) availability = 30 seconds/year of downtime.2 • 1 hour of downtime equates to $2.0M/hour in the telecom industry.3 • It’s terrible for your customers: $1.4 million/hour to financial institutions.3 • $80B: the amount of yearly losses due to power disturbances in the US.

66% of that caused by momentary disturbances, 34% caused by sustained interruptions of more than 5 minutes.4

References: 1: GilderGroup 2: The dawn of on-site power: http://ecmweb.com/cee-news-archive/dawn-site-power 3: Overhead: The cost of downtime: http://itknowledgeexchange.techtarget.com/overheard/overhead-the-cost-of-downtime/ 4: Berkeley Lab Study Estimates $80 Billion Annual Cost of Power Interruptions: http://www.lbl.gov/Science-Articles/Archive/EETD-power-interruptions.html

http://ecmweb.com/cee-news-archive/dawn-site-power

http://itknowledgeexchange.techtarget.com/overheard/overhead-the-cost-of-downtime/

http://www.lbl.gov/Science-Articles/Archive/EETD-power-interruptions.html

It’s clear, YOU NEED BATTERIES.

30

31

The questions you need to ask:

• How much time do you need? • How much time do you really have? • In what state of health is your backup? • Can you consistently meet your average failure? • What happens if your backup doesn’t work?

So, let’s see what the role of Field Operations is when it comes to battery backup.

32

33

➡ DISCLAIMER




➡ CONCLUSION


Field Maintenance and Operations teams are typically responsible for two aspects of battery backup:

MAINTENANCE ROUTINES & OUTAGE INTERVENTION.

34

35

➡ DISCLAIMER




➡ CONCLUSION


MAINTENANCE ROUTINES

The two reasons why you do battery preventative maintenance routines are:

BATTERIES WILL FAIL & NERC COMPLIANCY.

36

NERC COMPLIANCY

What is NERC COMPLIANCY?

37

NERC COMPLIANCYNERC has multiple RELIABILITY STANDARDS.

One of them is PRC-005-2, which will be enforced starting April 1, 2015, but should have been used starting February 24, 2014.

It covers the MAINTENANCE of all Protection Systems affecting the reliability of the Bulk Electric System (BES) so that these

Protection Systems are kept in working order.

38

NERC COMPLIANCY

Basically, it tells you what tests & time intervals to follow for your BATTERY MAINTENANCE.

It is applicable to both protection relays & TELECOMMUNICATIONS SITES.

39

NERC COMPLIANCYMaximum

Maintenance IntervalMaintenance Activity VLA (wet cell) VRLA

4 Calendar Months

Verify • Station DC supply voltage

Inspect • Electrolyte level • For unintentional grounds

✓ ✓

6 Calendar MonthsInspect • Measure battery cell internal ohmic

values for each battery cells✓

18 Calendar Months

Verify • Float voltage of battery charger • Battery continuity • Battery terminal connection resistance • Battery intercell or unit-to-unit

connection resistance

Inspect • Cell condition of all battery cells (visible

cells) or measure battery cell internal ohmic values (not visible cells)

• Physical condition of battery rack

✓ ✓

18 Calendar Months

or

6 Calendar Years

Verify • Internal ohmic values or float current

per cell compared to baseline

or Verify • Full or modified capacity (discharge,

load) test on the entire battery bank

✓✓

But every 3 years for the capacity test

40

NERC COMPLIANCY

If, like most companies, you have VRLAs, you need to test your batteries EVERY 6 MONTHS using an

ohmic test (conductance, impedance).

And you have to demonstrate COMPLIANCY…

41

What’s a test procedure like?

42

PSMP Always refer to your

Protection System Maintenance Program

Start Here

43



Start Here

Ticket A technician will typically receive an automated ticket to perform a

maintenance routine

44



Start Here


maintenance routine

Plan Workload A technician will typically receive an automated ticket to perform a

maintenance routine

45



Start Here


maintenance routine


maintenance routine

Execute Routine A technician will typically receive an

automated ticket to perform a maintenance routine

46

ROUTINE EXECUTION1

- Use the turkey baster to measure the specific gravity of each cell. There’s a big drawback: you need protective gear.

Test specific gravity of each wet cells’ jars

47

SPECIFIC GRAVITY

48

ROUTINE EXECUTION1

2 Visual inspection of each battery jar and battery rack



- Get a flashlight and look for bottom jar’s sediment, warping of place, cracked or broken plates, seeping of acid around the posts, acid leakage at the bottom of the rack, installation issues, electrolyte level for wet cells…

49

ROUTINE EXECUTION1





3 Full or modified capacity test (aka discharge/load)- Get a load bank & volt meter. As the battery is discharging, you want to take voltage reading of each jar, and at the power

plant level, at specific time intervals. Your goal is to measure the time it will take to reach your end-voltage threshold.

60

CAPACITY TEST

Capacity test procedure for wet cell (VLA) batteries is detailed in IEEE Standard 450-2010.

Capacity test procedure for VRLA batteries is details in IEEE Standard 1188-2005.

61

CAPACITY TESTPROS CONS

• Definitive test to measure your battery reserve time

• Takes time & is relatively expensive • Need a load bank • It’s relatively technical and requires a high

level of security & training, dangerous for people without proper training

• It damages the battery • Two plates push electrons from one plate to the other.

• The more you do it, the more porous the plates become.

62

ROUTINE EXECUTION1







4 Manual or automated ohmic test- Manual using a portable ohmic testing device, or automated using a battery monitoring system.

64

German physicist. High school teacher.

Began his research with a recent invention by Italian Count

Alessandro Volta. Using equipment of his creation, he

determined that there is a: direct proportionality between the

potential difference applied across a conductor and the resultant electric

current.

66

Who is

Georg Simon Ohm (17 March 1789 – 6 July 1854)

67

OHM’S LAW

Ohm's law states that the current through a conductor

between two points is directly proportional to the potential difference across

the two points, and inversely proportional to the

resistance between them.

68

CONDUCTANCEConductance is a measurement of the plate surfaces available for

chemical reaction. High relative conductance is a reliable indication of a healthy battery. Conductance declines as the battery fails.

By applying an electrical voltage of a known frequency and amplitude across the jar and measuring the current that flows in response to it. The conductance (acceptance) is the ratio of the AC test current impressed on the jar that is in-phase with the AC voltage, compared to the AC voltage

producing it.

DC Resistance: Short duration DC load on the cell/unit to measure step change in current and voltage. By dividing the change in voltage by the

change in current, a DC resistance is calculated using Ohm's law.

69

IMPEDANCEPerformed by sending an electrical current of a known frequency and

amplitude, across the cell/unit/block and measuring the AC voltage drop.

Compute the resulting impedance using Ohm's law.

70

CONDUCTANCE VS. IMPEDANCE

Conductance - Decreases with battery age

Impedance - Increases with battery age

71

OHMIC TEST

Ohmic test procedure for wet cell (VLA) batteries is detailed in IEEE Standard 450-2010.

Ohmic test procedure for VRLA batteries is details in IEEE Standard 1188-2005.

72

BATTERY MONITOR

You can automate ohmic testing using a battery monitoring system.

Even if the cost is high (~$2,500 per site), the ability to catch a battery before it fails outpace its cost.

Selection and use of a battery monitoring system is detailed in IEEE Standard 1491-2005.

74

OHMIC TESTPROS CONS

• Non-intrusive • No torquing required as you test the

intercell straps • Quicker & safer than a capacity test • Standardized w/ choice of equipment

out there • Statistically-proven relationship

between ohmic & capacity as tested by discharge

• Good way to test that a new battery is good & creating a baseline

• Rapid way of tracking relative health & identifying jars/batteries that fail prematurely

• General agreement that it is not as definitive as capacity test… • % of negative-pass

• Some statistical anomalies that can cloud results

• Make sure you have the right reference value when analyzing test results

75

ROUTINE EXECUTION1







4 Manual or automated ohmic test- Manual using a portable ohmic testing device, or automated using a battery monitoring system.

76

A 5FT “TEST”…

If you didn’t upgrade your power plant to an intelligent one, you should EQUALIZE your batteries, by making sure float voltage is at the right value for each jar.

77



Start Here


maintenance routine


maintenance routine



Continue Here

Review Tech’s live review & analysis of

test data

78

TEST DATA REVIEW

79



Start Here


maintenance routine


maintenance routine



Continue Here


test data

Close Routine Complete & close PM ticket

80



Start Here


maintenance routine


maintenance routine



Continue Here


test data


Send to Eng. Email test results to engineering

for further analysis & replacement request form filled-out if need be

81



Start Here


maintenance routine


maintenance routine



Continue Here


test data


Send to Eng. Email test results to engineering

for further analysis & replacement request form filled-out if need be

Engineering Takes Over

• Approve/refuse replacement • Crunch test data and load in

central repository • Provide metrics for NERC’s

compliancy

82

83

➡ DISCLAIMER




➡ CONCLUSION


OUTAGE INTERVENTION

Field technicians are the first people to be called/dispatched in case of:

MAIN POWER OUTAGE & POWER EQUIPMENT FAILURE.

84

Alarm to NOC NOC received “Battery On Discharge” & other alarms

Start Here

85

ALARM

There are typically 2 events that will trigger a BOD alarm: AC FAIL & CHARGER FAIL.

86


Start Here

Dispatch NOC will dispatch closest

technician to site

87


Start Here


technician to site

Validate ETA If not done by NOC, a tech will

validate the cause of outage, length & ETA to restore

88


Start Here


technician to site



Validate BRT Based on outage information,

technician will validate the estimated battery reserve time (BRT) on site

89

AS A RULE OF THUMB

Validated ETA - Validated BRT

= Time to get to a site & react/repair

90

VALIDATE BRT

A validated Battery Reserve Time (BRT) is a key metric field technicians need to react in optimal fashion in

case of outage or equipment failure.

• Get there as soon as necessary • Bring the right equipment (generator, generator

connector fitting plug and cable, replacement parts) • Do as few truck rolls as possible • Keep the site up

91

VALIDATE BRT

There are 4 steps needed to validate BRT:

(1) look for the last available LOAD reading (2) look for records of exact battery MODEL installed on site

(3) find battery model’s DATASHEET with specific capacity measurements at specific end-voltage value

(4) use rule of thumb or PEUKERT to estimate BRT

92

German scientist. In 1897, presented his “law” that

expresses the capacity of a battery in terms of the rate at which it is

discharged. As the rate increases, the battery's

available capacity decreases.

94

Who is Peukert

95

PEUKERT

Manufacturers rate the capacity of a battery with reference to a discharge time. A battery might be rated at 100 Ah when discharged at a rate that will fully discharge the

battery in 20 hours. In this example, the discharge current would be 5 amperes. If the battery is discharged in a shorter

time, with a higher current, the delivered capacity is less.

96

PEUKERT

What you need to know: (1) The law is not a law, it’s a recognized concept.

(2) Peukert determines a “k” factor to apply to a battery technology. For telecom batteries: 1.2.

97

PEUKERT

The law:

98

PEUKERTUse a form!!!

99


Start Here


technician to site





Continue Here

Truck Roll Get all you need and get to the

site

100


Start Here


technician to site





Continue Here


site

Repair Proceed with repair or hook-up

generator

101


Start Here


technician to site





Continue Here


site


generator

Turn On Power Once repaired, or AC is back on,

switch off generator and go back to commercial power

102


Start Here


technician to site





Continue Here


site


generator

Turn On Power Once repaired, or AC is back on,

switch off generator and go back to commercial power

Wait!!!!!

• Make sure you have enough charger capacity to manage: • in-rush current • recharge batteries • carry the DC load • hope the batteries are recharged

before power goes out again… • … all at the same time

103

You’ve just glanced at what a technicians needs to know when it comes to battery backup… And you thought engineers had it hard!?

104

105

➡ DISCLAIMER




➡ CONCLUSION


Engineering teams are typically responsible for two aspects of battery backup: PLANNING/SIZING & REPLACING.

106

107

➡ DISCLAIMER




➡ CONCLUSION


PLANNING

When planning new projects in new or existing sites, engineering needs to RIGHT-SIZE the batteries to be deployed.

Let’s see what that means.

108

Project Issued Once strategic engineering is

done & funding approved

Start Here

109



Start Here

Estimate Load Look for the THEORETICAL DC

load of new equipment

110



Start Here



Challenge That Theoretical loads will always vary, typically they are inflated. Agree on

a maximum load

111



Start Here




a maximum load

Pick Site Type The type of site you pick with have a

huge influence on the battery solution you end up with

112

PICK SITE TYPETwo important aspects to site type:

(1) Available FLOOR SPACE (2) FLOOR LOADING capacity

Large site Small site

Wet cell VRLA

300 lbs/sq.ft 125 lbs/sq.ft

Wet cell VRLA

113



Start Here




a maximum load



Continue Here

Pick Battery Pick the battery solution & model you want in your site

114

PICK BATTERY

There are two battery technologies to pick from: WET CELL (VLA) & VRLA.

115

PICK BATTERYWET CELL (VLA) VRLA

• Proven since… >100 years

• Robust

• Reliable

• Recover well

• Long lifetime

Pros Cons

• Acid!!!

• Lead!!

• Takes space

• Heavy

• Requires “maintenance”

• Produces hydrogen gas during charging

• Needs hydrogen sensor, proper ventilation & equipment in battery room

• Small & portable

• Easier to install

• Can fit anywhere

• Lighter

• Requires no “dangerous” maintenance

• Shorter lifetime

• Smaller value over the long run

• Higher risks of thermal runaway

ConsPros

116

PICK BATTERY

Before picking a battery manufacturer & model, check with your STANDARDS group or TECHNOLOGY ADVISOR, they typically have a few selections they want you to use.

117



Start Here




a maximum load



Continue Here


Find Specs Use the battery manufacturer spec sheet to decide which capacity version you need

118

FIND SPECS

Each battery model has a specific capacity value for specific end-voltage designs. It’s the engineer’s job to

make sure he designs with the right specs.

119

FIND SPECS

Each battery model has a specific NOMINAL CAPACITY rating (in Ah) for specific END-VOLTAGE designs, for a

REQUIRED RESERVE TIME duration. It’s the engineer’s job to make sure he designs with

the right specs.

120

FIND SPECS

121

FIND SPECS

122

FIND SPECS

123



Start Here




a maximum load



Continue Here



Size After picking a model, you now need to decide on the quantity needed to support your load

124

SIZE

You’re almost there. You know which model you want to pick & you know your load. BUT…

125

SIZE

BUT… You need to provision for growth @ battery end of life.

The reality of telecom facilities is that they change. New technologies get added, load will vary. You need to

provision for that from day one.

126

SIZE

Total Projected Load ➗ Nominal Capacity Specs of chosen model

= THE NUMBER OF STRINGS NEEDED OF THAT MODEL

127

SIZE - EXAMPLE

Projected Load (A) 100

Battery Model Specs (Ah)(Enersys SBS170F @ 1.75Vpc @ 8hours)

21.6

Number of Strings Needed 4.6

Note: Some people add an aging factor, like 1.25, to the number of strings needed. You don’t have to, but you can if you feel you have the money or your projected load is not relevant.

128



Start Here




a maximum load



Continue Here



Size After picking a model, you now need to decide on the quantity needed to support your load

Issue Install & Project Request

• Once the project is sent out: • make sure you wait until the project

is completed before updating records, in case of outage

• don’t forget to update records with new battery model & information

• make sure technicians know the ohmic reference value for that model

129

130

➡ DISCLAIMER




➡ CONCLUSION


REPLACING

Based on Murphy’s law, if you have bad batteries, even if you did everything else right, for sure, THAT ONE SITE WILL FAIL.

Replacing a battery is as important as installing a new one.

131

Get Data Get test data from field

technicians at regular intervals

Start Here

132



Start Here

Analyze Data Look for the THEORETICAL DC


133

ANALYZE DATA

The data you’ll receive from field technicians is the only & best metric you have to make a decision to maximize

uptime. It’s important, don’t just file it without looking at it.

134

ANALYZE DATA

Things to look for: (1) Has the load reading changed since last time?

(2) Was the temperature in the site constant at 25°C? (3) Is the float voltage of the jars and strings at the right value?

> Remotely connect to the controller to make sure. < (4) Where are the ohmic test results compared to the reference?

135

OHMIC REFERENCEThe ohmic reference value is a hot topic:

(1) Do not trust the reference value the tech was using. (2) Ideally, have each new string benchmarked at 6 months.

(3) Get the reference from the battery manufacturer. Don’t use it just yet.

(4) Get reference value from Midtronics or other equipment vendors. Use this one if it’s slightly lower than the mfg’s.

(5) Repeat for each of your sites.

136

OHMIC REFERENCEAt which reference value should you REPLACE?

Status Conductance Reference

Impedance Reference

Jars are GOOD when > = 80% < = 70%

Jars are WARNING when < 80% and > = 60% > 70% and < = 90%

Jars are BAD when < 60% > 90%

137



Start Here



No Data = Age For the sites for which you didn’t

get data in the past 24 months, use age as your replacement trigger

138

NO DATA

What do you do if you do not have data? (1) Don’t even try hiding it from NERC.

(2) Use age as your replacement trigger. (3) Send a tech as fast as you can to inspect & measure.

139



Start Here





Replacement List Build a replacement list based on all

the data files you received.

140

REPLACEMENT LIST

What do you do if you do not have data? (1) Don’t even try hiding it from NERC.

(2) Use age as your replacement trigger. (3) Send a tech as fast as you can to inspect & measure.

141



Start Here





Replacement List Build a replacement list based on all

the data files you received.

Continue Here

Recalculate BRT When you received your test results, you also go a new DC load reading

142

RECALCULATE BRT

Ohmic test results can be represented a % of your reference. Can you use that % as a de-rating factor to reduce battery capacity?

143

RECALCULATE BRT

Yes.

144

RECALCULATE BRT

145

RECALCULATE BRT

146

RECALCULATE BRT

When you got your test data back from field operations, you also got an updated DC load reading. If it changed, use the new value to recalculate your

updated Battery Reserve Time (BRT).

147



Start Here





Replacement List Build a replacement list based on SOH

& age

Continue Here


Repeat Don’t forget to repeat this for

each site in your network

148



Start Here






& age

Continue Here




BRT List For any site lacking BRT, create a

list to add battery capacity

149

BRT LIST

Build a list of all the sites that require added batteries to support:

(1) De-rated capacity based on ohmic test results (2) Updated DC load reading

150



Start Here






& age

Continue Here




BRT List For any site lacking BRT, create a

list to add battery capacity

Cross Reference Both Lists to

Create Projects

• You end up with two replacement lists, based on: • Ohmic state-of-health (SOH) & age • De-rated capacity & updated load

• Cross-reference to: • Prioritize replacement needs • Preventing mistakes such as adding

capacity when you should install new model

151

So, who has the toughest job? FIELD TECHNICIANS or ENGINEERS?

152

My point: to maximize battery backup & uptime, you need both teams to work in full cooperation.

It’s a question of COMMUNICATION & CULTURE.

Promoting this should be a high priority…

Everything else is a process.

153

154

➡ DISCLAIMER




➡ CONCLUSION


EXTEND LIFETIME

My conclusion today: what can you do to help extend the lifetime of your batteries?

(4) Top wet cells up in the spring. (3) Never discharge them… hahaha!

(2) 25°C all the time (1) Float them at the right voltage

155

➡ DISCLAIMER




➡ CONCLUSION


We’re at > 150 slides. It’s a little overwhelming.

KEEP CALM AND

HAVE ENOUGH

BACKUP

At Multitel, we’ve developed a tool that’s AUTOMATING most of the processes & steps you’ve seen here today.

FIRM Battery ManagementIt’s the first enterprise software that gives snapshot &

detailed views of your at risk sites when it comes to batteries.

12

4 6

3

5

1. # of routines completed for the network, per region, per technician

2. # of cells failed or failing based on impedance

3. # of cells failed or failing based on conductance

From the Dashboard, you have access to:4. List of sites with problems that need escalation

5. List of routines w/ problem or warnings needing approval

6. List of sites for which technicians did an inventory change requiring approval before being updated in FIRM

160

Through a simple interface, it helps field technicians quickly upload test data.

And crunches it automatically to generate prioritized replacement lists.

Saving engineers valuable time.

Complete battery replacement program.

And automatically updates Battery Reserve Time (BRT).

No matter who you are in the company, no matter the event, no matter where you are: always know the updated reserve time

in each site within the network.

FREEBIE

Come see us at booth #308 to learn more about FIRM Battery Management and we’ll show you the Top 10 Best

& Worst battery models for both Wet Cells & VRLAs.

Data analytics - always know which batteries are best or worst performing within your network, drill to understand more.

Worst 10 models Worst failures rates Worst production years

May 5-‐8, 2015 | Atlanta, GA | www.utctelecom.org

Dakx Turcotte Mobile: +1-418-262-0575

Email: [email protected]

Booth # 903

mailto:[email protected]

UTC Telecom & Technology 2015 - Battery Backup Workshop

Technology

Transcript of UTC Telecom & Technology 2015 - Battery Backup Workshop