1

VIDEO SURVEILLANCE WITH THE STORSIMPLE 8000 SERIES

Case Study of deployment with Microsoft Global Security – May 2015

2

Copyright This document is provided "as-is." Information and views expressed in this document, including URL and other Internet website references, may change without notice. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy, use and modify this document for your internal, reference purposes. © 2015 Microsoft Corporation. All rights reserved. Microsoft, Microsoft Azure, StorSimple, Active Directory, Hyper-V, Internet Explorer, Silverlight, SQL Server, Windows, Windows PowerShell, and Windows Server are trademarks of the Microsoft group of companies. All other trademarks are the property of their respective owners.

3

Contents VIDEO SURVEILLANCE WITH THE STORSIMPLE 8000 SERIES ........................................................................ 1

Copyright ......................................................................................................................................... 2

Introduction .................................................................................................................................... 4

Business Needs ............................................................................................................................... 4

Solution ........................................................................................................................................... 5

Overview of Storage Environment prior to StorSimple .............................................................. 5

Overview of Storage Environment with StorSimple ................................................................... 5

Initial Installation: Network Architecture ................................................................................... 6

Camera Workload Analysis ......................................................................................................... 7

Genetec Application Architecture ............................................................................................... 9

Deployment Phases ................................................................................................................... 10

Test Results ............................................................................................................................... 12

Benefits ......................................................................................................................................... 14

Appendix: Common Video Surveillance Terminology .................................................................. 15

4

Introduction Storage requirements for video surveillance data can be difficult to predict – motion detection based cameras with varying bitrates, different compression techniques and camera malfunctions can cause storage requirements to fluctuate wildly. In addition, data retention requirements for video data can cause storage requirements to be high. This typically leads to over-allocation but under-utilization of on-premises storage hardware – 40% utilization rates can be common. Microsoft Azure StorSimple hybrid cloud storage allowed us to use Azure to extend the capacity of our on-premises storage hardware infrastructure without needing to incur additional costs for power, cooling and support for on-premises hardware. Being able to take regular backups of the data to Azure was an additional bonus. – Brian Tuskan, Senior Director of Technology and Investigations, Microsoft Global Security

This paper details the deployment of Microsoft Azure StorSimple 8000 Series for video

surveillance with a customer internal to Microsoft called Microsoft Global Security. The paper

will discuss the following topics:

What the business needs were for this customer

How we sized the video workload

What the storage environment was before and after deploying StorSimple

What results were seen

Readers not familiar with video surveillance workloads should read the Appendix section for

Common Video Surveillance Terminology.

This paper is for Microsoft internal use only and should not be shared externally at this time.

Business Needs Microsoft Global Security (MSGS) provides video surveillance monitoring for over 700 Microsoft corporate buildings worldwide. They use video surveillance software from Genetec. MSGS was interested in a storage solution that could help them address the challenges below.

High on-premises hardware footprint The size of data generated from video surveillance cameras can vary a lot, making it hard to estimate how much storage capacity to plan for. This causes storage administrators to over-allocate storage capacities. Over-allocation of traditional on-premises storage hardware leads to high hardware maintenance costs (power, cooling and hardware support). Additionally, when a building is decommissioned and video surveillance needs to be stopped, the hardware that holds the surveillance footage has to be physically retained to preserve the data.

5

Poor data protection for video surveillance data Without easy mechanisms to protect video surveillance data, storage administrators have to rely only on the built-in redundancy in traditional storage arrays. If data is lost due to disk failure, security professionals do not have a good way to investigate incidents. For one of the Microsoft corporate campuses, MSGS had kicked off a consolidation effort to move Network Video Recorders (NVRs) and associated direct attached storage from each building that was under surveillance to a centralized location with fewer NVRs running against shared storage as part of a cost savings and go-green initiative. As part of this consolidation exercise, MSGS took the opportunity to replace the shared storage with StorSimple 8000 Series.

Solution Overview of Storage Environment prior to StorSimple

There are a total of 8 NVRs that support a total of 413 cameras.

Each NVR is a Dell PowerEdge R620 server with 16-cores, 12GB RAM and is running

Windows Server 2008 R2 Enterprise x64 Edition. Shared storage for all 8 NVRs is provided

by 3 Dell EqualLogic Arrays (Dell Equalogic PS6500E x3 w/ 24x 3TB NL-SAS Drives).

Each NVR runs the Genetec Omnicast (Security Center 5.2.1164.1056) software.

No backups were being taken of the video data.

Overview of Storage Environment with StorSimple

MSGS moved a total of 212 production cameras across 5 NVRs that were associated with

the EqualLogic arrays to a StorSimple Hybrid Cloud Storage array Model 8100 running

software version 6.3.9600.17215 (July 2014 release).

Each camera produced a single stream and was configured to record 100% of the time

(however it uses a motion-compensation codec H.264).

Video feeds were protected using nightly Cloud Snapshots. Data needed to be retained for

30 days, however the NVR and storage environment was not configured to delete data after

this period.

6

Initial Installation: Network Architecture

Data0 (1GbE) – Cloud enabled (no iSCSI) – connected to external (Corporate) network

Data1 (1GbE) – Cloud enabled (no iSCSI) – connected to external (Corporate) network

Data2 (10GbE) - Disabled

Data3 (10GbE) - Disabled

Data4 (1GbE) – iSCSI enabled (no Cloud) – connected to Camera network

Data5 (1GbE) – iSCSI enabled (no Cloud) – connected to Camera network

All the cameras are on a separate network from the corporate network and connect to the NVRs via a separate switch. Video is accessed via a Genetec client application running on laptops/desktops that are connected to this client or management network. Note: Multipath IO (MPIO) was not configured on the NVR for this deployment. It is however always a best practice to configure StorSimple with MPIO. Additionally, the 10GbE Data2 and Data3 ports should be configured for iSCSI access if the switch is 10GbE capable. In this particular deployment the switch was not 10GbE capable. The corporate network switch used was capable of a bandwidth of 1Gb/s.

7

Camera Workload Analysis This section details the video camera load assigned to the StorSimple solution, the approach used to size the workload for storage/throughput requirements and the resulting sizing data. The cameras were configured to capture video 100% of the time. However, the H.264 codec employs a motion compensation based video capture algorithm (see details below) which makes it essentially a motion-detection workload, which is harder to size. Summarized below are 2 load sizes – the Low size represents a point-in-time value of the bitrate seen against each NVR while the High size represents the maximum bitrate that was programmed into each camera. It is expected that the bitrate seen in production will lie somewhere in the middle of these 2 extremes. Description of the camera load

Load Size Low (Real-time bitrates)

High (Max-Programmed bitrate for cameras)

Camera Manufacturer

Sony

Camera Models

Various (SNC-DH140, SNC-DH220, SNC-DH280)

Codec

H.264

Resolution

Various (1280x720, 1280x1024, 1920x1080)

Frame Rate (Frames per second)

8

Key-Frames

2

NVR1 Per Camera Bit Rate (kbps)*

252 1745

NVR1 Camera Count*

20

NVR2 Per Camera Bit Rate (kbps)*

400 1745

NVR2 Camera Count*

50

NVR3 Per Camera Bit Rate (kbps)*

480 1745

NVR3 Camera Count*

61

NVR4 Per Camera Bit Rate (kbps)*

692 1745

NVR4 Camera Count*

62

NVR5 Per Camera Bit Rate (kbps)*

241 1745

NVR5 Camera Count*

19

Total Camera Count

212

Data Retention Period (days)* 30

8

*Fields required to compute sizing. Sizing Results

Load Size Low (Real-time bitrates)

High (Max-Programmed bitrate for cameras)

Total bit rate (Mbps) across all cameras 2

= (NVR1 Per Camera Bit Rate * Camera Count) + (NVR2 Per Camera Bit Rate * Camera Count) + (NVR3 Per Camera Bit Rate * Camera Count) + (NVR4 Per Camera Bit Rate * Camera Count) + (NVR5 Per Camera Bit Rate * Camera Count) / 1024

99.42 361.36

Storage per day all cameras (GB) = Total bit rate across all cameras * 3600 * 24 / (1024 *8)

1048.54 3811.25

Storage across retention period all cameras (TB) 1

= Storage per day all cameras * Retention Period / 1024

30.72 111.66

Approx. days of storage for all cameras accommodated into local working set of StorSimple 8100 appliance (days) = Local Capacity of 8100 (10.8TB * 1024) / Storage per day all cameras

10.5 2.9

1 Capacity StorSimple 8100 supports a total capacity of 200TB with a local capacity of up to 10.8 TB. If we assume that the actual storage generated will be between the Low (30.72 TB) and High (111.66 TB) values, and we assume that data that is 31 days or older will be deleted, the 8100 can accommodate this load.

2 Device to Cloud Throughput Given that the total storage requirement is going to exceed the local capacity of the 8100 appliance, this workload will be generating large amounts of data that will need to tier to Azure and may be bottlenecked on the maximum write throughput from device to cloud. StorSimple supports a max of 41 MBps = 328 Mbps, which is higher than the Low value for the bitrate expected (99.42 Mbps) but is lower than the High value for the bitrate expected (361.36 Mbps). Additional Details

Codec: H.264 with 2 key-frames (also called Inter-frames). This is a motion-

compensation-based video compression standard that captures good quality video at

low bitrates. It captures the entire image at key frames but only captures the parts of

9

the image that changed between key frames. In this workload 2 key frames and 6

differential frames were captured in 1 second.

Recording Strategy: Cameras are configured to record continuously, however the codec

is motion aware. See above.

Writes versus Reads: This workload is expected to generate a far higher write

throughput than reads. Only 5-10% of the generated video is expected to be read.

Retention Period: While the retention period should be 30 days, the environment was

not configured to delete data after this time period.

Genetec Application Architecture Genetec Security Desk: Each security professional installs this Genetec client application

on their workstation desktop/laptop to view surveillance video footage. There are

typically 2-3 clients for live monitoring of incoming video feeds. There are 3-6 clients

used typically to pull video for investigations. Video is being pulled from the NVR it is

stored in. The back-end for this application is hosted on a clustered server running on

physical hardware.

Genetec Config Tool: This application runs on the administrator’s work-station and is

used to program cameras, add/remove users from the system and do trouble-shooting.

Middle tier: Genetec requires several other servers (login server, database server) and

some of these run on VMs.

Network Video Recorder (NVR or Archiver): Physical hardware hosts the Genetec

Omnicast (Security Center 5.2.1164.1056) software. Each NVR’s primary storage

requirement is for captured video. Genetec supports a maximum of 64 cameras per

NVR.

Storage Configuration: A volume of type Primary was created in a new volume container

on the StorSimple appliance, for each NVR.

NVR1 – 20TB volume

NVR2 – 10TB volume

NVR3 – 10TB volume

NVR4 – 20TB volume

NVR5 – 10TB volume

Genetec was configured to use these volumes through its UI. Nightly Cloud Snapshots

were enabled for each volume container with a 30 day snapshot retention policy.

10

Video files (.G64 extension) are stored in this volume. A new file is created for each 20

minute interval. The file size can vary from 5MB to 250MB. The directory structure looks as

follows:

I:\

\Date

\Camera ID

[Video file - .G64 extension]

Deployment Phases This deployment was conducted in phases to be able to analyze the load at any given point and make changes if needed. The deployment and testing ran from Oct 15, 2014 to Jan 19, 2015, approximately a 3 month period. Phase 1 (Oct 15, 2014): An existing NVR with a load of 20 cameras was directed to use StorSimple as the storage device. [Total Camera Load: 20] Phase 2 (Oct 17, 2014): The maximum programmable per-camera bitrate was increased to 2048 kbps to increase throughput and storage utilization. [Total Camera Load: 20] Phase 3 (Nov 11, 2014): 2 existing NVRs with a load of 50 and 61 cameras each were configured to use StorSimple as the storage device. [Total Camera Load: 131] Phase 4 (Nov 23, 2014): In order to exceed the local capacity of the appliance quickly and analyze how Genetec handles video that was fetched from Azure, a large existing video dataset of 13TB was copied to the appliance. This triggered video to tier to Azure. Note that this was done for testing only – this is not typical of a customer deployment. [Total Camera Load: 131] Phase 5 (Dec 8, 2014): In order to test how Genetec handled video data that was fetched from Azure, we played back video from the first set of 20 cameras, since that was most likely to have tiered to Azure. [Total Camera Load: 131] Phase 6 (Dec 9, 2014): 2 existing NVRs with a load of 62 and 19 cameras each were configured to use StorSimple as the storage device. [Total Camera Load: 212]

11

Note: Although 30 days was the period for which data needed to be retained, the environment was not configured to delete data after 30 days. In order to delete data older than 30 days and in order for the reduction in data to be reflected in storage cost savings the following would be necessary –

Configure daily cloud snapshots of the data and retain snapshots for 16 days

Configure Genetec to retain data for 15 days

Use Windows Server 2012 R2 on the NVR that has the iSCSI connection to StorSimple, in

order to be able to leverage the UNMAP feature to reflect deletion of data on the file

system to the iSCSI volume.

Note that with the StorSimple 8000 series, even with the above configuration, it is not

guaranteed that the data will be deleted on the 31st day because the StorSimple appliance

garbage collects data lazily.

At each phase, the following metrics were captured and analyzed.

Storage Utilization: Primary, Device, Cloud (TB)

Device Throughput: Initiator to Device writes/reads, Device to Cloud writes/reads (kbps)

In addition, the following aspects were analyzed:

Data reduction due to deduplication/compression

Perceived latency of the Lenel application when video is played back from Azure

12

Test Results At each of the phases listed above, Storage Utilization and Device Throughput metrics were captured. These results are detailed in the sections below.

Storage Utilization

Chart 1 – Storage Utilization

The Sizing Results estimated that the Low estimate of storage required would be 30.72

TB and the High estimate would be 111.66 TB, for a total of 212 cameras after 30 days.

During the deployment, we added 13TB of existing data to increase the local capacity

utilization and quickly tier to Azure. We also did not configure the data to be deleted

after 30 days. The maximum primary storage utilization for this workload was 66.4 TB

which was slightly higher than the Low estimate (if you add the 13 TB of data that was

added and account for the fact that data was not being deleted after 30 days) but

nowhere close to the High estimate.

0.000

0.2

3.4

15.0

30.8

37.3

66.4

0.002

0.13.4

10.7 10.79.2

10.7

0.000 0.1

3.1

13.5

30.0

49.0

70.2

0

10

20

30

40

50

60

70

80

Added 20cameras (Oct

15, 2014)

Increasedbitrate to 2048kbps (Oct 17,

2014)

Added 111cameras - total

131 (Nov 11,2014)

Data tiers toAzure (Nov 23,

2014)

Added 81cameras - total

212 (Dec 9,2014)

Average Max

Sto

rage

Uti

iliza

tio

n (

TB)

Primary Device Cloud

13

Device storage maxed out at 10.7 TB as expected (10.8TB is the total local capacity).

This workload generated approximately an average of ~2TB of primary video data per

day (including the addition of the existing 13TB dataset).

Throughput

Chart 2 – Device Throughput

The estimate from Sizing Results was that the Low/High bitrate across all cameras would

be 99.42/361.36 Mbps.

The highest observed throughput for the Initiator to Device Writes was 15.6 Mbps

(15,974 kbps) – the appliance is capable of 5,760 Mbps (720 MB/s) from SSD and 2,000

Mbps (250 MB/s) from HDD which is much higher than what we saw.

The highest observed throughput was for Device to Cloud writes at 20.9 Mbps (21,412

kbps) – the appliance is capable of 328 Mbps (41 MB/s), again much higher than what

was observed in this workload1.

This chart shows that this workload generates far more write throughput than read

throughput, as expected. Approximately 2% of the data written to the StorSimple

solution was read back.

1 https://msdn.microsoft.com/en-us/library/azure/dn772370.aspx

238

1126

1233411697

12718 12646

4782

15974

0.5

0.02 16 3 3 4 6 410.03940

4475

18777

12524 12687

4659

21412

0.0010.3 24 28 225 234 228

358

0

5000

10000

15000

20000

25000

Added 20cameras (Oct

15, 2014)

Increasedbitrate to2048 kbps

(Oct 17,2014)

Added 111cameras -total 131(Nov 11,

2014)

Data tiers toAzure (Nov23, 2014)

Read videofrom Azure

(Dec 8, 2014)

Added 81cameras -total 212

(Dec 9, 2014)

Average Max

Thro

ugh

pu

t (k

bp

s)

Initiator to Device Writes Initiator to Device Reads Device to Cloud Writes Device to Cloud Reads

14

Data Reduction Video data stored by Genetec on the StorSimple solution was reduced by approximately 0.8% with deduplication and compression.

Perceived Latency The average observed Device to Cloud read latency was ~50ms. When fetching video data from Azure and doing playback, there was a brief delay in loading the video (thumbnails and clips) in the UI. This delay was considered acceptable by the customer – moreover, the delay was only visible for the first video load and went away for subsequent loads (since the content was cached locally on the appliance).

Benefits MSGS’s deployment of this video surveillance workload on the StorSimple solution offered the following benefits:

Higher Storage Capacity with Lower Hardware Footprint MSGS was able to increase their storage capacity while reducing their on-premises storage hardware footprint. The rack-space utilization for storage hardware went from 6U to 2U and this translated into cost savings in power, cooling and hardware support. The on-premises capacity of the StorSimple appliance was utilized at a 100% - keeping the most recent video data on premise for fast access and using Azure to accommodate the older video data.

Reduced risk of data loss MSGS was able to take nightly Cloud Snapshots of their video surveillance data, such that in the

event of a disaster, older video surveillance footage was still preserved and could be accessed by

investigators.

Improved Ability to Access Data In scenarios where older video surveillance data needed to be accessed, MSGS was able to easily

allow investigators to read the data by restoring a Cloud Snapshot onto the Azure Virtual

Appliance. This functionality can also be leveraged in the scenario where video surveillance data

from a decommissioned building needed to be viewed and the storage hardware had been

repurposed.

15

Appendix: Common Video Surveillance Terminology

Camera Bit Rate: Rate at which data is generated by the camera every second. Depends

on the camera resolution, codec, frame rate and other factors. See an example

calculator here: http://www.stardot.com/bandwidth-and-storage-calculator

Camera Frame Rate (or Frames per Second or Frame Frequency): Number of frames per

second at which the camera records.

Camera Frame Size (KB): Size of each frame generated by the camera. Typically depends

on the Codec and Resolution in use.

Camera Resolution

o 640x480 (VGA) – Analog

o 704x480 (4CIF) - Common for analog cameras - Standard resolution

o 1280x720 (720P) – HD

o 1280x960 (1M) - 1-megapixel - High-resolution

o 2048x1536 (3M) - 3-megapixel - Ultra-high resolution

o 1920x1080 (1080P) - Full HD

o 4096x2160 (4K or 4K2K) - Ultra HD

Codec (or Format): Video Compression format used to record and compress video data.

There are 2 common codecs in use today.

o H.264: Also known as MPEG-4 Part 10, Advanced Video Coding (MPEG-4 AVC).

Higher CPU overhead but results in smaller output size.

o MJPEG: Each video frame compressed separately as a JPEG image. Low CPU

overhead but larger output size.

Network Video Recorder (NVR): Network Video Recorder which is a hardware device

that a set of cameras connect to and runs some component of a Video Management

Software solution. It typically has local storage to hold some video streams and can be

either a custom piece of hardware or an off-the-shelf server.

Recording Strategy: how often to record.

o Continuous - most conservative strategy - record all the time.

o Time-based - record only during specific hours e.g. Business Hours o Basic image analysis - record only when motion is detected, etc.

Retention period: Time period for which content needs to be retained.

Video Management System (VMS): Application software that handles recording,

viewing, archiving and management of video streams. Examples include: Genetec,

Milestone Systems, Lenel.