Addressing Data Management and IT Infrastructure Challenges

in a SharePoint Environment By Michael Noel

2 Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS Sponsored by Netapp

Introduction: Data

Management with SharePoint

and Its Challengesin recent years, the success of Microsoft’s SharePoint products and technologies has led to an explosion in data management requirements for many organizations. it environments struggle to manage the rapid growth of SharePoint, while at the same time ensure that their exist-ing environment will scale to fit their growing needs.

SharePoint has a few native controls to help manage data, but organizations that don’t implement plans for manag-ing their data growth can become overwhelmed with the sheer volume of data created within a SharePoint environ-ment. in addition, if proper thought is not put into server design, storage considerations, content security, and disas-ter tolerance, small issues soon turn into larger ones.

organizations need a comprehensive approach to SharePoint design that takes into account all the data man-agement challenges a SharePoint environment creates and that gives administrators control over their infrastructure.

this whitepaper addresses the most common types of data management challenges SharePoint administrators face and discusses different solutions organizations have taken to address those challenges. the paper incorporates information obtained from a survey of more than 350 SharePoint administrators, architects, and business deci-sion makers that focused on understanding what the data management challenges are and how they’re currently being addressed.

Addressing Infrastructure Sprawl and Scalabilityin a production environment, there are few simple SharePoint deployments. SharePoint was designed to be scalable and distributed, allowing for data to be easily distributed and allowing the environment to grow. it’s this scalability, however, that can lead to infrastructure sprawl, a challenge that was identified as a key concern by 29 percent of all respondents to the survey.

➔ Contents

Data Management with SharePoint and its Challenges ...............................................2

addressing infrastructure Sprawl and Scalability ................................................2

Understanding the Need for Multiple Farms and Servers ....................................3

Scaling a SharePoint environment ............3

Using Server Virtualization to address Server Sprawl...........................................4

Providing high availability and Disaster tolerance for SharePoint ..........................5

Clustering and Mirroring SharePoint’s Data tier ..................................................5

Providing load Balancing options for the web tier ..................................................6

addressing Service application availability ..............................................6

Understanding Backup and restore requirements for SharePoint ....................7

addressing the Content growth explosion .................................................8

Content Database and Site Collection Structure limitations ................................8

Understanding eBS and rBS BloB Storage options ..................................................9

Providing Storage Solutions for Managing Data ........................................9

Upgrade and Migration Data Management Challenges ...............................................9

Data Management Challenges ...............10

rearchitecting the Data tier ...................10

Sponsored by Netapp Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS 3

Understanding the Need for Multiple Farms and Serversa SharePoint farm is a self-contained set of SharePoint servers that forms the largest logical unit that you can design. each individual farm may be composed of one or several servers that are part of the farm. a single SharePoint server can only be a member of a single farm, with the exception of SQl Server database servers, which can hold databases from multiple farms. there are multiple reasons why organizations may need to deploy multiple farms within their organization, including, but not limited to, the following:

Geographic Restrictions –• a single SharePoint farm does not lend itself well to communicating with other farm mem-bers across slower network links such as waN connections. Farm members need high bandwidth and low latency connec-tions. Subsequently, organizations typi-cally either decide to deploy centralized SharePoint farms and have clients access those farms across the waN, or they de-ploy multiple farms in different locations, contributing to infrastructure sprawl.Security Boundaries –• while there are ways of establishing multi-tenancy within a SharePoint farm, the ultimate security boundary within SharePoint is the farm itself. So, if different sets of administrators need to access a farm, or if some of the farm content is highly sensitive, separate farms may be created for security purposes.Dedicated Services Farm –• Multiple farms can consume Service applications that are hosted on other SharePoint farms. For ex-ample, several farms within an organization can use the search service application, the user profile sync service, and the managed metadata store on a single centralized farm that provides those services to all farms within an organization. this is particularly true with larger distributed organizations.Development and/or Test Farms –• having a place to test service patches, updates, and new features is critical for any production environment, because it greatly reduces the risk associated with updating SharePoint. even the smallest SharePoint environments should ensure that they have test and/or development farms for this reason.

Staging Farms –• Some environments choose to have “staging” farms deployed, which serve as a location for SharePoint content to be reviewed before it’s pub-lished into production. this is particularly the case in web content management sce-narios such as those involving published SharePoint web sites on the internet.Disaster Recovery Farms –• Depending on the disaster recovery option chosen, it may include the creation of a standby Disaster recovery farm in a remote loca-tion to provide for SharePoint functionality in the event of an outage.

For these reasons, and as a general reflection of natural growth within SharePoint, it is very common for administrators to become challenged with the growth and sprawl of a SharePoint environment.

Scaling a SharePoint EnvironmentSharePoint 2010 was designed to be highly scal-able, and to allow organizations to take small, sin-gle-server SharePoint environments and massively scale them to hundreds of thousands of users. it does this by providing the following strategies that can be implemented when the existing environ-ment becomes overcommitted:

Distributing Databases –• SharePoint al-lows for databases to be distributed across multiple SQl Servers, allowing the data tier to scale very easily. Farm databases such as the search databases can reside on their own SQl Server, for example, or data in the content databases can be split among multiple content databases and stored on multiple servers.Splitting Roles onto Dedicated Servers –• a SharePoint farm consists of one or more servers that each have a designated role within that farm, such as the web role and the various Service application roles. Simply by breaking a role off from a server and moving it to a new server, overcom-mitted servers can have their load re-duced, allowing administrators to scale the environment.Adding servers to the farm –• adding addi-tional servers to an existing SharePoint farm has the effect of load balancing the role across the various role servers. adding extra web role servers to a farm, for example,

4 Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS Sponsored by Netapp

allows them to become ad-ditional servers within a load balanced pool, offloading some of the web traffic from the other servers in that pool.

Using Server Virtualization to Address Server SprawlServer virtualization technologies are a powerful weapon in the fight against server sprawl. indeed, today, just over 50 percent of the survey respondents indicated that they’re currently using server virtualization for at least part of their SharePoint environment, and many more are considering deploying it as well. Virtualization has advantages that go beyond the basic tenants of con-solidation and optimization of the hardware; it also gives SharePoint architects the design flexibility to be able to deploy farm servers that are dedicated to a specific role, but without the need to deploy additional physical hardware.

to illustrate, in Figure 1 the SharePoint architect has taken advantage of server virtualization to deploy multiple farms on a small number of sys-tems. in this case, the SharePoint role systems are deployed across two hosts, and the databases are distributed between a pair of physical SQl Servers. the virtualization allows this particular model the flexibility of having multiple farms and multiple dedicated servers running on a much smaller num-ber of physical systems; in this case, four.

this type of design flexibility gives virtualization an edge over traditional design techniques, and allows for better control over infrastructure sprawl. Microsoft supports SharePoint in a virtualized en-vironment, per the guidelines established as part of the Server Virtualization Validation Program (SVVP), outlined at http://support.microsoft.com/kb/897615. as a caveat, while supported, multiple factors must be taken into account before virtualizing SharePoint, including, but not limited to, the following:

Consider Passthrough/RDM Devices –• the best performance is gained on a virtual guest by allocating what Microsoft refers to as Passthrough devices (also known as

raw Device Mapping, or rDM devices) to the individual guests. this is a concept of taking a physical disk or NiC and directly mapping it to the virtual guest, rather than using a virtualized disk or NiC. Consider the advantages and disadvantages of this approach, however, because using Passthrough devices limits the ability to use virtual host failover solutions.Scale out Individual Tiers –• an equivalent virtual machine will not get the same performance as a physical one, and you may need to scale out an individual tier more quickly to get better performance. Fortunately, virtualization allows you to do this quickly and easily simply be deploying more virtual guests to offload capacity from other farm members.Don’t Push the Limits –• Using memory and process overcommit technologies is one of the biggest source of performance issues in a SharePoint farm. Be very cau-tious when using them. at the same time, highly consider dedicated virtual hosts for your SharePoint environment as well.Closely Watch Performance –• Disk i/o, processor, and memory must be closely watched to ensure that the virtual guests are not being overwhelmed. this involves collecting performance counters from the hosts and guests and watching for greater than normal disk read/write times (higher

Figure 1: Sample Server Virtualization Model

Sponsored by Netapp Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS 5

than 15ms), high processor utilization (higher than 90 percent), high network bandwidth (higher than 60 percent), and low memory conditions. if these exist, consider scaling out the tier of servers or add performance with better storage or passthrough/rDM devices. Consider Quick Provisioning Options – •Virtualization management software allows for the creation of server templates that you can use to quickly add SharePoint serv-ers into a farm through a scripted process. Consider creating SharePoint template serv-ers that already have SharePoint installed, but not configured. after provisioning these servers, they can be quickly added into an existing farm or be used to set up a new farm using PowerShell provisioning techniques.

Providing High Availability and Disaster Tolerance for SharePointin the survey of SharePoint admins and decision makers, more than 74 percent of respondents indicated that SharePoint is a mission-critical or very impor-tant tool for their organization. SharePoint has in many cases become as important to organi-zations as email or phones, and is considered mission critical. it’s subsequently critical for most organizations to ensure that the environment is made both highly available and disaster tolerant.

Clustering and Mirroring SharePoint’s Data Tierone of the most visible and criti-cal tiers in a SharePoint architec-ture is the data tier. indeed, all the SharePoint content is stored in this tier; recovering it is the most important requirement for disaster recovery of SharePoint content.

SQl Server, which runs the SharePoint data tier, includes multiple options that provide for a combination of disaster toler-ance and high availability. these include the following:

Shared Storage Clustering – • the most traditional high availability option for SQl servers involves the creation of a shared storage cluster: multiple nodes have ac-cess to the physical storage where the databases are stored and can take over the processing of those databases in the event of an outage of one or more of the nodes. Clustering is available in either Standard or enterprise editions of SQl Server (though the standard edition only supports a two-node cluster.) Some third-party solutions also provide for geo-clustering capabilities, which allow for nodes of a cluster to be located across a waN.SQL Database Mirroring –• SQl Database Mirroring is newer than clustering, having been introduced with SQl Server 2005 and improved in SQl Server 2008 and SQl Server 2008 r2. Mirroring com-bines both high availability and disaster recovery options into a single solution. it works by creating a mirrored copy of the SharePoint databases on a completely

Figure 2: SQL Server High Availability Mirroring for SharePoint

6 Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS Sponsored by Netapp

different SharePoint server. when used together with a third “witness server,” such as Figure 2 shows, Database Mirroring allows for the SharePoint databases to be able to failover to the mirror server. Note, however, that asynchronous mirroring (typically used in waN failover scenarios) can’t mirror certain system databases such as the config DB; only synchronous mir-roring is supported in these scenarios.

Providing Load Balancing Options for the Web Tierthe web tier of SharePoint, run by windows Server internet information Services (iiS) running on web role servers, can be easily load balanced to pro-vide for high availability. load balancing takes the form of two supported options:

Hardware-Based Load Balancing –• Many organizations take advantage of dedi-cated hardware-based load balancers, which take the web traffic destined for SharePoint and distribute it between multiple SharePoint web servers. the only caveat with using hardware-based load balancers is that the session affinity of the connection must be set to “single” in nor-mal operations, a concept also referred to as “sticky sessions.” Because SharePoint is a session-based application, this means that a single client’s traffic must be routed through a single server, unless that server fails. Not configuring the load balancer this way can lead to an interruption of the user’s session with the server.Windows Network Load Balancing – •Microsoft’s windows Server Network load Balancing (NlB) is a fully supported solu-tion as well. NlB runs as a service on the web servers, and is used to balance the traffic sent to SharePoint. windows Server has two types of NlB:, Unicast, which requires dedicated network cards for the heartbeat traffic, and multicast, which requires special router configuration.

Addressing Service Application AvailabilitySharePoint is divided into three tiers, which Figure 3 shows. this includes the user-facing web tier, the data tier, where all of the information is stored,

and a third tier. the third tier, new in SharePoint 2010, is the tier where the various SharePoint 2010 Service applications operate.

the Service application tier is an extensible layer comprised of multiple Service applications that perform various workloads throughout a farm. Service applications include common services such as search, but also include other more com-plex functionality such as the Managed Metadata Service, which provides a location for centralized configuration of metadata and content types, en-suring consistency across the types of data within an organization.

the number and type of service applications provided varies depending on the version of SharePoint utilized. table 1 illustrates the native SharePoint 2010 Service applications, what they’re used for, and which versions of SharePoint they’re included in. Because the Service application tier is extensible, third-party Service applications can also be written—and Microsoft may expand this list in the future.

typically, addressing high availability within Service applications involves simply adding an ad-ditional server to a farm and configuring that farm to run the Service application as well. Because most service applications store their data within custom Service application databases, this makes it easier to allow for multiple servers at the Service application tier to exist, offloading some of the workload and providing for high availability in the event of an outage of one of the servers.

Figure 3: SharePoint Architecture Layers

Sponsored by Netapp Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS 7

Some Service applications, such as Search, require more complex customization, but can still be con-figured to be highly available. indeed, SharePoint 2010 now allows for redundant search indexes to be created, something that wasn’t possible with SharePoint 2010.

Understanding Backup and Restore Requirements for SharePointSharePoint 2010 is extremely capable and scal-able, but those capabilities come with a price in complexity. Understanding all the moving parts in the various tiers of SharePoint is a complex task, and is a source of significant anxiety among

SharePoint administrators, with data protection issues being a major concern for 33 percent of the survey respondents.

Microsoft provides for several native backup and restore options for SharePoint 2010, including the following:

Backup and Restore from the SharePoint •Central Administration Tool – this is the most obvious method of backing up SharePoint, directly from the web-based Central admin tool. New features in SharePoint 2010 include status bars that

Table 1: Service Application Matrix

Service Application Description SharePoint Foundation 2010

SharePoint Server 2010 Standard

SharePoint Server 2010 Enterprise

Access Serviceslets users view, edit, and interact with access 2010 databases in a web browser.

X

Business Data Connectivity service

gives access to line-of-business data systems. X X X

Excel Services Applicationlets users view and interact withexcel 2010 files in a web browser.

X

Managed Metadata serviceManages taxonomy hierarchies, keywords and social tagging infrastructure, and publish con-tent types across site collections.

X X

PerformancePoint Service Application

Provides the capabilities of PerformancePoint.

Search serviceCrawls content, produces index partitions, and serves search queries.

X X

Secure Store serviceProvides single sign-on authentication to access multiple applications or services.

X X

State serviceProvides temporary storage of user session data for SharePoint Server components.

X X

Usage and Health Data Collection service

Collects farm wide usage and health data, and provides the ability to view various usage and health reports.

X X X

User Profile serviceadds support for My Site web sites, profile pages, social tagging and other social comput-ing features.

X X

Visio Graphics Servicelets users view and refresh published Visio 2010 diagrams in a web browser.

X

Web Analytics service Provides web service interfaces. X X

Word Automation Services Performs automated bulk document conversions.

X X

Microsoft SharePoint Foundation Subscription Settings Service

Provides multi-tenant functionality for ser-vice applications. tracks subscription iDs and settings for services that are deployed in partitioned mode. Deployed through windows PowerShell only.

X X X

8 Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS Sponsored by Netapp

indicate the progress of the backup and the ability to get more granular, backing up at the Site Collection level.PowerShell Scripted Backup Processes –• SharePoint backups can now be automated using PowerShell, which kicks off the same process as the SharePoint Central admin tool does, though from a command-line. it provides for the same level of granularity.SQL Server Database Backups using •SQL Server Maintenance Plans – the most common native backup approach to SharePoint involves backing up the SharePoint databases using a SQl Server Maintenance Plan. however, this ap-proach doesn’t back up the search index, or any custom configuration made to individual SharePoint servers.

the most important factor to understand when planning for SharePoint backup and restore is that the native tools do not fully address all the concerns most organizations have in terms of SharePoint backup and restore. For example, no built-in tools provide for individual item-level restore from SharePoint backups. Backup and restore times, as well as the amount of storage required for multiple backups, can also be greatly optimized. Most enterprise organizations look to third-party solutions for help in this area.

Addressing the Content Growth Explosionthe rapid growth of content within SharePoint envi-ronments is a significant area of concern for many organizations. in the survey, 29 percent of respon-dents indicated that rapidly growing server and

Figure 4: Sample Best Practice Content Database and Site Collection Layout

Sponsored by Netapp Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS 9

storage requirements presented a significant chal-lenge. in addition, more than 86 percent indicated that refreshing their server and storage infrastructure was important. tools and methods to deal with this explosion in content growth are required.

Content Database and Site Collection Structure Limitationshow SharePoint is logically structured con-tributes to challenges around data manage-ment. Specifically, the fact that SharePoint Site Collections can only exist in a single content database can severely restrict the scalability of the data tier, particularly for those organizations that don’t install multiple site collections and content databases when initially creating their SharePoint farm.

ideally, site collections and content databases should be deployed so that they’re distributed, as Figure 4 shows. in this model, the SharePoint environment is used for document management, and each department within the organization is given its own Site Collection, each installed into a separate content database. with this approach, the data is distributed across the environment from the beginning, and the total size of indi-vidual content databases can be brought under control. as a general guideline, Microsoft recom-mends not allowing individual content databases to grow larger than 200gB. Using this strategy can help avoid individual databases from ap-proaching this size.

Understanding EBS and RBS BLOB Storage OptionsFor larger organizations or those with sig-nificantly higher data storage requirements, Microsoft provides a supported option within the SharePoint data tier that allows for documents stored in SharePoint to be physically stored out-side the content database. this concept works by taking the Binary large oBjects (BloBs), which include documents, photos, and anything other than the metadata and properties of the docu-ment, and storing it in a separate storage loca-tion. Because 80 percent to 90 percent of the space taken up in a SharePoint content database is comprised of BloBs, this can have the effect of drastically reducing the size of the content databases.

the original, SharePoint-specific BloB storage option was the external BloB Storage (eBS) op-tion. eBS was created for SharePoint 2007, and is still supported in SharePoint 2010. Microsoft has indicated, however, that eBS won’t be supported in the next version of SharePoint.

the more recent BloB storage option is a SQl-specific technology known as remote BloB Storage (rBS.) rBS operates at the SQl layer, so SharePoint is not even aware that the content is separated. this has performance enhancements, but the one limiting factor of rBS over eBS is that it must be enabled for all BloBs in a single data-base, whereas eBS can be more granular.

when using eBS or rBS, you must take into account factors such as how it changes backup and restore requirements, and the fact that it changes the way that other SQl features such as database mirroring and SQl transparent Data encryption (tDe) operate. that said, proper use of an rBS/eBS solution, which typically takes the form of a third-party option that uses the aPi, can reduce the overall data manage-ment challenges with SharePoint 2010.

Providing Storage Solutions for Managing Datathe majority of SharePoint performance issues are directly related to issues at the data tier, so it’s incredibly important to ensure that the data tier is properly designed to allow for high levels of disk i/o. typically, this means housing the SharePoint databases on a fast Storage area Network (SaN) or equivalent storage infrastructure. indeed, 74 percent of participants in the survey indicated that they are using SaN disk with their SharePoint environment.

For planning SharePoint disk i/o requirements, Microsoft gives the equation of 0.75 ioPS per gB of SharePoint content, although this can vary depend-ing on the nature of the SharePoint environment, the number of users, and other factors. work directly with your storage vendor to implement the proper amount of disk performance for SharePoint and to determine the appropriate storage architecture. take into account the different workloads your business will be implementing through SharePoint’s feature set (eg., workflow, Document automation, eCM). the various workloads will help drive the discus-sion around anticipated storage requirements for the

10 Data MaNageMeNt with SharePoiNt aND itS ChalleNgeS Sponsored by Netapp

business in terms of amount, growth, and data type for performance and cost optimization. also consider availability requirements for the various content and business processes being relied upon SharePoint.

Upgrade and Migration Data Management Challengeseven a year after SharePoint 2010 was released, only 22 percent of survey respondents indicated that they’re already using SharePoint 2010 in production. however, 38 percent indicated that they plan on migrating to this new version within the next six months. this makes upgrade and migration a major topic of concern for many organizations. indeed, because the architec-ture of the back-end SharePoint environment has changed so much, this poses challenges for these organizations that must plan their migra-tion strategy carefully to avoid running into the types of data management challenges listed in this document.

Data Management Challengesthe biggest issue from a data management perspective that most organizations face when migrating to SharePoint 2010 is how to archi-tect the data tier to allow for both environments to co-exist during the migration process. in addition, changes to the taxonomy between SharePoint 2007 and SharePoint 2010 also create challenges when migrating content between sys-tems. Microsoft provides two migration options natively, an in-place upgrade and a separate ap-proach known as the “database attach” migration process.

the in-place upgrade has significant disadvan-tages, namely risk and inflexibility, so it’s not com-monly used. this means that the database attach process, which involves building a brand new SharePoint 2010 environment and then migrating to that environment database by database, ends up being the preferred option. a third option is to rely on third-party migration tools, which also assume the creation of a new parallel SharePoint 2010 environment.

Because the database attach processes and third-party migration tool processes require a separate newly created SharePoint farm, organizations must

plan on new storage for the SharePoint 2010 farm, because the existing legacy SharePoint storage is still utilized throughout the process. if there is a large amount of data, this can lead to some serious challenges in balancing the storage requirements during the migration.

Rearchitecting the Data Tier Because of these migration challenges, organiza-tions often restructure and rearchitect their storage environment during the migration process. this has the side effect of allowing for “greenfield” deployment for the new farm, giving administra-tors the ability to design around best practices and to build for future performance and scalability. it also provides for an opportunity to implement the other best practices discussed, such as SQl Server Database mirroring, geo-clustering, or third-party storage solutions that provide for better ha and Dr of the new target environment.

as evident in the survey results and as discussed in this white paper, SharePoint administrators and architects face significant data management chal-lenges that can lead to performance issues, infra-structure sprawl, and other serious problems. Putting the proper thought into architecture of the three tiers of SharePoint 2010, including web, Service application, and data, can help minimize the effects of these challenges—and provide for a robust and high performance SharePoint 2010 environment that can scale to the needs of your organization.

Michael Noel is an MVP for SharePoint Server and an MCSe+i. he has been involved in the com-puter industry for nearly two decades, and has significant real-world experience helping organi-zations realize business value from information technology. Currently a partner at Convergent Computing in the San Francisco Bay area, Michael’s writings and worldwide public speak-ing experience leverage his real-world expertise designing, deploying, and administering it infra-structure for his clients.

to learn more about Netapp solutions for SharePoint Server environments, go tohttp://www.netapp.com/us/solutions/applications/microsoft-sharepoint-server/