achieving enterprise data performance 2013 database growth survey

Achieving Enterprise Data PerformanceBy Joseph McKendrick, Research Analyst Produced by Unisphere Research, a Division of Information Today, Inc. July 2013
Sponsored by
Expanding Data Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Long-Term Data Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32
ACHIEVING ENTERPRISE DATA PERFORMANCE: 2013 IOUG Database Growth Survey was produced by Unisphere Research and sponsored by Oracle. Unisphere Research is the market research unit of Unisphere Media, a division of Information Today, Inc., publishers of Database Trends and Applications magazine and the 5 Minute Briefing newsletters.
To review abstracts of our past reports, visit www.dbta.com/About_Us#Unisphere. Unisphere Media, 630 Central Avenue, Murray Hill, New Providence, NJ 07974; 908-795-3701.
EXECUTIVE SUMMARY
It’s no secret that today’s organizations are awash with data. Data is streaming into transaction systems, appliances and devices from a wide variety of applications, and new sources including social media. Proponents of Big Data state that data contains veins rich with information for decision makers and the business, and many organizations have made it a priority to capture and use this data. However, what many organizations are also discovering is that managing and storing this all this data has a cost. While there is a drive across the industry to introduce new and more digitally compact forms of data storage, as well as cloud storage, these solutions do not get to the heart of the problem for enterprises—data needs to be managed more effectively, and tied closer to the business, from the start.
This paper summarizes the findings from a survey of 322 data managers and professionals who are members of the Independent Oracle Users Group (IOUG). The survey was underwritten by Oracle Corporation and conducted by Unisphere Research, a division of Information Today, Inc.
Survey respondents hold a variety of job roles and represent a wide range of organization types and sizes and industry verticals. The largest segment of respondents, 51%, holds the title of database administrator followed by that of director or manager. Close to one-third work for very large organizations with more than 10,000 employees. By industry sector, the majority of respondents come from IT service providers, educational institutions, utilities, financial services, healthcare, and manufacturing. (See Figures 44–46 at the end of this report for more detailed demographic information on job titles, company sizes, and industry groups.)
Key highlights and findings from the survey, which explore database growth challenges and solutions, include the following:
What’s keeping data managers up at night? Increases in data variety, concerns about database performance, and the need to control data management costs are the key challenges arising from data growth. To deal with these issues, most respondents are focusing on ramping up database performance and consolidation efforts.
Fueling today’s rapid data growth—in many cases, exceeding 25% a year—is rising business demand at respondents’ organizations. A multiplier adding to this growth is data duplication across organizations for various purposes. In most cases, data is duplicated three or more times.
Another driver of the data explosion is the fact that it’s getting more difficult to dispose of data. Forty percent of respondents retain data well beyond the seven-year legal requirement in order to meet compliance mandates as well as maintain data in the event of litigation. More of this data is kept online for easy access, despite the additional resources and costs incurred.
A large number of companies still attempt to manage data growth through hardware acquisition and provisioning, versus more advanced and efficient approaches such as tiered storage or data lifecycle management. A majority of enterprises rely on tape for backup and archiving. Most are now also seeking more automated approaches to better manage growing data volumes.
On the following pages are the results of this latest examination into today’s pressing data growth concerns, and the most effective solutions.
CHALLENGES OF GROWING DATA ENVIRONMENTS
What’s keeping data managers up at night? Increases in data variety, concerns about database performance, and the need to control data management costs are the challenges arising from data growth. To deal with these issues, most respondents are focusing on ramping up database performance and consolidation efforts.
As data grows, organizations are seeking ways to more effectively manage not only growing volumes of information, but also data in various forms beyond traditional relational data. When asked about their leading challenges for the coming year, two main challenges are emerging—challenges that may be at odds with each other.
First, respondents are concerned about their ability to analyze different data types, such as machine-generated data, documents and graphics that have become critical to analytics efforts. “Users have started showing interest in accessing more unstructured data in business apps that use relational databases,” says one respondent. “My biggest question is where to store that data and how to mange its change in a scalable fashion so that the application is still usable in five-plus years.”
In addition, respondents are concerned with their ability to keep the costs of information management under control. As will be discussed further in this report, many of these costs stem from the storage and hardware that most organizations are purchasing and provisioning to handle their growing data stores. (See Figure 1.) Database administrators in the sample are more likely to be focusing on new data types and infrastructure costs than their managers. IT executives show greater concern over helping organizations get to market faster. (See Figure 2.)
In terms of technical challenges to managing growing data environments, performance is top of mind for many respondents, the survey finds. Twenty-seven percent say the ability to increase the performance and availability of their data environments is a key issue. More than one out of five report that the ability to consolidate different computing platforms/applications is the top concern. Close to one-fifth, 17%, also say they need to focus
more effort on infrastructure modernization. Organizations may experience difficulties in cost-effectively managing large stores of unstructured data without improvements and upgrades to their data environments. (See Figure 3.) In terms of attitudes toward technical challenges by job title, database administrators are more concerned with performance and consolidation efforts than managers, who tend to be more concerned with overall application and platform modernization. (See Figure 4.)
Performance is a complex process, and respondents cite multiple aspects to the challenges with which they are dealing. The number-one factor hampering performance in growing data sites is the inability to keep pace with storage requirements, the survey finds. A majority of respondents, 51%, say their data growth is outpacing storage capacity, and this is the most critical performance issue they face. I/O performance issues are the second-most cited obstacle to a fast-moving data environment, cited by 38% of respondents. (See Figure 5.)
The most common approaches to handling these emerging performance issues is to either add more power to databases or to put more hardware in place. A majority of respondents either tune or upgrade underlying databases (63%) or upgrade server hardware/processors (52%). Another 49% report they also upgrade their server hardware or memory. (See Figure 6.)
Still, there are other issues that have been mentioned by respondents. “Security breaches of stored data are at least as much—if not a bigger—concern than live data,” says one. “The prospect of unauthorized acquisition of the data— especially personally identifiable information—given the growing capabilities of Big Data analysts, is one concern that keeps me awake at night.”
5
Figure 1: Greatest Business Challenges for Data Sites Over the Next 12 Months
Analyze more and different kinds of data 31%
Reduce information infrastructure costs 30%
Data security and retention compliance 15%
Get to market faster 12%
Recruit staff with specialized IS/IT skills 10%
Deciding what to keep beyond compliance 1% requirements
0 20 40 60 80 100
Figure 2: Top Business Challenges for Data Sites Over the Next 12 Months—By Job Role
IT Manager/CIO DBA
Reduce infrastructure costs 29% 32%
Get to market faster 20% 13%
6
Figure 3: Greatest Technical Challenges for Data Sites Over the Next 12 Months
Increase performance and availability 27%
Consolidate different computing 21% platforms/applications
Modernize information infrastructure 17%
Provide real-time data 6%
Support increased number of users 3%
Decentralize architecture 2%
0 20 40 60 80 100
Figure 4: Top Technical Challenges for Data Sites Over the Next 12 Months—By Job Role
IT Manager/CIO DBA
Consolidate 15% 27%
Modernize 26% 12%
Figure 5: Issues Affecting Application Performance
Data growth outpacing storage capacity 51%
Applications currently are, or are expected 38% to become, I/O bound
Server virtualization and consolidation 36%
Increasing number of users sharing data 34%
Increasing files sizes associated with 22% unstructured data
Network virtualization 12%
Don't know/unsure 9%
0 20 40 60 80 100(Multiple responses permitted.)
Tune or upgrade underlying databases 63%
Upgrade server hardware/processors 52%
Upgrade server hardware/memory 49%
Upgrade/expand storage systems 45%
Upgrade networking infrastructure 20%
Attempt to compress, or deduplicate, 16% database data using third-party storage hardware
EXPANDING DATA ENVIRONMENTS
Fueling today’s rapid data growth—in many cases, exceeding 25% a year—is rising business demand at respondents’ organizations. A multiplier adding to this growth is data duplication across organizations for various purposes. In most cases, data is duplicated three or more times.
Almost nine-tenths of respondents say they are experiencing year-over-year growth in their data assets. For many, this growth is in double-digit ranges. Forty-one percent report significant growth levels, defined as exceeding 25% a year. Seventeen percent report that the rate of growth has been more than 50%. (See Figure 7.) Respondents within the utilities and telecommunications sector are seeing the fastest data growth, with 43% reporting annual expansions greater than 50%. The services and retail sector follows with 22%. (See Figure 8.)
This data growth is being fueled by a number of factors, but the bottom line is that customer bases and accompanying transactions keep growing. Credit the economy, even though the recovery is progressing slower than many people would like. A majority of respondents, 52%, say growing business demand is creating more data. The push to “compete on analytics” is driving businesses to prep and store data within analytical platforms and tools. Close to half, 48%, also cite the rise of analytical data and associated data warehouse environments as reasons why there is so much data growing within their enterprises. Additional sources of data proliferation include business protection backup, recovery, replication, and redundant mirroring, cited by 37%. More than one-fourth, 34%, say their data is growing due to more reporting data from ERP and other core systems. (See Figure 9.)
In addition, much of this data is stored as historical data intended to service analytical or BI environments. The survey sought to identify how much data is “active,” or predominantly read-write (such as online transactional processing data). One-third say a majority of their data is active data versus 61% reporting that the bulk of their data is “less active” or read-only (such as data warehouse or archival data). (See Figure 10.) Most of this data is still in structured relational databases. For a majority, 56%, most of their data is in this format. (See Figure 11.)
By industry, respondents within the utility/telecommunications sector are most likely to be engaged in Big Data projects, as indicated by 38% of this group. Manufacturing and services/retail follow with 27% within each group. (See Figure 12.) Surprisingly,
there is little differentiation in terms of company size—smaller firms are just as likely to be looking into Big Data initiatives as their corporate counterparts. (See Figure 13.)
However, while there is a great deal of interest and initiative to move data into analytics environments, respondents report they are not ready for Big Data analytics. At present, there are relatively few respondents looking at specific solutions or products that will enable their users to analyze massive volumes of structured and unstructured data—more than 50TB. A total of 14% say they either have solutions or will be implementing such solutions over the coming year. (See Figure 14.)
A great deal of data growth comes from duplication of data across organizations. Respondents were asked how many copies of the data in their production databases are sent out for nonproduction purposes (including development, testing, backup, mirroring, standby, and training). A majority, 59%, say they make three or more copies available. Sixteen percent say more than five copies of their data sets are being distributed across enterprises. (See Figure 15.) Respondents in organizations with more than 1,000 employees are more likely to have this number of data copies circulating—though the largest enterprises in the survey appear to have controls in place. (See Figure 16.)
Respondents were also asked to provide the total amount of disk-resident data at their organizations—taking into account all clones, snapshots, replicas and backups. Thirteen percent say they now manage more than one petabyte of data, while 23% manage data in the hundreds of terabytes. Since the last survey was conducted in 2011, the percentage of companies managing more than 100TB has grown from 27% to 36%. The percentage of companies with more than a petabyte has kicked up from 9% to 13%. (See Figure 17.)
Not surprisingly, the amount of data grows dramatically with organization size. Three-fifths of the largest enterprises in the survey (with more than 10,000 employees) are managing data stores now in the hundreds of terabytes, versus one-fifth of the smaller organizations. (See Figure 18.)
Increased 1% to 10% 14%
11% to 25% 34%
26% to 50% 24%
51% to 100% 10%
Figure 8: Industries Experiencing >50% Annual Data Growth
Utilities/telecommunications 43%
Services/retail 22%
0 20 40 60 80 100
Growing business demand 52%
Data warehouse/BI applications 48%
More reporting data from ERP and other 34% core systems
Compliance information for governments/ 30% standards bodies
Increasing data online 29%
More video/graphics files 20%
Social media content 13%
No plans 42%
Under consideration 24%
Yes, will be implementing solutions during 8% next 12 months
Yes, solutions are already installed 6%
0 20 40 60 80 100
Figure 11: Looking at Big Data Solutions (> 50TB)—By Industry (Solutions already installed or will be implemented within 12 months)
Utilities/telecommunications 38%
Manufacturing 27%
Services/retail 27%
0 20 40 60 80 100
13
Figure 12: Looking at Big Data Solutions (>50TB)—By Company Size (Solutions already installed or will be implemented within 12 months)
1 to 1,000 employees 13%
1,001 to 10,000 employees 17%
10,000+ employees 18%
Figure 13: Percent of “Active” Versus “Less Active” Data
<5% active data 7%
>75% active data 11%
0 20 40 60 80 100
<10% 5%
0 20 40 60 80 100
Figure 15: Number of Copies of Data Made Available for Non-Production Purposes
None 4%
0 20 40 60 80 100
15
Figure 16: Multiple Copies of Data Made Available for Non-Production Purposes—By Company Size
1,001 to 10,000 employees` 21%
Figure 17: Total Amount of Data Managed
2011 Now
>1PB 9% 13%
>10PB – 5%
Figure 18: 100+ Terabytes of Data Managed—By Company Size
1,001 to 10,000 employees 33%
0 20 40 60 80 100
LONG-TERM DATA STORAGE
Another driver of the data explosion is the fact that it’s getting more difficult to dispose of data. Forty percent of respondents retain data well beyond the seven-year legal requirement in order to meet compliance mandates as well as maintain data in the event of litigation. More of this data is kept online for easy access, despite the additional resources and costs incurred.
Data tends to be stored in respondents’ archive systems for significant lengths of time, either because of company policy or compliance mandates. Forty percent say they keep data well beyond the standard timeframe, which is seven years. Twelve percent of respondents, in fact, say they keep their data “forever.” (See Figure 19.)
What are respondents’ primary reasons for holding on to data for this length of time? A majority, 62%, indicate this is to meet federal or state/provincial government compliance mandates. Close to half, 47%, state that they need to hang onto data as part of their corporate policy for potential litigation defense. (See Figure 20.) The motivations for long-term data storage vary by industry, the survey finds—79% of respondents in financial services/insurance and 77% in the utilities and telecommunication sector cite government regulations as their driving reasons, while manufacturers (83%) are compelled to retain data in the event of legal issues that might arise. (See Figure 21.)
A majority of respondents, 61%, say they have increased the proportion of data kept online in the past five years (versus moving to archived tape) to address the requirements of increased information accessibility. Close to one-fourth of respondents say this increase has been “significant.” (See Figure 22.)
“The data growth in my enterprise is driven by the need to store data for extended periods, keeping non-production copies of databases for development, testing, reporting, BI, code deployment, security and integration between systems,” says one respondent. “Strategies that are being considered are cloning rather than copying entire databases for non-production, deduping on backups, keeping archived data on lower cost storage.”
What are the main challenges of keeping data online and more quickly accessible? Most respondents, 72%, say the primary challenge is the fact that maintaining data for extended periods requires more hardware resources. Close to half, 48%, say that management complexity increases, while 43% cite bandwidth issues. (See Figure 23.)
Figure 19: Length of Time Data is Stored
Forever 12%
<1 year 6%
0 20 40 60 80 100
18
Figure 20: Primary Reasons for Holding on to Data for Maximum Length of Time
Federal or state/provincial government 62% compliance mandates
Corporate policy for potential litigation 47% defense
Business purposes (e.g., track/analyze 39% customer history)
Industry guidelines for information storage 24%
0 20 40 60 80 100
Figure 21: Primary Reasons for Holding on to Data for Maximum Length of Time—By Industry
Gov’t mandates Legal Business Industry
IT services/solutions 52% 45% 48% 19%
Education 39% 36% 18% 18%
Manufacturing 67% 83% 58% 42%
Financial services/insurance 79% 42% 32% 16%
Healthcare 71% 47% 24% 18%
Utilities/telecom 77% 62% 46% 54%
Services/retail 50% 56% 44% 12%
19
Figure 22: Increased Proportion of Data Kept Online in Past Five Years?
Yes, significantly 24%
0 20 40 60 80 100(Total does not equal 100% due to rounding.)
Figure 23: Main Challenges of Keeping Data Online and More Quickly Accessible
Requires more hardware resources 72%
Increased management complexity 48%
More security challenges 38%
0 20 40 60 80 100
STRATEGIES FOR MANAGING DATA GROWTH
A large number of companies still attempt to manage data growth through hardware acquisition and provisioning, versus more advanced and efficient approaches such as tiered storage or data lifecycle management. A majority of enterprises still rely on tape for backup and archiving. Most are now also seeking more automated approaches to better manage growing data volumes.
Respondents were asked to identify the one strategy they are undertaking to manage data storage growth today. For close to half, the primary response is to simply throw more disk at the problem. Only about 15% say they have moved to a tiered-storage approach, and even fewer, 7%, rely on database compression. Barely a handful, 5%, say they have a formal information lifecycle management process in place, though this may deliver a cost- effective approach to managing and eventually retiring data, versus storing it forever on active disks. (See Figure 24.)
Public cloud is not yet an option for most organizations, either. Only about one-fourth of respondents say their organizations’ backup data is stored in a public cloud (storage resources managed by a third-party offsite service). (See Figure 25.)
Factors influencing respondents’ primary approaches to managing storage growth include business drivers and unstructured data. Business requirements for data retention top the list, cited by 23% of respondents. In addition, the variety or types of data managed follow at 20%. (See Figure 26.)
Among those respondents using tiered storage as their primary data retention approach, the ability to reduce storage costs was the leading factor in their decision, as indicated by 39%. About one-fifth are also interested in leveraging existing investments to manage their data lifecycle. (See Figure 27.)
For which types of data do respondents think they could save the most space via compression? A majority, 57%, see backup as the most effective approach, while 48% turn to relational table data. (See Figure 28.)
Automation plays a powerful role, the survey finds. Would respondents’ companies benefit from the ability to create policies based on data usage statistics collected by the database to automate data movement or compression and require little or no administrative intervention? A majority, 57%, say they would find this capability useful. (See Figure 29.)
Among respondents that do seek to leverage data usage statistics, the most significant benefit is reduced administrative costs related to their data environments, cited by 45%. Another 44% cite the fact that it provides database administrators with a level of control over their databases’ storage management, while 43% also see that automation would make data movement/ compression easier to manage. (See Figure 30.)
Twenty-six percent of respondents are also considering using Hybrid Columnar Compression (HCC) to improve performance and significantly reduce storage consumption. (See Figure 31.)
Close to one-fourth of respondents, 23%, say a significant amount of their Large Object Data (LOB) stores (defined as exceeding one- quarter of their data) is managed in a database. (See Figure 32.) Among the three-quarters of respondents who store less than 25% of their LOB data in databases, a majority, 52%, say this data is maintained with a storage area network. (See Figure 33.)
While SAN adoption is fairly consistent across the company size ranges, larger organizations are more likely to embrace network-attached storage. (See Figure 34.)
A sizable percentage of respondents’ IT budgets are spent on storage, including hardware, software, services, and management. Sixty-three percent of respondents provided their estimates, and one out of seven report that they spend more than 25% of their IT budgets on storage. One-third spend between 11% and 25% of their budgets, and another one-third spend 6% to 10% of their IT budgets on storage. (See Figure 35.)
Respondents’ storage budgets (including hardware, software, services, management) have also been strong over the past year. More than one-third, 34% report that their budgets have increased, while only 5% have seen cutbacks in this area. The percentage seeing increases is jumping, with 46% predicting budget increases over the coming year. (See Figures 36 and 37.)
Who in respondents’ organizations make decisions about storage allocation/acquisitions related to Oracle Database data? A majority, 51%, say their database administrators are in charge. Forty-three percent say they either have dedicated storage administrators or that CIOs themselves are in charge. (See Figure 38.)
Respondents also report working with a plethora of vendors to manage storage hardware, software, and related services. Twenty- eight percent report they work with three or more vendors. Another 34% work with two storage vendors. Only 16% report having a single vendor they do business with. (See Figure 39.) Having multiple vendors isn’t exclusively a “big company” practice either—the number of vendors is consistent across company size ranges in the survey. (See Figure 40.)

storage area network. Half have network-attached storage or unified storage. Another 41% have direct-attached storage. Only 28%, however, have tiered storage with disk tiers, and 26% have tiered storage with disk and tape. (See Figure 41.)
Two-thirds of respondents indicate that tape is still part of their data backup/archiving storage tiering strategy. (See Figure 42.)
How do respondents manage user demands on accessibility to archived data from tape? The largest number of respondents,
39%, say they only use tape for deep archiving when there is no anticipated need for quick user access or other useful operational purpose—other than when data needs to be retained for legal reasons or historical analysis. Consequently, more than one-third report that data is kept online for longer periods of time before moving it to tape. (See Figure 43.)
Figure 24: Primary Strategies for Managing Data Growth
Add more disk storage 47%
Implement tiered storage 15%
Database level compression 7%
Put limits on how much data users can store 6%
Implement a formal information lifecycle 5% management process
Data deduplication 4%
Incorporate thin provisioning for dev/test/QA 2%
Invest in purpose-built storage appliances 2% and engineered systems
File system compression 1%
0 20 40 60 80 100
22
Figure 25: Percent of Backup Data is Stored in a Public Cloud
None at this time 65%
1% to 5% 13%
6% to 10% 4%
11% to 25% 4%
26% to 50% 2%
Figure 26: Factors Influencing Storage Growth Management
Business requirements for data retention 23%
The variety or type(s) of data managed 20%
The types of applications generating/utilizing 17% the data
Meeting service level agreements related 15% to data access/performance
The time it takes to deploy new storage 11%
The decrease in specialized staff managing 8% storage
The number of users accessing data 6%
0 20 40 60 80 100
Reducing storage costs 39%
Leverage existing investments 19%
Scale out storage 9%
Scale up storage 8%
Figure 28: Where Compression Saves the Most Space
Database backups and exports 57%
Relational table data 48%
Database copies for development and 37% testing
Relational index data 33%
Figure 29: Benefit from Ability to Create Data Usage Statistics?
Yes 57%
No 12%
Other 1%
Figure 30: Most Significant Benefits from Ability to Create Data Usage Statistics
Reduces administrative costs related to 45% data management
Provides DBAs with a level of control over 44% their databases storage management
Automation would make data movement/ 43% compression easier to manage
Frees up administrators for other activities 42%
Improves performance by optimizing 37% placement of hot data vs. cold data
Would allow us to implement capabilities 19% we had never before considered implementing
0 20 40 60 80 100
Yes 26%
No 42%
Figure 32: Proportion of Large Object and File Data Managed in a Database
<10% of data managed in a database 37%
11% to 25% in a database 24%
26% to 50% in a database 8%
>50% in a database 15%
0 20 40 60 80 100
SAN storage 52%
NAS storage 22%
0 20 40 60 80 100
(If less than 25% of LOB data is stored in the database)
Figure 34: Where Large Object Data is Stored—By Company Size
SAN NAS Shared file Local file Hadoop
1 to 1,000 employees 51% 15% 17% 15% 2%
1,001 to 10,000 employees 59% 25% 9% 3% 0%
10,000+ employees 54% 31% 6% 6% 0%
(If less than 25% of LOB data is stored in the database)
<5% 10%
Figure 36: Changes in Storage Budgets Over Past Year
Storage budget declined 5%
0 20 40 60 80 100
Storage budget will decline 4%
No change 19%
Figure 38: Oracle Database Decision-Makers
Database administrators 51%
Storage administrators 43%
CIOs/IT executives 43%
0 20 40 60 80 100
One 16%
Two 34%
Three 16%
Figure 40: Three or More Storage Vendors—By Company Size
1,001 to 10,000 employees 29%
0 20 40 60 80 100
Fibre channel SAN 64%
Tiered storage with disk and tape 26%
Flash/SSD 24%
Figure 42: Tape Part of Your Data Backup/Archiving Storage Tiering Strategy?
Yes 66%
Under consideration 2%
We only use tape for deep archive 39%
Data is kept online for longer periods of time before moving it to tape 35%
We use LTO tape file system for faster access to data stored on tape 16%
Other 10%
Database administrator (DBA) 51%
IT consultant 7%
Analyst/systems analyst 5%
IT operations manager 3%
0 20 40 60 80 100
Figure 45: Respondents’ Company Sizes (Includes all locations, branches, and subsidiaries)
1 to 100 employees 11%
101 to 500 employees 16%
1,001 to 5,000 employees 19%
5,001 to 10,000 employees 13%
>10,000 31%
0 20 40 60 80 100
IT services/consulting/system integration 16%
Education (all levels) 15%
0 20 40 60 80 100

achieving enterprise data performance 2013 database growth survey

Documents

Transcript of achieving enterprise data performance 2013 database growth survey