Dark Data Revelation and its Potential Benefits
-
Upload
promptcloud -
Category
Data & Analytics
-
view
99 -
download
5
Transcript of Dark Data Revelation and its Potential Benefits
What is dark data?
The information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.
- IT Glossary by Gartner
In simple terms, dark data is all that useful data an organization possesses, but doesn’t actually meaningfully use or analyze for the improvement of the business.
The enormous digital universe
2013
2020 44 ZB 37% 27% 10%
4.4 ZB 22% 17% 2%
Total size of digital universe
Data usefulIfanalyzed
Data frommobiledevices
Datafrom Embedded systems
According to IDC (a research firm), up to 90 percent of the digital universe is unstructured data.
Traditional sources of dark data
Server log files
Networking machine data
Point-of-sale feeds
Customer queries recorded in calls, emails, forms
Underused employee data
Meeting notes
Unstructured information arising out of business mails and presentations
Unused data resulting from business research and surveys
Why is it important?
Businesses are heavily invested when it comes to collection of data; however, tangible value can be derived only after companies start to understand their dark data and how it can be applied.
It is also a sensible step for any company which is getting started with big data and building a data warehouse.
In this case, dark data can be a reliable source of historical data.
3 facets of dark data
Existing unstructured data
01Nontraditional unstructured data
02Data in the deep web
03
Existing unstructured data
Many businesses already have large collections of both structured and unstructured data.
Unstructured data such as emails, notes, messages, documents, logs, and notifications (including from IoT devices) are confined to the organization and remain largely unused (due to lack of tools and techniques or their absence in the database).
These data assets could be potentially having valuable insights related to competitors, pricing and consumer behavior.
Nontraditional unstructured data
Data present in the web pages, audio and video files and still images are largely untapped data that can be mined via data extraction solutions, computer vision, advanced pattern recognition, and video and sound analytics.
This can help businesses perform advanced analytics on data present in nontraditional formats to better understand their customers, employees, operations, and markets.
Data present in the deep web
The deep web presents the largest pool of unused information—data curated by academics, consortia, government agencies, communities, and other third-party domains.
Companies can potentially curate competitive intelligence using a type of emerging search tools developed to help users target scientific research, activist data, or even hobbyist threads found in the deep web.
An example of such tool can be Stanford University’s search engine called Hidden Web Exposer that scrapes the deep web for information using a task-specific, human-assisted approach.
Legal and regulatory
issues
If the data stored is covered by legal regulations such as credit card data, exposure of such data could expose companies into financial and legal liabilities.
Intelligence risk
Companies could intentionally or unintentionally disclose proprietary or sensitive data on business operations, products, financial status and business plans.
PR disaster
Companies are considered as protector of data they collect. So, any loss of data, especially sensitive and confidential data, can lead to loss of reputation.
Opportunity costs
If a company avoids analysis and processing of dark data but its competitors do, then its competitors will be in a better position to capture more market share by leveraging the insights from dark data.
Stitch Fix, an online subscription shopping service, uses images from social media and other sources to track emerging fashion trends and evolving customer preferences.
Personalization in retail
Questionnaire
filled by clients
Customer’s Pinterest board
and social media scanned
Data augmentation
Deeper insight of customer’s
style preference
Appropriate clothing
shipped to the customer
A financial services firm wanted to gain insight from its trading terminal data to find correlations between trading patterns and abuses like money laundering and other fraudulent activities.
Most of the data was dark owing to the volume and geographically scattered storage.
After the customer was able to utilize what was previously underutilized, and completed the data prep and analysis process to determine suspect patterns in transactional records, they took that analyzed data and created sophisticated predictive models that can identify activities that indicate the potential for fraud, and take measures to prevent fraud before it occurs.
Fraud detection
Instead of attempting to discover and collect all of the dark data hidden within and outside your organization, work with the business team to find answers for specific business problems.
Getting the right data
Source data from the web to augment your own data with publicly available demographic, location, and statistical information.
Being open to third party data
Data scientists are valuable resources, especially those who have the skills to combine deep modeling and statistical techniques with industry or function-specific insights.
Building data talent
Advanced visualization software can boost business intelligence by repackaging big data into smaller, more meaningful chunks, delivering value to users much faster.
This is crucial since information can be more easily consumed when presented as an infographic, a dashboard, or another type of visual representation.
Utilizing advanced
visualization tools
Future of dark data
Most of the companies in general will learn to better tap into their dark data, it’s the way connected and measurable world is progressing.
The real value will be delivered to those business that would open their data sources in a secure and responsible manner within their business so that the workforce is empowered enough to become problem solvers in own right.
Reach out to PromptCloud — a pioneer in custom, managed and cloud-based web extraction services.
https://www.promptcloud.com | [email protected]
Looking to augment data assets with web data?