Next-Generation ITOA Solution€¦ · of the use cases that ITOA can be utilized for includes:...
Transcript of Next-Generation ITOA Solution€¦ · of the use cases that ITOA can be utilized for includes:...
INNOVATION PLATFORM WHITE PAPER 1
technicacorp.com
The Technica Innovation
Platform White Paper
Series presents
advanced topics that
will drive competitive
advantage for
next-generation IT
over the next
three-to-five years.
NEXT-GENERATION ITOA SOLUTION Next-generation IT Operations Analytics (ITOA) is the practice of using data
science principles, techniques, and algorithms on data that is collected from
underlying IT infrastructure, widening the lens through which to view available
data. This provides actionable insight into data for operational intelligence that
enables more effective decision-making and better service assurance. A sampling
of the use cases that ITOA can be utilized for includes: anomaly detection, pattern
discovery, correlation, and root cause analysis. These outcomes are simply a
matter of choosing the right data set to marry with the appropriate algorithm.
ITOA fuses Big Data concepts and technologies with traditional IT operations tools
to acquire, analyze, and visualize data which provides operators access to the
data they need for operational efficiency. ITOA provides insight into both
business execution and IT operations. To fully appreciate the value proposition
for Next-generation ITOA, it is helpful to understand the data center without
ITOA. We will contrast this as the “Tools-Driven” approach vs. the “Data-
Informed” paradigm present in Technica’s next-generation ITOA Platform.
Tools-Driven
In the effort to guarantee service level agreements and commitments—both
internally and externally—enterprises responded to IT “fires” by purchasing
products in an ad-hoc fashion based on vendor product or technology domain.
Figure 1 portrays examples of these various “point” solutions that perform a
particular job like monitoring applications, event management, network
performance, and security.
Each tool is implemented to focus on operating, administering, and maintaining
a specific IT domain. The individual tools can only function and respond to the
limited IT operations data set for which they have access.
This paper explains a next-generation
ITOA Platform that integrates three
vectors for effective IT operations –
Data-to-Data, People-to-Data, and
People-to-People. Technica’s
next-generation, ITOA Platform
ingests data from any data source
in the data center, persists the data
into a data lake, and then provides
analytics and configurable
visualizations, along with dynamic
and real-time collaboration.
Figure 1 – Tool-Driven Data Silos
INNOVATION PLATFORM WHITE PAPER 2
technicacorp.com
Operators can only react to the information that an individual tool provides, and
must rely on manually interrogating and correlating data across an array of
individual tools. Each of these tools utilizes individual/proprietary access rights,
user interfaces, and nuances that inhibit agile responsiveness. This also degrades
the security posture of the enterprise due to lack of end-to-end visibility under a
single lens.
The complexities of a tool-driven approach magnify operational expenses with
continuous training on the individual tools due to infrequent usage and operator
churn, account management overhead, finger-pointing across administrative
domains that delay service restoration, duplication of effort with multiple
operations teams chasing the same problem, redundant tools, and use of
expensive subject matter experts to fight fires. With increased velocity of IT
solutions being implemented to deliver enterprise services, the net result of a
tool-based approach to IT Operations is siloed functions and fractured data within
the data center that promotes reactive instead of proactive operations and
inhibits service and security assurance for the enterprise.
Distributed data center locations add to the complexity of a tool-driven approach.
While disparate systems can be integrated to move towards a common operating
picture, this entails high capital and operational expenses. Additionally, the
end-solution typically lacks the agility required by dynamic business conditions
due to brittle data sharing interfaces that require tedious and lengthy integration
activities. Inevitably, one of the data formats changes and the entire cycle
must be repeated—often incurring system down-time until resolution.
Moreover, there is no overall service visibility—no overall ability to see services
enterprise-wide.
Data-Informed
Next-generation ITOA enables enterprises to become data-informed – fusing
human insight about the mission with unified data. The data-informed paradigm
enables the enterprise to move away from the islands of inoperable data and gain
visibility into aggregated and integrated data, from all the tools, making the data
accessible, useful, and valuable to operators and business managers.
Data generated by all the tools, devices, and sensors flow into a “data lake,”
where it can be viewed, interrogated, analyzed, and correlated in totality for
actionable insight. Conceptually this results in the situation pictured in Figure 2
(vs. Figure 1).
This data-informed approach takes advantage of analytics and visualizations to
democratize data, and provide a common operating picture for decision makers.
It enables the enterprise to extract meaningful information from the rapidly
growing landscape of disparate data sources and islands of applications.
The purpose of this paper is to explain a next-generation ITOA Platform that
integrates three vectors for effective IT operations – Data-to-Data, People-to-
Data, and People-to-People. Technica’s next-generation, ITOA Platform ingests
Figure 2 – A Unified Data Approach that Facilitates Overall Service Visibility
INNOVATION PLATFORM WHITE PAPER 3
technicacorp.com
data from any data source in the data center, persists the data into a data lake,
and then provides analytics and configurable visualizations, along with dynamic
and real-time collaboration.
The Platform gives an organization an enterprise-wide operating picture of their
environment, using data analytics to provide better performance, availability, and
security analysis. At the same time, executives can make more informed
investments because they have an integrated end-to-end situational awareness
of the health and status of their enterprise. Thus, an enterprise can leverage
Technica’s ITOA Platform to improve efficiencies, agility, and responsiveness—
while at the same time reducing costs.
2. ITOA PLATFORM Over the years, Technica has developed numerous reference architectures,
including a Next-generation Big Data Reference Architecture and a Fog
Computing/Internet of Things (IoT) Reference Architecture. The research and
development of these reference architectures allows the company to investigate
capabilities before looking at individual components.
Then Technica moves to integrating actual components to create a prototype of
the reference architecture. During component selection, the company looks for
best-of-breed and cost-effective products. We define interfaces that ensure
vendor lock-in is prohibited. Additionally, many enterprises have already selected
products. In these cases, Technica uses the reference architecture approach to
extend the existing software assets to meet the required capabilities.
Technica’s ITOA Platform is based on its research and experience with data
analytics reference architectures. It can be visualized in a number of ways
including:
Functional View
The essential functions of the ITOA Platform are Collect, Analyze, and Act.
The Collect, Analyze, Act paradigm drives the functional view of the platform,
as shown in Figure 3.
COLLECT— The Collect function acquires, parses, and integrates data from
data sources. For complete visibility into IT operations at least four different
data sets should be captured:
Machine Data - Any digital data produced by devices, sensors, servers,
and applications, to include IT operations, administration, and
maintenance tools.
Wire Data - All Layer 2 to 7 communications between all systems.
Agent Data - Byte-code instrumentation and call-stack sampling. This
data has typically only been visible to the development and QA teams,
but is crucial for IT Operations moving toward continuous delivery and
the latest DevOps techniques.
Synthetic Data - From synthetic transactions and service checks.
Synthetic data originates outside the application delivery chain through
hosted monitoring solutions or as a part of active service checks, e.g.,
ICMP pings to fully scripted checks, etc.
Figure 3 – Functional View of ITOA Platform
INNOVATION PLATFORM WHITE PAPER 4
technicacorp.com
ANALYZE— This function includes configurable visualizations, the data lake,
and specific analytics including search and more sophisticated algorithms.
Examples of Technica-developed algorithms to extend off-the-shelf
capabilities include anomaly detection and tangential data references.
ACT— These are the outcomes based on the data being appropriately
collected and analyzed. Depending on the use case, alarms can be generated,
trouble tickets can be created, etc. Additionally, collaboration and problem-
solving might be needed to derive a resolution.
Component View
Technica integrated the following components into the reference architecture
(Figure 4) to meet the Collect, Analyze, Act mantra.
Figure 4 – Component View Reference Architecture
Technica elected to use Logstash for Acquisition and Integration of IT operations
data. It collects, parses, and transforms the operations data. Then, Logstash
persists the data to Elasticsearch for Analytics. Elasticsearch serves as the data
lake and provides robust and scalable search capabilities. Finally, Technica
selected SitScape to provide visualization and high-level analytics to end-users.
Importantly, SitScape has extremely robust collaboration features that leverage
enterprise-grade security to enable best-of-breed User Defined Operational
Pictures (UDOPs).
Other products can be utilized—each with different strengths and weaknesses.
For example, Apache Kafka can be used as an alternative to Logstash. If streaming
analytics were desired, in addition to the batch analytics available via the
Elasticsearch data lake, Kafka could play an important part in enabling a Lambda
design pattern within the ITOA Platform. Hadoop or another no-SQL database,
like MongoDB, can be selected for the data lake.
However, for most use cases, Technica believes the aforementioned components
provide the best value.
INNOVATION PLATFORM WHITE PAPER 5
technicacorp.com
The Technica ITOA Platform provides full-service visibility across the enterprise
incorporating the following objectives:
Data consolidation into a data lake that serves as a single integration point
Enhanced decision-making aided by both high-level and deep analytics
Increased operational efficiencies through self-service,
zero-programming access to data
Better collaboration between operations support staff
User-defined Visualizations
Open Application Programming Interfaces (APIs) minimize vendor lock-in
and ensure interfaces are not brittle
Infinitely scalable architecture, using cloud-enabled technologies on
commodity infrastructure
“Zero-programming” is enabled by SitScape’s intuitive interface for integrating
new data sources, performing analytics, and creating UDOPs, shown in Figure 5.
Figure 5 – SitScape Visual Editor for Data Analytics Processing
The SitScape Visual Editor couples a flexible data analytics process design with a
drag-and-drop user interface. This interface enables optimal data management
and operations, while maximizing flexibility and usability.
Scaling the solution for the cloud allows capacity to be added on-demand, only
when needed. This valuable feature is enabled by the products selected for the
ITOA Platform, combined with Technica’s vast experience with cloud-scaling
technologies. Cloud performance and operational availability are assisted with
in-memory cache, distributed cloud-enabling infrastructure techniques and
software, and replication/high-availability mechanisms.
INNOVATION PLATFORM WHITE PAPER 6
technicacorp.com
3. ITOA PLATFORM CORE CAPABILITIES AND BENEFITS The ITOA Platform provides a synergistic and adaptable architecture that meets a
variety of ITOA use cases.
Figure 6 portrays the elements: software-defined computing, collaboration,
analytics and visualization, and correlation synergistically providing a securely
shared data management framework for effective decision making. Open APIs
and cloud-enabling technologies/concepts ensure quick and effective preparation
and transformation of data from disparate sources for analytics.
The core capabilities function harmoniously to provide a system that is secure,
robust, and reliable—with multiple levels of access controls. Additionally, the
ITOA Platform enables agile development and fielding of modular capabilities.
The Platform’s architecture enables IT operations to become a strategic
advantage, providing an integrated, near real-time Common Operational Picture
(COP) of the IT landscape necessary for advanced collaborative data-informed
decision making.
Figure 6 – A Collaborative Data-Driven Framework for Effective Decision Making
Collaboration
With zero-programming, IT operators quickly create UDOPs that can be shared,
allowing advanced collaboration and optimized decision actions through rich and
dynamic messaging capabilities. Advanced role-based or attribute-based security
architectures ensure that only the right people get access to authorized
information.
Rapid Visualization and High-Level Analytics/Correlation
The user selects the appropriate data sets, tracks key metrics, and analyzes
salient trends to quickly create UDOP visualizations. If needed, actions can be
triggered based on user-defined criteria. Thus, end-users are enabled to create
user-defined, dynamic, web-based visualizations of big data sets using an array of
data sources, as exemplified in Figure 7.
INNOVATION PLATFORM WHITE PAPER 7
technicacorp.com
Figure 7 – Example UDOP
The user-defined functionality extends to high-level analytics, deep analytics,
data correlation, numerous drill-down visualizations, and dashboard options—all
without software programming knowledge or skills. The dashboard options
include data-driven, real-time monitoring and alerting.
Technica’s ITOA Platform easily integrates data from any system to ensure
capture of all available data that may be relevant to the decision-making process.
The data acquisition and analytics engine discovers connections and patterns that
are not apparent with traditional approaches.
Often, given the number of point-solutions as pictured in Figure 1, enterprises are
not even aware of all data that is available. In other words, the ITOA Platform’s
data Collect function surfaces data that enterprises may currently be overlooking.
Moreover, the Collect function allows the enterprise to combine information in
ways not possible before—thereby revealing new insights.
The flexible visualization components allow users to realize the full value of
the system features without in-depth knowledge of the underlying technology.
The customizable dashboards present the information visually and intuitively,
allowing users to focus on the data and information—not the technology.
Synergistic Agility
Current tools-driven IT operations systems lack the flexibility and scalability to
incorporate agile development processes and DevOps workflows, especially as
enterprises incorporate containerization and microservices architectures.
Technica’s ITOA Platform delivers an agile solution that increases the velocity
of data acquisition, integration, management, and visualization.
INNOVATION PLATFORM WHITE PAPER 8
technicacorp.com
Self-service software-defined technologies are built on modular off-the-shelf
components that utilize an open architecture for interoperability with web-scale
features.
A key enabler for the adaptability and scalability of the ITOA Platform is the
underlying modular and component-based approach with an intent-based
analytic framework, and its no-programming required design principle.
These architecture and design tenets enable enterprises to rapidly develop new
solutions for incremental capabilities with continuous feedback from prototype
and testing through the deployment and sustainment activities.
Data collection, categorization, aggregation, tabulation, and analytic processing
for analysis and decision making is assured through a robust data pipeline
assurance system that manages the integration of the modular components for
data acquisition, fusion, inspection, analysis, and visualization of data from
disparate and federated sources.
The data Collect function of the ITOA Platform will auto-ingest or manually ingest
data, and provide built-in storage and caching for ingested data, as well as use
external databases, Big Data stores, sensors, and legacy applications and data
sources. The formats include but are not limited to relational, CSV, Excel, log files,
Netflow data, JSON, unformatted text, Microsoft Word, PDF, static web sites,
image, and video. This built-in adaptability allows the enterprise to meet new
requirements without vendor support services.
Other Critical Platform Capabilities
Technica’s ITOA Platform incorporates the following additional capabilities:
DATA SYNCHRONIZATION— Supports the acquisition and ingestion of data
from multiple concurrent and federated sources. It includes capabilities that
ensure guaranteed delivery of data; integration with Disrupted,
Disconnected, Intermittent, and Limited (D-DIL) bandwidth environments
that load data only periodically or may go down for extended periods of
time; and supports backward capability with legacy applications.
DATA MANAGEMENT— Performs scheduled and ad-hoc Extract, Transform,
Load (ETL) functions, query, discovery and visualization of structured,
unstructured and semi-structured data. It supports a flexible and holistic data
architecture that addresses data in motion, data stores, data items and
mapping of data artifacts.
SMART DATA DISCOVERY— Supports smart data discovery through its
machine learning capability that automatically models the behavior of
ingested data, and identifies trends, periodicity, and issues in real time to
streamline root cause analysis. This increases the effectiveness of data
management and reduces the burden of configuring and manipulating tools
by automatically learning a predictive model for the distribution of feature
values at a given time, based on the historical values seen to date. This
predictive model computes the probability of current behavior given
historical behavior.
RESILIENCY— Supports high availability by design. The data architecture is
based on sharding, providing 100% data availability.
INNOVATION PLATFORM WHITE PAPER 9
technicacorp.com
DYNAMIC DATA MODEL— Supports the automatic detection of data field
types paired with the ability to dynamically add fields to a given index.
A user could start saving new information with each record without having
to change or define new schemas. This flexibility provides a means to
get started quickly and to iteratively improve the schema as needs
become clearer.
INTEROPERABILITY— Provides a REST API for integration with external
systems and applications, with low-level ingest plugins that support TCP
and UDP messages allowing for ease of ingest for most network protocols
that other tools produce and consume.
ROBUST SECURITY— Provides multiple layers of security with robust access
controls to compartmentalize data. The layered security provides protections
for data in transit and data at rest, down to the data field level. The system
controls who can add or delete a document in an index, who can access
sensitive documents, and who can access individual fields in documents.
The auditing feature maintains a complete record of all system and user
activity, with an alerting feature that can be used to proactively notify
appropriate personnel of changes or approaching thresholds which are
relevant to security compliance.
4. TECHNICA’S VALUE ADD Technica maximizes the use of commercially available solutions and integrates
them to deliver agile and effective capabilities to our clients. Over time, Technica
has developed value-add capabilities in the areas of Analytics, Artificial
Intelligence (AI), and Cloud Computing Infrastructure & Orchestration that can be
integrated into the ITOA Platform to meet mission-specific unique requirements.
Broad Experience/Past Performance
As a company that has been around for more than 25 years, Technica possesses
deep experience in the Federal, DoD, and Intelligence Community spaces.
Moreover, Technica possesses the requisite skill sets needed to implement the
ITOA Platform, including extensive experience with integration, cloud scaling,
Big Data, and ETL.
Technica IR&D
Additionally, the company maintains a division devoted to Independent Research
and Development (IR&D). This is exceedingly rare for a company the size of
Technica. IR&D focusses on next-generation technologies, especially AI
algorithms that utilize machine learning, graph analytics, and deep learning.
IR&D’s algorithms and solutions directly apply to the ITOA Platform for certain
use cases.
FLASHLIGHT— Flashlight is a plug-in for Elasticsearch developed by
IR&D. Flashlight provides edge/vertex graph analytic capabilities (path,
connectivity, community, and centrality analysis) and machine learning to
connect “dark data” or fringe data associated with a query result, and
performs predictive analytics. Flashlight can be used to enable
user-defined graphs, reports, charts, tables, geospatial maps and other
desired visualizations for IT operations situational awareness.
INNOVATION PLATFORM WHITE PAPER 10
technicacorp.com
AI ANALYTICS— IR&D has devoted a great deal of time developing
next-generation algorithms that can be used to enhance the high-level
analytics present in the ITOA Platform. Most of these AI analytics leverage
deep learning to train models for prediction. The algorithms are packaged
as Docker-based microservices. The AI Analytic Microservices Catalog
available to the ITOA Platform are:
Deep Learning Algorithms
o Fall Detection—Convolutional Neural Network (CNN) o Image Classification—CNN o Time Series Analysis—Recurrent Neural Network (RNN) o Long Short-Term Memory (LSTM) o Anomaly Detection—Autoencoder
Genetic Algorithms to Hyper-tune Parameters
Federated Learning
A complete discussion of these AI Analytic Microservices and deep learning is
beyond the scope of this document, but the following Anomaly Detection section
details a specific use case of anomaly detection for ITOA.
5. ITOA USE CASES ITOA use cases are as endless as IT Operations data streams/applications.
This document presents two current use cases Technica is advancing. This list
will grow over time.
Vulnerability Management
Tracking and remediating vulnerabilities, such as unpatched software and
operating systems and the installation of unsupported applications, plays a
crucial role in ensuring availability and integrity of an enterprise’s services.
Technica’s ITOA Platform can be utilized to strengthen an enterprise’s
vulnerability management systems by correlating the data from disparate tools
that manage security auditing, patching, asset inventory, and software inventory,
bringing to light new, actionable insight.
Figure 8 – Screenshot and Architecture of Vulnerability Use Case using SitScape
SECURITY SERVICES
DATA PIPELINE ASSURANCE
BrowserClient
MobileClient
RichApplication Client
ACQUISITION INTEGRATION ANALYTICS OPERATE VISUALIZE
SCCM
Data Sources
SCOM
ACAS
HBSS
Tanium
INNOVATION PLATFORM WHITE PAPER 11
technicacorp.com
For example, in one real-world data set, the correlation of security auditing and
asset inventory data revealed that the security auditing tool was scanning certain
assets multiple times and inflating the vulnerability numbers every week.
Figure 8 portrays a screenshot and architecture of an actual ITOA vulnerability
implementation.
Beyond data correlation, trending analysis is necessary to isolate issues in
vulnerability management processes and to construct predictive scenarios.
The Architecture provides the capability to automate this process. This, in turn,
lowers the mean time to identify and resolve critical issues, such as the
stagnation of a number of hosts afflicted with a vulnerability due to a halt in
patching. This also frees up analysts and subject matter experts to proactively
pursue innovations, rather than reactively responding to small problems, i.e.,
“firefighting.”
Anomaly Detection
Technica IR&D developed an Artificial Neural Network (ANN) deep learning
algorithm based on an Autoencoder—the Anomaly Detection Microservice.
As portrayed in Figure 9, an Autoencoder is a dense neural network with a
decreasing number of nodes per layer until the middle.
Figure 9 – Notional Autoencoder ANN Architecture
The second portion of the ANN architecture is a mirror image of the first half. As
the Anomaly Detection Microservice is trained, the ANN works to recreate data
streams. As it sees more of the same data, it gets better at recreating the data.
However, when an outlier data stream is presented to the model, the Anomaly
Detection Microservice has trouble recreating the stream. This is an anomaly.
When an anomaly is recognized, a configurable array of alarms, alerts, and
notifications can be generated.
Technica’s AI Anomaly Detection Microservice can be applied to any data
captured in the Collect function. Thus, anomalies could be spotted in vulnerability
management, DevOps, Application Performance Monitoring (APM), network
performance, capacity management, etc.
Inp
ut
Ou
tpu
t
Compressed Feature Vector
DecoderEncoder
INNOVATION PLATFORM WHITE PAPER 12
technicacorp.com
CONCLUSION There is no one-size-fits-all approach to ITOA. Technica’s ITOA Platform is
modular and component-based. The Platform delivers the right information, to
the right people, at the right time. This capability ensures effective decision
making by providing a comprehensive, scalable, end-to-end data management
and operations capability with an architecture that is highly adaptable and
extensible to meet the dynamic and evolving data requirements of the data
center.
Pervasive throughout the ITOA Platform are touchpoints for Technica value-add
services and innovative capabilities that enable and facilitate effective, agile data
management and IT operations.
Technica’s ITOA Platform enables the required agility for IT operations by
minimizing redevelopment and customization activities—focusing on analytics,
functionality, and visualization through graphical-based configuration and
integration utilities.
Technica provides professional
services, products, and innovative
technology solutions to the Federal
Government. We specialize in network
operations and infrastructure; cyber
defense and security; government
application integration; systems
engineering and training; and product
research, deployment planning,
and support.
22970 Indian Creek Drive, Suite 500
Dulles, VA 20166
703.662.2000
©2019 Technica Corporation. All rights reserved.