Transforming SharePoint into a Unified Information Access ... · Search-Driven Applications and UIA...

Transforming SharePoint into a Unified Information Access Platform How the BA Insight Software Portfolio Extends SharePoint 2013

BA Insight

2015

How BA Insight Turns SharePoint 2013 Into a Unified Information Access Platform

© Copyright 2015 BA Insight. All Rights Reserved. Page | 1

Table of Contents

The Need for Unified Information Access .............................................................................. 2

Search-Driven Applications and UIA ......................................................................................................... 2

Approaches to UIA Platforms .................................................................................................................... 3

Leveraging SharePoint .............................................................................................................................. 4

The BA Insight Software Portfolio .......................................................................................... 5

Going Beyond SharePoint 2013 OOB ....................................................................................................... 6

Connectivity Beyond OOB SharePoint ...................................................................................................... 6

MultiSearch: Beyond OOB SharePoint Federation ................................................................................... 7

Content Enrichment Beyond OOB SharePoint.......................................................................................... 8

User Experience Beyond OOB SharePoint ............................................................................................... 9

Incorporating Other SharePoint 2013 Capabilities .................................................................................. 11

Summary .................................................................................................................................12

References ..............................................................................................................................12

About BA Insight .....................................................................................................................13



The Need for Unified Information Access

The modern enterprise is struggling to make sense of the structured and unstructured information that is

being generated and leveraged in exponentially growing volumes across an increasing variety of

application silos. The fragmentation, diversity, and overall inaccessibility of this data can lead to a number

of suboptimal outcomes, such as the lack of responsiveness to customer inquiries, widespread

duplication of effort, general inability to locate and nurture expertise, and decreased employee

productivity.

As awareness grows of the challenges posed by the volume, variety, and velocity of content, today’s IT

leaders are increasingly evaluating technologies that facilitate access to and consolidation of structured

and unstructured content to gain timely insights into various business critical scenarios. One such

technology is Unified Information Access (UIA), an advanced implementation of Enterprise Search

technology that transcends simple keyword searching to allow for secure, ad hoc interrogation of multiple

content sources in a highly targeted and contextually specific manner.

This white paper discusses UIA platforms and applications, and in particular describes how the BA Insight

Software Portfolio transforms SharePoint into a powerful UIA platform, providing capabilities beyond

SharePoint 2013 out-of-the-box.

Search-Driven Applications and UIA

A search-driven application is a software application that uses a search engine as the primary information

access backbone, the main purpose of which is to surface critical information at the right time within the

right context. There are many examples, including:

Customer Service – empowers customers to engage and resolve issues online, enables self-service,

and reduces repetitive questions and answers.

Professional Services – enables identification of experts who have addressed similar needs, delivers

immediate and relevant knowledge to drive projects forward, and supports subscriptions to particular

topics for automatic notification of updates.

Helpdesk – provides actionable resolutions through instant access to critical information through one

portal, and delivers a complete 360 degree view of customer information across multiple sources such as

CRM and ERP systems.

Research and Development – minimizes duplication of effort, increases the speed of innovation, finds

product information quickly, and creates tailored R&D environments.

e-Commerce – provides a responsive, contextual, and remarkable user experience to allow customers to

interact with a brand, find relevant products, and follow up on orders.

While Internet Search has evolved rapidly over the past two decades, Enterprise Search has, for the most

part, remained the same. Internet Search has gone from being a largely horizontal keyword driven

catchall for web content (Lycos, Alta Vista, Google) to being the driver of a number of targeted



applications that have fundamentally transformed the way information is accessed. Search now propels

applications that help us to book travel, navigate from point A to point B, research general interest topics

and purchase vital goods and services (Travelocity, MapQuest, and Wikipedia). These ‘Search-Driven

Applications’ (SDAs) have a number of notable characteristics:

1. They are contextually specific: far from being one search box that yields “ten blue links”, SDAs

are Hotel Room Finders, Flight Finders, and Product Finders, along with research apps for travel

and general interest.

2. They leverage search to access and surface content from a number of sources of both structured

and unstructured data in a unified view.

3. The user experience involves exploration and navigation rather than being simply query-driven,

and it features the use of graphical refinement options like pick-lists, map parts, and sliders that

obviate the need for more complex query formulation.

SDAs are an intuitive, dynamic, and targeted means of rapidly accessing a diverse and fragmented data

set. This approach “offers businesses a rapid, low risk way to eliminate some of the peskiest and most

common information systems (IS) problems: siloed data, poor application usability, shifting user

requirements, systemic rigidity, and limited scalability” (Grefenstette, 2011). The pattern of multiple

tailored and targeted applications is familiar to us in the consumer world and, as our experience suggests,

it can be a successful pattern in the enterprise as well.

Unified Information Access platforms support search-driven applications particularly well. Typically, there

is not one but many different sources of information needed in a search-driven application, and there may

be structured as well as unstructured information. Deploying multiple applications on a UIA platform is far

more cost-effective than fielding each application on a standalone basis.

Approaches to UIA Platforms

UIA platforms include a wide range of technology such as connectivity, text analytics, and data

visualization. There are different approaches you can take to cover the range of capabilities involved in

UIA (IDC, 2014):

Assemble the capabilities as ‘bits and pieces’, with components from different sources and

vendors.

Procure a standalone platform, with all of the capabilities included, and integrate it into your

environment.

Leverage the existing infrastructure (such as search core, workflow engine, and BI stack) and add

a pre-integrated software or components to cover the range of UIA capabilities.

The BA Insight Software Portfolio takes the third approach and leverages existing infrastructure, in

particular SharePoint.



Leveraging SharePoint

SharePoint is the most common content and collaboration platform found in enterprises today. It focuses

on providing pre-integrated capabilities in a wide range of areas and is an extensible infrastructure for

business applications. SharePoint 2013 in particular has strong capabilities in a number of areas relevant

to UIA:

Presentation Framework: SharePoint provides a Familiar UI that is integrated, extensive, and

easy to brand and extend using standard tools.

FAST Search Core: SharePoint includes a Powerful, Scalable, Extensible search core based on

FAST technology.

Integrated BI: Dashboards, Analytics, and Excel Services are all built into SharePoint.

Integrated SharePoint Workloads: WCM, ECM, Workflow, and e-Discovery capabilities are all

included with SharePoint and can be leveraged and extended.

SharePoint has an extensive supporting ecosystem which includes numerous ISVs and System

Integrators, as well as trained administrators within many organizations. SharePoint 2013 is also built with

search-driven applications in mind, and is set up to support multiple search-driven applications running on

the same infrastructure. (Tordgeman, 2013)

SharePoint 2013 provides a great infrastructure to leverage for UIA, as the underlying search core has

many strong capabilities. (BA Insight, 2013) Managed Metadata is pre-integrated via the SharePoint term

store; and workflow, BI, and social features are easily accessed and leveraged.

However, SharePoint 2013 by itself is not a UIA platform. It does not include many of the connectivity,

text analytics, and user interface capabilities needed for Unified Information Access. Nor does SharePoint

claim or aspire to be a UIA platform – its focus is on providing a general purpose, mass market

infrastructure with pre-integrated workloads.

The rest of this paper illustrates how to transform SharePoint 2013 into a Unified Information Access

Platform using BA Insight’s Software Portfolio (BA Insight, 2014). After introducing the platform, we

examine each part of it with specific emphasis on how it extends SharePoint 2013 and the capabilities

that go beyond out-of-the-box (OOB) SharePoint 2013.



The BA Insight Software Portfolio

BA Insight’s Software Portfolio extends the core capabilities of Microsoft SharePoint and FAST to deliver

Unified Information Access and, in doing so, facilitates the delivery of a truly Internet-based search

experience. It enables organizations of all sizes to leverage their existing investment to rapidly deliver

role-specific SDAs without the risk and cost of custom development.

The platform is shown in Figure 1 below.

Figure 1: BA Insight's Software Portfolio

BA Insight’s software portfolio provides a comprehensive set of capabilities:

User Experience Applications - Smart Previews, User-Generated InfoSites, and Matter Comparison to

help users find the right information faster for improved productivity.

Content Classification - Auto-tagging, metadata generation, and text analytics to increase findability.

Content Connectivity - Secure connectors and federation to a wide variety of content systems, enabling

unified views of all knowledge assets.

This flexible portfolio can be implemented as a comprehensive solution suitable for organizations at the

beginning of a unified information access project, or components of the portfolio can be leveraged to

augment existing technology.

The Software Portfolio is based in SharePoint. It fits in with the SharePoint paradigms and toolsets so that

IT Pros and developers who are familiar with SharePoint can leverage their existing skills to implement

and work with BA Insight products. Most importantly, it fully leverages the capabilities of SharePoint. The



Software Portfolio can be used with OOB SharePoint, or it can be combined with SharePoint

customizations and add-ons in a solution. SharePoint capabilities such as workflow and BI are available

as part of applications created with the Software Portfolio.

Going Beyond SharePoint 2013 OOB

Each component within the Software Portfolio provides capabilities that go well beyond SharePoint 2013

OOB. BA Insight does not duplicate capabilities available with SharePoint OOB, but instead adds

capabilities that complement the OOB features.

All of these capabilities support the creation of unified, role-based environments in which users are able

to find relevant information. These solutions need to be search-driven, highly scalable, and connected to

the rest of the enterprise systems in order to make information access dynamic, ad hoc, and relevant to

users.

Connectivity Beyond OOB SharePoint

Any Unified Information Access platform requires what IDC calls “smart connectors” (IDC, 2013). BA

Insight’s Content Connectivity software provides secure connectivity to a wide variety of content systems

to provide a unified view of all knowledge assets.

Connectors: The most obvious connectivity capabilities are indexing connectors. BA Insight provides

connectors to 50+ enterprise sources – all with high performance, flexibility, and security. These work

alongside the OOB connectors (for SharePoint, SharePoint profiles, file shares, and websites).

Connector Development: New connectors, including custom connectors, are built using either the

Universal SQL toolkit or a Web Services API, both of which complement and extend the BCS framework.

These both build upon a Connector Framework, which provides more sophisticated control and faster

development compared to working directly with BCS. The Connector Framework includes pre-built

administrative controls, which new connectors gain automatically, resulting in significant time savings

when creating new connectors as well as better and easier administration.

Targets: These provide the ability to replicate and aggregate information outside of the SharePoint

Search Index, making a variety of interesting scenarios possible. Targets include a search index other

than SharePoint, a SharePoint List, the User Profile Store, or custom targets. As an example, enhanced

People Search uses a User Profile Store target to aggregate a consolidated profile based on information

from many different systems.

Security: BA Insight extends the early binding OOB security (claims and ADFS) to handle the most

difficult search security scenarios. Advanced security capabilities such as role-based and attribute-based

security are supported, and security mapping can be extended into custom fields. These security

capabilities have proven to be essential for Unified Information Access applications, which require strong

and flexible security across many different sources.



Smart Mapping: The manual property mapping available OOB is enhanced to reduce administrative

effort and make sophisticated scenarios possible. This includes Dataset Connectors, Scripting, and

Automated Mapping. Access to all Crawled Properties and Managed Properties is maintained so that

administrators have full visibility and control.

Test Bench: BA Insight provides an integrated Test Bench so that connections and sources can be

tested and debugged in isolation from the rest of the search system. Testing content sources and seeing

the metadata and performance without putting content into the search index has proven to be a huge

timesaver for administrators and developers alike. Scheduled tasks are also provided so that repetitive or

off-hours work can be managed easily.

Associated Crawls and Dataset Connectors: BA Insight includes capabilities for sophisticated content

crawling and association. These extend the OOB crawling mechanism to reduce data quality issues and

make sophisticated scenarios possible. Data Set connectors provide a lookup into external systems

during indexing and allow metadata to be normalized. Associated Crawls allow additional content to be

retrieved and included as part of an indexed item. For example, metadata from a database can be

indexed along with associated content in a file system.

MultiSearch: Beyond OOB SharePoint Federation

Query Federation is an important tool in the Unified Information Access platform toolkit, and it comes in

several forms (Schubmehl, 2013). Requirements for this come up in several ways:

Data residency, information export, and contract restrictions may make centralized indexing

impossible.

Organizational control, varying requirements, or logistical needs may make separate search

implementations necessary or desirable.

External content such as subscription content, news sources, web search engines, or large-scale

research repositories may not allow crawling, or it may be prohibitively expensive to crawl and

index.

These scenarios require a unified view but do not allow for all content to be in a single index. To address

them, a capability to query against multiple indexes and collect the results is necessary.

In SharePoint 2013 there is an OOB query federation capability called Remote Result Sources. BA

Insight provides a MultiSearch capability which builds on the OOB mechanism to provide a seamless

unified view across multiple remote sources. It uses the OOB federated security model and provides

several important capabilities beyond OOB query federation.

Interleaved Results: MultiSearch extends the OOB result source mechanism to provide a true unified

view experience which extends beyond result blocks and local refiners to a fully interleaved result set.

Refiner values are merged, and metadata can be augmented – including the creation of a source refiner

which allows users to see which results come from which source, as well as the ability to drill down to a

specific source or set of sources. A single result set has proven to be much more natural and effective



than different tabs or lists in many scenarios. The more sources, the more important it is to have an

interleaved result set.

Federated Back-ends: These complement the OOB result sources, which are sealed and limited to

remote SharePoint, Exchange, and OpenSearch. BA Insight provides additional sources (called back-

ends) which include full security and refiners and work cross-platform. These include FS4SP, FAST ESP,

Google Drive, SharePoint 2010, and SharePoint 2013.

Custom Back-ends: A back-end toolkit makes it possible to create custom federated sources with all the

same attributes as the native back-ends. This includes relevance scores, refiners, and full security, which

goes far beyond the OpenSearch mechanism available OOB.

Hybrid Search: BA Insight extends the OOB hybrid search model to support a wider variety of scenarios

for unified search results between Office 365 and on-prem SharePoint. These include cross-version

federation, content enrichment, and federation across multiple farms and tenancies. MultiSearch supports

either SP2013 or SP2010 on-prem, working with SharePoint 2013 in Office 365. There is also an O365

connector which supports crawling O365 from on-prem SharePoint search.

Content Enrichment Beyond OOB SharePoint

Another essential element of a Unified Information Access platform is text analytics applied to pre-

processing of content. BA Insight’s Content Classification software increases the findability of content

through the creation and normalization of metadata. It generates and extends managed metadata and

provides a collection of powerful text analytic capabilities.

SharePoint includes a Managed Metadata Service (MMS) which holds taxonomies and ontologies and

allows for manual tagging of content to drive findability, workflows, compliance, etc. It also includes a

number of other powerful elements used in information architecture, such as content types. There are

also OOB verbatim extractors (string matchers) to populate managed properties in search. However,

SharePoint does not include text analytics or auto-classification. This is where the Content Classification

software comes in.

Automatic Metadata Creation: Complements OOB metadata, leveraging the SharePoint MMS (aka term

store) to assign metadata to content by machine rather than by hand. Event triggering allows composition

of metadata generation, forms, and workflow. Metadata generation can be done in-place (in SharePoint

lists and libraries), or during indexing (against any content source with a connector). Automatic metadata

creation enables a wide variety of metadata-driven scenarios around UIA, search, ECM, workflow, and

compliance.

Entity Extraction: is one of the forms of automatic metadata creation. BA Insight complements and

extends the OOB verbatim extractors (string matchers) to support sophisticated scenarios. This includes

support for Regular Expressions, Partial Matchers, and Custom Extractors.



AutoClassification: adds the ability to create metadata beyond string-matching. It supports taxonomy

and ontology-based scenarios such as a hierarchy of product names. The Content Classification software

provides Rules-Based classification, Hierarchical (taxonomy-based) classification, and Extended Tags

that support content linking and ontologies.

Content Types: are provided in SharePoint and were enhanced with SharePoint 2013. BA Insight adds

the capability to automatically assign content types. This can be used with the SharePoint Content

Organizer and other OOB facilities to support a variety of different scenarios and provide better

compliance and content quality.

Taxonomy Management: a Taxonomy Manager in the Content Classification software complements the

OOB taxonomy management UI and can be used alongside or instead of it. The Taxonomy Manager

includes a drag & drop UI and a Test Bench which reduces administrative effort and speeds development

and troubleshooting of classification rules.

Import/Export: supports interoperability across industry tools by import and export to the SharePoint

term store and includes important elements beyond the OOB CSV import/export terms. Support includes

CSV, SKOS, RDF, and ARTX formats for import/export; the files include terms, nodes, and relationships

plus Synonyms, Rules, and Extended tags.

The Content Classification software brings machine-made metadata into the SharePoint infrastructure.

This is an essential element of Unified Information Access and provides a very powerful basis for effective

search-driven applications. Since the metadata is integrated into SharePoint, it is immediately accessible

for workflow, analytics, forms, and content management. For example, when a document is uploaded to a

drop-off library, a document can be auto-classified, with a form presented to the user to verify the

metadata. The Content Organizer (part of SharePoint) can then move it to the appropriate library –

ensuring consistent metadata including content types as well as consistent organization of document

locations.

User Experience Beyond OOB SharePoint

The user interface makes a remarkable difference in the effectiveness of Unified Information Access, and

tailored search-driven applications need tailored user interfaces. Yet the effort and expertise required to

do this well can be so daunting that many organizations don’t undertake it, or do so and fail. That’s why

BA Insight has developed software that reduces the risk and cost of crafting these applications. User

Experience Applications enable the creation of compelling and effective user interfaces for search-driven

applications.

SharePoint 2013 provides powerful mechanisms for tailoring the UI, including display templates, query

rules, and web parts. User Experience Applications use and extend these mechanisms. IT Pros configure

the user interface using familiar tools and paradigms, and developers can further customize the result

using standard techniques – while leveraging a set of prebuilt, proven components. User Experience

Applications provide many capabilities that go beyond OOB SharePoint:



Workspace: adds a per-user workspace capability allowing users to tag, remember, and share content

from a search session. Users can Select, Arrange, and Export from the workspace; the workspace is

persistent across devices and sessions so you can continue to work seamlessly from the office PC to

your iPad at home.

Content Assembly: enables the creation of composite documents once they are tagged into the

Workspace. For example, users can create Word documents, PowerPoint presentations, PDFs, Excel

manifests, and ZIP fields using search derived content while preserving images and native formatting.

This works on selected content in any supported format, stored in any repository with a connector.

Content Assembly has proven to be a huge timesaver, as users select and re-use content into a new

report automatically, eliminating cut-and-paste work.

Smart Previews: are visual images for quick recognition and exploration of relevant information,

including features for exploring information including search inside, hit mapping, hit highlighting, and

contextual actions. These complement the OWA previews by providing previews of content stored outside

of SharePoint 2013 and in file formats beyond Office 2013. The BA Insight HTML Viewer shows Smart

Previews on documents stored anywhere in a wide variety of file formats including Office, PDF, Zip,

email, image – and takes the user to the most relevant page.

This capability is based on Smart Preview Generation, which understands the format and structure of

many file types and creates data structures for navigating inside documents and connecting across

documents. Smart Preview Generation allows large complex documents to be processed at index time,

providing a dramatically quicker response time and lower bandwidth usage. Smaller documents can be

processed on demand at query time.

Navigation Controls extend the OOB Refiner Panel and Web Part paradigm to provide an extensive set

of pre-built controls for intuitive navigation. These include visual controls for drilldown, refinement, and

content exploration, including many different presentation elements such as multi-select tree refiners,

graphs, and dashboard layouts. Collapsing refiners preserve precious screen real estate while surfacing

the many different facets of content. Searching inside refiners allows users to navigate across large

amounts of data quickly and with great precision. Query Controls are configurable elements such as

breadcrumb, site tree, and date picker that make it easy for the user to form effective queries.

Saved Searches: Adds the ability to save, review, and run queries, including complex queries with

multiple refinements applied. A web part provides immediate access to them, along with the display of

query history. Saved Results adds the ability to export a result set into excel for analysis, sharing, and

support of sophisticated scenarios. These have proven to be highly valuable both in time savings and in

providing analytics on content to drive insight and decisions.

Dynamic Query Transformation: brings the concept of Query Rules to the next level by allowing full

control of query processing via plug-in processing stages. It can be used alongside or instead of query

rules, enables a wide range of search scenarios and provides full control of behavior like linguistic query

processing, query steering, and application-driven results. These transformations change and modify

search queries to model the intent of the user, expand queries with external information, and affect



relevance on-the-fly. All of this is done transparently to the UI, which simplifies development and

administration.

Dynamic Result Transformation: brings the concept of Result Sources to the next level by allowing full

control of result processing via plug-in processing stages. It can be used alongside or instead of result

sources. These transformations update the result set, mixing in dynamic data (such as adding real-time

inventory levels, stock market values, or weather information with a result), and reorder or translate the

values to deliver the best result for each user. This enables a wide range of search scenarios and content

mashups.

All together, the User Experience Applications provide a very powerful suite of pre-built components

proven to be essential for success. Rather than rebuilding the wheel, enterprises can assemble

remarkable applications using these components, and style the UI using standard CSS and JavaScript.

Incorporating Other SharePoint 2013 Capabilities

You have seen how the Software Portfolio turns SharePoint 2013 into a Unified Information Access

platform, and read about capabilities that go beyond OOB SharePoint 2013 in every part of the platform.

There are also many capabilities of SharePoint 2013 OOB that contribute to UIA. In particular, SharePoint

has a complete BI stack including Excel Services, PowerView, PerformancePoint, and PowerBI, so that

insights from traditional BI on structured content can be part of search-driven applications. Workflows can

be configured to work with metadata generation and also with searches, directly as part of these

applications. The social, composite, and e-Discovery workloads within SharePoint can also be tapped.

The combination of SharePoint and the Software Portfolio is a more capable UIA platform than any

standalone UIA product, bringing a wider range of pre-integrated capabilities into the mix. The

development and administrative tooling available is robust, and there are a large number of professionals

familiar with it. The Software Portfolio transforms SharePoint into a UIA platform that is unparalleled in the

depth and breadth of its capabilities, its ease of administration and extension, and its cost-effectiveness.



Summary

The BA Insight Software Portfolio is not another search engine or standalone UIA platform. Instead, it

leverages the FAST index in SharePoint as well as the metadata management, user profile, and BI

capabilities - and extends these with a comprehensive set of connectivity, text analytics, and user

interaction capabilities. It enhances the systems you already have in place by connecting to them,

enriching the content, and unifying the presentation of this content with information from other systems.

The result enables rapid assembly of powerful search-driven applications that give users the ability to

view, manage, and access content in one place - regardless of its structure or where it resides in your

organization.

References

BA Insight. (2013). The Essential Guide to Enterprise Search in SharePoint 2013. BA Insight.

BA Insight. (2014). Software Portfolio Data Sheet. available from www.bainsight.com.

Grefenstette, G. a. (2011). Search-Based Applications. Morgan & Claypool.

IDC. (2013). Unified Information Access 2013: Unifying Content SIlos to Uncover Hidden Value.

IDC. (2014). Unified Information Access: Turning Enterprise Search Into a Killer App. available at

www.bainsight.com: IDC Analyst Connection.

Schubmehl, D. (2013). Agile Information Integration Using Federated Search. IDC.

Tordgeman, J. (2013). Learning Search-driven Application Development with SharePoint 2013. Packt

Publishing.



About BA Insight

BA Insight’s software portfolio accelerates and future-proofs the creation of cloud-based and on-premise

SharePoint portals for enterprises, transforming how users find information. Our software dramatically

reduces the time, cost, effort, and risk of implementing SharePoint portals while providing a greatly

improved user experience. Customers use our applications to improve productivity by finding the right

information faster using Smart Previews, User-Generated InfoSites, and Matter Comparison; our

classification software to increase findability using auto-tagging, metadata generation, and text analytics;

and our pre-built connectors and federation to provide secure connectivity to a wide variety of content

systems.

We serve organizations of all sizes around the globe including the Australian Government Department of

Defence, CA Technologies, Chevron, Deloitte, Ford Motor Company, Keurig Green Mountain, Pfizer, and

Travers Smith. BA Insight is a Microsoft Gold Certified Partner and member of the Microsoft Business-

Critical SharePoint Program.

Visit www.BAinsight.com for more information and follow us at @BAinsight.

Transforming SharePoint into a Unified Information Access ... · Search-Driven Applications and UIA...

Documents

Transcript of Transforming SharePoint into a Unified Information Access ... · Search-Driven Applications and UIA...