Enterprise Search in SharePoint 2010

Bob GermanTechnology ArchitectMicrosoft Technology Center | Boston

Agenda

SharePoint Search OptionsSharePoint Search EngineFAST for SharePoint

Social SearchContent AcquisitionFederation

Scale – content and query scale becomes a non-issue

User experience –guide users with rich, visual navigation

Social and people search – search unlocks the value of the people and expertise

Relevance –improving our matching and ranking capabilities and language support

Platform –tools to customize, extend, or build new applications. 

Management & reliability –reliable crawling, scriptable admin, improved monitoring.

Rich Content Processing–extract metadata from my enterprise content

You Asked, We Listened!

Quick, easy, powerful search

(for free!)

Complete intranet search

High-end search delivered through

SharePoint

A Family of Search Solutions

Family Values

• Common UI Framework

• Social search features and integration

• SharePoint platform integration

• End user and site administrator enablement

• Common Connector Framework (BDC)

• APIs and developer experience

• Admin & deployment capabilities

• Operations advantages (SCOM, scripting)

SharePoint Search

End Users Content +

Profiles

FAST Search for SharePoint

FAST Search Extends SharePoint ServerSuperset of capabilities; common base functionality and platform

QueriesPeople

Unique Social Search from SharePoint Server

Common connectors and content APIs

Common query and federation APIs

Best High-End Content Search based on FAST ESP

Common developer experience and IT Pro Foundation Content

SharePoint Search Engine

What’s new in 2010?

SharePoint 2010 Shared Service Applications

SSP

SharePoint 2007 Model

http://hrweb/

SearchUser

Profiles

Excel Calc

Corp Farm

Business

Data

http://hrweb/

Corp Farm

SharePoint 2010 Model

http://itweb/ http://itweb/

Search

User Profiles

Excel

Calc

Visio

3rd party Servic

e

HRBusine

ss

Data Office

Web Apps

IT Searc

h

SharePoint Search SA

Each Search Service Application (SSA):

Administration component , databaseQuery Topology

Query Components (1+)Property Databases (1+)

Crawl TopologyCrawl Components (1+)Crawl Databases (1+)

Services (Query, Admin)

Default Search Configuration

Out of the Box:1 Admin component1 Crawl component1 Query componentDatabases on drive specified in post-setup configuration

Fully functional:Crawl contentDevelop customizations

CrawlAdmi

n

Property DB

Admin DB

Crawl DB

Query

(other SharePoint DBs)

Component Scale-out

Query:SegregatePartition for scalePassive mirrors provide failoverActive mirrors increase throughput

Crawl Additional crawl componentsCrawl Database

Crawl

Query

Admin

Crawl DB

Property DB

Admin DB

Crawl DB

Crawl

Crawl DB

Crawl

Crawl

100 million items!

Query Q2Q1

Q2 Q1

Admin DB

Q6

Q5

Q5

Q6

Q4

Q3

Q3

Q4

Q2

Q1

Q1

Q2

Manageability – Monitor Crawl

“Levels” of reports

Current rateCrawl rate per typeCrawl processing

Filter by SSA, timeOptimize crawl resources to achieve freshness goals.

Manageability – Monitor Query

“Levels” of reports

Current rateOverall Query LatencyDetailed Backend Query Latency

Optimize query latencyEnable verbosequery latency, trend

FAST Search Engine

High-end Search for SharePoint

Go Beyond the Search BoxVisual, Conversational Search

Thumbnails

Sorting on any property

Similarity Search

Visual Best Bets

Scrolling PowerPointPreviewsRefinement with

countson any property

FAST for SharePointArchitecture

DocumentProcessing

Indexing SearchQuery &Result

ContentConnectors• SharePoin

t• FAST• CCF

SharePointAPI’s

Administration

FAST Search Engine Shared with SharePoint Search Engine

SearchCenter

Users

Content PipelineAdvanced Content Processing Out-of-the-Box

Configurable content processing pipeline

Transform and process content automatically

Strong linguistic coverage80+ languages detected45 languages with advanced linguistics features

Property Extraction to create new metadata

Prebuilt extractors: Person, Location, CompanyGeneric extractors to extendExtensibility

Processing & refinement

…

FormatConversio

n

LanguageDetection

EntityExtraction

Configurable

Stages

Mapper

REDMOND, Wash., and OSLO, Norway — Jan. 8, 2008

Microsoft Corp. (Nasdaq “MSFT”) today announced that it will make an offer to acquire Fast Search & Transfer ASA (OSE: “FAST”), a leading provider of enterprise search solutions, through a cash tender offer for 19.00 Norwegian kroner (NOK) per share. This offer represents a 42 percent premium to the closing share price on Jan. 4, 2008 (the last trading day prior to this announcement), and values the fully diluted equity of FAST at 6.6 billion NOK (or approximately $1.2 billion U.S. ). FAST’s board of directors has unanimously recommended that its shareholders accept the offer.

Location

Company

Company

Company

Enable Conversational SearchDeep Refiners with Exact Counts

Search results can be refined based on:● Metadata from SharePoint (such as Author)

● Enriched meta data based on entity extraction (such as

Company)

Custom refiners can be made by:● Enabling new Managed Properties to be used as refiners

● Configuring the Refinement Web Part to display the new

refiner

User Context MattersAlan Brewer, Sales

What should I know about selling ERP consulting?

Renee Lo, Engineer

What should I know about implementing ERP?

Large Scale Deployment

Add columns to extend document capacityAdd rows to extend query capacity or fault tolerance

Indexer/Search

Indexer/

SearchRow 0

Indexer/

Search

Indexer/

Search

Indexer/

Search

Indexer/Search

Row 1

Column 0

1

Indexer/

Search

Indexer/

Search

2 3

Social Search

Go Behind the Search BoxConnect with People and Expertise

Phonetic and nickname matching

Improved result layout and hit-

highlightingRecently authored

content

Refine by query type, and many

other pivots

Sort by relevance, name

or social distance

Vanity search

Leveraging the Managed Metadata Service for better People Search

Out of box:Responsibilities, Interest, Skills, Schools, are all taxonomy propertiesAutomatic refinement on relevant pivots

ExtensibilityAdding a new taxonomy property to the profile store

Click through improves relevanceThe wisdom of the masses helps the masses

Documents are boosted by ‘popularity’: if many users find a document useful, it must be useful.

Search suggestionsHelping users form better queries

Pre-query suggestions

Post-query suggestions

Social tags

Recall

Ranking

Refinement

Results

Results with social tags are returned.

Results with social tags are boosted.

Users can filter results on social tags.

Top social tags are displayed as metadata on each result.

Content Acquisition

Feeding the Search Engines

Making Content Searchable

Use Query FederationQuery across external indexesBased on OpenSearch standard(http://www.opensearch.org/)

Use Indexing ConnectorsCrawl, enumerate and create local index of contentUse out-of-box connectors or write custom connector

Shared Out-of-box Indexing Connectors in SharePoint 2010SharePoint content (Protocol Handler -

PH)File shares (PH)Web sites (PH)People profiles (PH)Lotus Notes™ (Connector Framework - CF)Exchange public folders (CF)External systems(CF)Documentum™ content (CF)

Fast Search out-of-box Connectors

Enterprise crawlerGood for large scale crawling across multiple nodesSupports dynamic data including JavaScript

JDBC™ connectorSimple configuration using SQL commandsSupports push based crawlingSupports change detection through checksums

SharePoint Connector

Accesses data through SharePoint web service (using Windows credentials)Supports full crawl through enumeration of content and incremental crawl through change logBuilt-in support for security trimming (Windows and pluggable)

Connector Framework

Easy to buildWrite managed code instead of PH interfaceBuilt connectors shared across SharePoint Search and FAST search‘No-code’ solution for Databases and Web Services

Easy to manageUse existing user interfaces for crawl configurationFaster iteration on connectivity using SharePoint Designer (SPD)

Shared advantages across SharePointCan use other BCS features such as external lists

Connector Framework Architecture

Search Engine/IndexSearch BCS

Runtime

WCF Databases

.NET BCS

Connector

Execution call routing through BCSConnectors using BCS model file

Custom BCS

Connector

Search user queries

BCSProfile Pages or External Lists

Search results page – results

based on metadata

Search page points to Profile Page or custom page to show search result

Fetch data from repository

Protocol Handlers

Enabling Connectivity

Every indexing connector needsA BDC model file to express connection information and the structure of the backendA BDC Connector for code to execute when accessing the backend

Out-of-box BDC ConnectorsDatabaseWCF (Web Services).NET (custom code)

Federation

Combining Search Results

Eliminate Compromise

Web

Files

Databases

Lotus Notes

Documentum

Easy to Build

OpenSearch Federation

Enterprise Content Mgmt

Systems

Line of Business Apps

Custom

Apps

Asset Mgmt

systems

Indexing Connectors

Easy to Connect

Easily Connect to Corporate Assets that live Outside of SharePoint

SharePoint

Exchange Server

…

“Out of the Box” FederationPeople Search and Related SearchesBing and other OpenSearch providersWindows 7 and IE8 searching SharePoint 2010Rich RSS feeds and federation generator

How OpenSearch federation works

HTTP request with query in the URL:http://www.site.com/srchrss.aspx?q={searchTerms}

RSS / Atom results:RSS results with <title>, <link>, <description>Best sources also include: <pubdate>, <author>, <category>, <media:thumbnail>Optionally include custom metadata: <recordid>, <projectname>, <contactnumber>

HTTP request

RSS/Atom Results

Search Provider

Search Client

Deploying a federation connectorUse or create a searchable feed

Use an existing searchable feed or create your own.Anything searchable – takes a query as a URL parameter.Must return results in RSS or Atom formatted XML.Creating a searchable feed is simple. Just code a basic ASPX page.

OpenSearchPage

Search Provider

Search Client

Make Search Pervasive!Deploy OpenSearch (OSDX files) Everywhere

SharePoint 2010:

Import the file.

Add it to a Search Center.

Windows 7:

Pull: Add a link to the OSDX file on the search center or team site.

Push: Deploy the OSDX file via Group Policy.

IE8

Link to Search Provider

Discoverable Search Provider

Extend and Create Search Web PartsThe OOB Web Parts are no longer sealed!

 Complete enterprisesearch solution

High-end search delivered through SharePoint

End UserGo Beyond the Search Box

Great out of box search experience 

Most visual and dynamic user experience

IT ProEliminate Compromise

Enterprise class search system integrated

with SharePoint

Highly capable and easy to manage

DeveloperDo more with Search

Open and flexible search platform

The power to build a whole new class of search

applications

Microsoft Enterprise Search - ProductivityProducts for Every Search Need

FAST Search for SharePoint and ESPA new choice; a different world

Bringing high-end search to the mainstream market

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

 Great OOB search experience Most visual and dynamic user experience

One-stop search center to find answers fast

Excellent OOB relevanceFederated resultsPC, phone, or browser access

Same capabilities, plus:Broader, better language support Deeper query syntax

Refinement to explore information quickly

Metadata based refiners Related searchesView in Browser

Same capabilities, plus:Deep refiners with countsSimilarity searchSorting on any property

Social search to connect with people and expertise

Phonetic & nickname matchingRelevance and refinersIntegrated presence

Same capabilities

Search gets better with use

Click-through drives relevanceColleague & Expertise suggestionsQuery suggestions

Same capabilities

Visual cues for rapid recognition of information

Thumbnails & Scrolling PreviewsVisual Best Bets

Contextual to meet the needs of diverse groups

User context from user profileMultiple relevance profiles

Go Beyond the Search BoxEnd User perspective

 Enterprise class search system integrated with SharePoint

Highly capable and easy to manage

Industrial strength scale-out and performance

Enterprise Scale-out (to 100M docs)Full Fault ToleranceNative 64 bit; Hyper-V support

Same capabilities, plus:Extreme scale-out

Easy deploymentWizard - driven installationConsolidated search dashboard

Same capabilities

Enterprise-class manageability

PowerShell supportSCOM support Full search reporting

Same capabilities

Secure, broad connectivity

Full set of connectors OOB Easy to add new sources via BDCStrong security

Same capabilities, plus:Enterprise Web Crawler

Advanced content processing OOB

Content Processing pipelineMetadata Extraction

Easy to configure high-end user experiences

Easy setup of User Context, Visual Best Bets, PromotionGive end users control over sorting, ranking, navigation

Eliminate CompromiseIT Pro perspective

Do More with SearchDeveloper Perspective

The power to build a whole new class of search applications

Same capabilities, plus:

Public web parts with high end

capabilities

Add custom property extractors

Extend content processing

Include external data in

relevance

Build multiple relevance profiles

Extend user context

Use advanced query capabilities

to create powerful applications

 Open and flexible search platform

Public web parts

New connector framework

Integrated with Business

Connectivity Services (BCS)

Integrate search with BI,

Workflow,

Social, & Collab

BDC tooling built into

SharePoint Designer

Application tooling in VS2010

Query Object Model

Concepts

Search Technology Concepts

Content Sources - Host the content we want to return in main results

Content

Content

Content

Federated Source

Crawling - Traverse URL space to record items in search catalog

Indexing - Extract information from items to enable efficient matching

Query Servers - Accept query requests from users and return results

Search Center - UI for users to issue queries and interact with results

Query Federation - Return results from non-SharePoint Indexes

Crawler

Indexer

Query Servers

Index Files

Indexing Connectors - Know how to process different content sources

Index Partition - Subset of the overall index