PRT376 SharePoint Search Technologies Kevin Laahs Principal Consultant Advanced Technology Group HP...

PRT376PRT376SharePoint Search TechnologiesSharePoint Search Technologies

Kevin LaahsKevin LaahsPrincipal ConsultantPrincipal ConsultantAdvanced Technology GroupAdvanced Technology GroupHP ServicesHP Services

ObjectivesObjectives

Understand differences between search Understand differences between search in Microsoft Windows SharePoint in Microsoft Windows SharePoint Services (WSS) and Microsoft Office Services (WSS) and Microsoft Office SharePoint Portal Server 2003 (SPS)SharePoint Portal Server 2003 (SPS)

Learn configuration and deployment Learn configuration and deployment options for both technologiesoptions for both technologies

Raise awareness of customization Raise awareness of customization options and opportunities for affecting options and opportunities for affecting the end-user experiencethe end-user experience

AgendaAgenda

Common Search TechnologyCommon Search Technology

Windows SharePoint ServicesWindows SharePoint Services

Microsoft SharePoint Portal Server 2003Microsoft SharePoint Portal Server 2003

Co-existence WSS & SPSCo-existence WSS & SPS

Customizing, Re-using and ExtendingCustomizing, Re-using and Extending

Windows Indexing Services

SharePoint Portal Server 2001

SharePoint Portal Server 2003 Exchange 2000/3

SQL ServerSQL Server 7

SQL Server 2000


Common Indexing ArchitectureCommon Indexing Architecture

1

1.Seed URL(s) added by "consumer".

2

2.PH fetches document or enumerates directory

3

3.Text extracted from object, tokenized and normalized.

4

4.Text is passed down pipeline. Custom plug-ins operate on stream.

Filter Daemon

Search Service

Pr o

to co l

Ha

nd ler( s)

Filter( s)

Word

Bre a

ker( s)

Norm

a l izer

UR

L

Te

xt

FilterQueue

“Indexing Pipeline”

Gatherer Alerts AutocatIndexer

File

HTTP

Custom

Extensible viaPlatform SDK

Notifications

Seed URLs

Catalogs

Important ConceptsImportant Concepts

Applications call search service to Applications call search service to create catalogscreate catalogs

Applications provide starting URLs to Applications provide starting URLs to the content they want indexedthe content they want indexed

Applications consume the index by Applications consume the index by constructing and performing queriesconstructing and performing queries

Applications render the results to Applications render the results to end-usersend-users

AgendaAgenda


Windows SharePoint ServicesWindows SharePoint ServicesMain Architectural ComponentsMain Architectural Components

End-user experienceEnd-user experience

SharePoint Portal Server 2003SharePoint Portal Server 2003



Terminology!Terminology!(Your mileage may vary (Your mileage may vary ))

Team SiteTeam Site

Site CollectionSite Collection

Sub-SiteSub-Site

Portal SitePortal Site

Portal AreaPortal Area

My Site/Personal SiteMy Site/Personal Site

Web siteWeb site

WSS – ArchitectureWSS – Architecture

WSS search is provided by SQL Full Text WSS search is provided by SQL Full Text IndexingIndexing

So NO search if using (W)MSDESo NO search if using (W)MSDE

SQL Indexing provides the SQL protocol SQL Indexing provides the SQL protocol handlerhandler

So seed URLs can only come from content in So seed URLs can only come from content in SQL tablesSQL tables

Result set can only contain content in SQL tablesResult set can only contain content in SQL tablesSo no “aggregated” search experienceSo no “aggregated” search experience

Catalogs are per databaseCatalogs are per databaseIndexing set per tableIndexing set per table

Individual columns specified Individual columns specified

Default location C:\Program Files\Microsoft SQL Default location C:\Program Files\Microsoft SQL Server\MSSQL\FTDATAServer\MSSQL\FTDATA

Enabling WSS searchEnabling WSS search

Enable through WSS Central AdministrationEnable through WSS Central AdministrationEnabled for ALL team sites hosted on the farm Enabled for ALL team sites hosted on the farm (i.e., ALL virtual servers)(i.e., ALL virtual servers)

Catalog created for each content database on Catalog created for each content database on farmfarm

Four indexes created in each catalogFour indexes created in each catalog

Change tracking enabledChange tracking enabledBackground method usedBackground method used

Rule of thumb – allow 30 - 40% disk space for Rule of thumb – allow 30 - 40% disk space for catalogscatalogs

Indexed TablesIndexed Tables

TableTable Indexed ColumnsIndexed Columns DescriptionDescription

UserInfoUserInfo tp_Login, tp_Title, tp_Login, tp_Title, tp_Emailtp_Email

Details of users that Details of users that have visited a site have visited a site collectioncollection

ListsLists tp_Title, tp_Descriptiontp_Title, tp_Description Title and description Title and description of listsof lists

UserDataUserData nvarchar1 to nvarchar64nvarchar1 to nvarchar64

ntext1 to ntext32ntext1 to ntext32

Text based custom Text based custom columns for items in columns for items in a list or librarya list or library

DocsDocs LeafId, ContentLeafId, Content Filename of Filename of document and its document and its content.content.

Note: Individual list items are indexedNote: Individual list items are indexed

But no support for attachments to list itemsBut no support for attachments to list items

And no metadata from documents (e.g. Office properties)And no metadata from documents (e.g. Office properties)

Indexing in ProgressIndexing in Progress

Full-Text Index PropertiesFull-Text Index Properties

End-User ExperienceEnd-User ExperienceLimited!Limited!

IFiltersIFilters.doc, .doc, .xls, .ppt, .txt, and .htm.xls, .ppt, .txt, and .htm3rd party can be added e.g. .pdf3rd party can be added e.g. .pdf

SQL SQL full-text search extensions are integrated into full-text search extensions are integrated into the T-SQL languagethe T-SQL language

CONTAINS and FREETEXT predicates are supported.CONTAINS and FREETEXT predicates are supported.Search from a team site uses FREETEXTTABLE Search from a team site uses FREETEXTTABLE

Limited in scope to the team siteLimited in scope to the team siteNo subsites searched No subsites searched

Search terms are basicSearch terms are basicNo phrasing or Boolean (multiple terms treated as an OR)No phrasing or Boolean (multiple terms treated as an OR)No narrowing of search, no number of hitsNo narrowing of search, no number of hits

Can scope a search to an individual listCan scope a search to an individual listCurrent view of list is searchedCurrent view of list is searched

Search ResultsSearch Results

Uses FreeTextTableUses FreeTextTable

proc_FetchListItemSearchResults proc_FetchListItemSearchResults

proc_FetchDocSearchResultsproc_FetchDocSearchResults

SearchResults.aspxSearchResults.aspx5 items/documents displayed on initial 5 items/documents displayed on initial page ordered by RANKpage ordered by RANK

Based on frequency of word(s) in indexed Based on frequency of word(s) in indexed columnscolumns

And overall occurrences of word(s) in And overall occurrences of word(s) in the tablethe table

Windows SharePoint Services Windows SharePoint Services SearchSearch

Screen shot of search results hereScreen shot of search results here

WSS Search SummaryWSS Search Summary

Focus is on team siteFocus is on team siteFunctionality ok for “small” team sitesFunctionality ok for “small” team sites

Scalability could be an issue in large Scalability could be an issue in large deploymentsdeployments

> 1 million items => thinking!> 1 million items => thinking!

End-user confusion when SPS is in End-user confusion when SPS is in picture as wellpicture as well

Designed primarily for stand-alone Designed primarily for stand-alone WSS installationsWSS installations

See later!See later!

AgendaAgenda



SharePoint Portal Server 2003SharePoint Portal Server 2003Design goalsDesign goals

Main Architectural ComponentsMain Architectural Components

ManagementManagement

End-user experienceEnd-user experience

Search in Shared Services ScenarioSearch in Shared Services Scenario



Laying Groundwork in V1Laying Groundwork in V1

SharePoint Portal Server is Microsoft’s SharePoint Portal Server is Microsoft’s flagship application for enterprise searchflagship application for enterprise search

Securely aggregates data from multiple sourcesSecurely aggregates data from multiple sources

Index metadata as well as contentIndex metadata as well as content

Probabilistic relevance rankingProbabilistic relevance ranking

Alerts notify you of new relevant informationAlerts notify you of new relevant information

Automatic categorizationAutomatic categorization

Adaptive crawling keeps search currentAdaptive crawling keeps search current

Best bet taggingBest bet tagging

Search Themes in SPS V2Search Themes in SPS V2

Enterprise abilitiesEnterprise abilitiesScale to large deploymentsScale to large deploymentsShared portal servicesShared portal servicesOverall better managementOverall better management

Relevant, rich resultsRelevant, rich resultsFast, easy-to-use search UIFast, easy-to-use search UI

Organization of informationOrganization of informationSite RegistrySite RegistryBest bet and metadata managementBest bet and metadata managementPeoplePeople

Team site integrationTeam site integrationExtends support for secure contentExtends support for secure content

SPS Indexing ArchitectureSPS Indexing Architecture

Filter Daemon

Search Service

Pr o

to co l

Ha

nd ler( s)

Filter( s)

Word

Bre a

ker( s)

Norm

a l izer

UR

L

Te

xt

FilterQueue

“Indexing Pipeline”

Gatherer Alerts AutocatIndexer

File

HTTP

Custom

Extensible viaPlatform SDK

Notifications

Seed URLs

Catalogs

Scalable Search ArchitectureScalable Search Architecture

IndexingIndexing IndexingIndexing

SearchSearch SearchSearch

WebWeb WebWeb

ServicesServicesDBDB

Portal Portal ContentContent SalesSales ProductsProducts

1.1.Multiple indexing Multiple indexing servers support up servers support up to 4 content to 4 content indexes (best indexes (best practice)practice)

11

2. Each index 2. Each index propagates to all propagates to all search serverssearch servers

22

SalesSales SalesSales

3. Search requests 3. Search requests are balanced are balanced across search across search serversservers

QueryQuery 33

Managing Indexed ContentManaging Indexed Content

What content?What content?

How often do I want it indexed?How often do I want it indexed?

Where do I want to store the index of Where do I want to store the index of the different content sources?the different content sources?

How will I partition my index for How will I partition my index for scoping of searches?scoping of searches?

What metadata will I index and how will What metadata will I index and how will end users avail of it?end users avail of it?

How many index and search servers How many index and search servers will I have?will I have?

Content IndexesContent Indexes

Another name for a catalog!Another name for a catalog!But separate property store is also maintainedBut separate property store is also maintained

\Program Files\SharePoint Portal Server\Data\Program Files\SharePoint Portal Server\Data

Best practice allow 50% diskspaceBest practice allow 50% diskspace

Can have as many as you likeCan have as many as you likeBest practice suggests max of 4 with 5 million items in Best practice suggests max of 4 with 5 million items in eacheach

Two indexes out of the boxTwo indexes out of the boxPortal_ContentPortal_Content

Non_Portal_ContentNon_Portal_Content

Ultimately, search is performed against one Ultimately, search is performed against one or more indexesor more indexes

Depends on content sources and search scopesDepends on content sources and search scopes

Content SourcesContent Sources

Provide starting URLs for gathererProvide starting URLs for gathererCreate as many as you likeCreate as many as you like

Assign each to a content indexAssign each to a content index

Configuration based on typeConfiguration based on typeWeb site, Site Directory, File Share, Public Folder, NotesWeb site, Site Directory, File Share, Public Folder, Notes

Starting URLStarting URL

Crawl depth and hopsCrawl depth and hops

Folders/SubfoldersFolders/Subfolders

Three provided out of the box:Three provided out of the box:This portal, Site Directory, PeopleThis portal, Site Directory, People

This Portal Content SourceThis Portal Content Source

Lives in Portal_ContentLives in Portal_Content

All lists and libraries residing in All lists and libraries residing in portal areasportal areas

URL to portal site is starting pointURL to portal site is starting point

Unlimited page depthUnlimited page depth

Site Hop of 1Site Hop of 1Only Home page for links to external Only Home page for links to external contentcontent

However external content could be However external content could be indexed via Site Directory Content Sourceindexed via Site Directory Content Source

Site Directory Content SourceSite Directory Content Source

Contained in Non_Portal_ContentContained in Non_Portal_Contentsps://<server>/site$$$site/scope=$$$default$$$sps://<server>/site$$$site/scope=$$$default$$$

Is NOT the "Site Directory"!! Is NOT the "Site Directory"!! Two separate "lists"Two separate "lists"

Yellow Pages: List of sites for browsing through Sites Yellow Pages: List of sites for browsing through Sites portal areaportal area

This list indexed via “This Portal” content sourceThis list indexed via “This Portal” content source

Can contain reference to anything (team site, web site, Can contain reference to anything (team site, web site, application etc.)application etc.)

Sites to be crawled: list of sites whose content is to be Sites to be crawled: list of sites whose content is to be indexedindexed

Optionally updated during "Create a site"Optionally updated during "Create a site"

Manual addition also adds entry to Yellow Pages listManual addition also adds entry to Yellow Pages list

No ongoing link between listsNo ongoing link between lists

People Content SourcePeople Content Source

Lives in Portal_ContentLives in Portal_Content

sps:<server>//site$$$peoplesps:<server>//site$$$people

Crawls user profile and associated "My Site"Crawls user profile and associated "My Site"urn:schemas-microsoft-urn:schemas-microsoft-com:sharepoint:portal:profile:UserProfile_GUIDcom:sharepoint:portal:profile:UserProfile_GUID

May have to tweak content source if My Sites May have to tweak content source if My Sites are not relative to portal siteare not relative to portal site

Need to set site hop of 1Need to set site hop of 1

Shared Services ScenarioShared Services Scenario

Self provisioningSelf provisioning

See KB article 824615 See KB article 824615

Use exclusion rule to exclude My Site Use exclusion rule to exclude My Site contentcontent

Site Restrictions and RulesSite Restrictions and Rules

Override the default content access account when Override the default content access account when crawling a specific site or pathcrawling a specific site or path

Specify the granularity for crawling listsSpecify the granularity for crawling listsPer item or per listPer item or per list

Allow alerts on individual list itemsAllow alerts on individual list items

Allow crawling of complex links Allow crawling of complex links URL includes a question mark (?)URL includes a question mark (?)

Allow sites to be traversed for links without content Allow sites to be traversed for links without content being added to the indexbeing added to the index

For example an .asp page generating links to source itemsFor example an .asp page generating links to source items

Exclude an area from the index completelyExclude an area from the index completely

Search Groups and ScopesSearch Groups and Scopes

Each content index and content source are allocated Each content index and content source are allocated to a content source groupto a content source group

As content is crawled, two properties are updated in As content is crawled, two properties are updated in the indexthe index

urn:schemas.microsoft.com:fulltextqueryinfo:cataloggroupurn:schemas.microsoft.com:fulltextqueryinfo:cataloggroup

urn:schemas.microsoft.com:fulltextqueryinfo:sourcegroupurn:schemas.microsoft.com:fulltextqueryinfo:sourcegroup

Search scopes are constructed using portal areas Search scopes are constructed using portal areas and source groupsand source groups

urn:schemas-microsoft-com:sharepoint:portal:IDPathurn:schemas-microsoft-com:sharepoint:portal:IDPath

End users can select scope through UIEnd users can select scope through UISQL query modified accordinglySQL query modified accordingly

Dynamic scopes generatedDynamic scopes generated““This topic”, “This personal site” etc.This topic”, “This personal site” etc.

SQL Query and Search ScopesSQL Query and Search Scopes

One or more source groups in scopeOne or more source groups in scopeMultiple WHERE clauses OR’d together Multiple WHERE clauses OR’d together using cataloggroup and/or sourcegroupusing cataloggroup and/or sourcegroup

……WHERE WHERE (CONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo(CONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo:cataloggroup", 'Non-portal content') RANK BY :cataloggroup", 'Non-portal content') RANK BY COERCION(absolute, 1000) OR COERCION(absolute, 1000) OR CONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo:CONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo:Sourcegroup", 'People') RANK BY COERCION(absolute, Sourcegroup", 'People') RANK BY COERCION(absolute, 1000)) AND…1000)) AND…


One or more portal areas in scopeOne or more portal areas in scopeMultiple CONTAINS clauses for IDPath of Multiple CONTAINS clauses for IDPath of portal areaportal area

……WHERE WHERE (CONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo:catal(CONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo:cataloggroup", 'Non-portal content') RANK BY COERCION(absolute, oggroup", 'Non-portal content') RANK BY COERCION(absolute, 1000) OR 1000) OR CONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo:SourceCONTAINS("urn:schemas.microsoft.com:fulltextqueryinfo:Sourcegroup", 'People') RANK BY COERCION(absolute, 1000)) OR group", 'People') RANK BY COERCION(absolute, 1000)) OR (CONTAINS("urn:schemas-microsoft-(CONTAINS("urn:schemas-microsoft-com:sharepoint:portal:IDPath", '4833aac5-0ecc-4510-8a24-com:sharepoint:portal:IDPath", '4833aac5-0ecc-4510-8a24-d781b835ce45') RANK BY COERCION(absolute, 1000)) OR d781b835ce45') RANK BY COERCION(absolute, 1000)) OR (CONTAINS("urn:schemas-microsoft-(CONTAINS("urn:schemas-microsoft-com:sharepoint:portal:IDPath", ‘e905c21f-b9ce-4410-aa0e-com:sharepoint:portal:IDPath", ‘e905c21f-b9ce-4410-aa0e-f5d0fd509a72') RANK BY COERCION(absolute, 1000))) AND…f5d0fd509a72') RANK BY COERCION(absolute, 1000))) AND…


This topicThis topicCONTAINS clause for IDPath of current CONTAINS clause for IDPath of current portal areaportal area

Search.aspx?k=cask&s=This topic&w= Search.aspx?k=cask&s=This topic&w= (CONTAINS('urn:schemas-microsoft-(CONTAINS('urn:schemas-microsoft-com:sharepoint:portal:IDPath', 'e905c21f-b9ce-4410-com:sharepoint:portal:IDPath', 'e905c21f-b9ce-4410-aa0e-f5d0fd509a72') RANK BY aa0e-f5d0fd509a72') RANK BY COERCION(absolute,1000))&ct=1&rs=DistilleriesCOERCION(absolute,1000))&ct=1&rs=Distilleries


This personal siteThis personal siteUserProfile_GUID matches GUID of current userUserProfile_GUID matches GUID of current user

OROR urn:schemas.microsoft.com:fulltextqueryinfo:siteurn:schemas.microsoft.com:fulltextqueryinfo:sitename matches entry from user profilename matches entry from user profile

Search.aspx?k=cask&s=This personal Search.aspx?k=cask&s=This personal site&w='urn:schemas-microsoft-site&w='urn:schemas-microsoft-com:sharepoint:portal:profile:UserProfile_GUID'='16091com:sharepoint:portal:profile:UserProfile_GUID'='160916e1-1927-4d79-8530-002cb142b186' OR 6e1-1927-4d79-8530-002cb142b186' OR (CONTAINS('urn:schemas.microsoft.com:fulltextqueryinfo(CONTAINS('urn:schemas.microsoft.com:fulltextqueryinfo:sitename',’http://:678f23b8-dc36-498b-90a6-:sitename',’http://:678f23b8-dc36-498b-90a6-a2d17e0a6dd6/personal/harry') RANK BY a2d17e0a6dd6/personal/harry') RANK BY COERCION(absolute, 1000))&ct=3&rs=Harry Potter's Home"COERCION(absolute, 1000))&ct=3&rs=Harry Potter's Home"

Indexing Lists and LibrariesIndexing Lists and Libraries

Lists treated as single item for search resultsLists treated as single item for search resultsAll columns from all items make up the “body”All columns from all items make up the “body”

No property search possible in advanced search No property search possible in advanced search

Can change via a ruleCan change via a ruleBe aware how this will affect rankingBe aware how this will affect ranking

Individual list items are indexed for alert resultsIndividual list items are indexed for alert resultsPortal area lists onlyPortal area lists only

Libraries treated as individual itemsLibraries treated as individual items

No incremental change system for WSSNo incremental change system for WSS““Something has changed” is all we knowSomething has changed” is all we know

Not a problem for listsNot a problem for lists

LibrariesLibrariesOnly one ows_modified per itemOnly one ows_modified per item

So Document AND metadata re-indexedSo Document AND metadata re-indexed

Managing IndexesManaging Indexes

Update method and scheduleUpdate method and scheduleEach content source can have its own scheduleEach content source can have its own schedule

Four methods:Four methods:Full, adaptive, incremental, incremental (inclusive)Full, adaptive, incremental, incremental (inclusive)

CatalogsCatalogsCan specify location at creation timeCan specify location at creation time

Subsequently must use catutil to moveSubsequently must use catutil to move

Gatherer LogGatherer LogCan view and choose to log successes and exclusionsCan view and choose to log successes and exclusions

RulesRulesInclude/exclude pathsInclude/exclude paths

Allow linked content but not sourceAllow linked content but not source

Allow complex URLsAllow complex URLs

Control how SharePoint list items and alerts are handledControl how SharePoint list items and alerts are handled

Use different access accounts/methodsUse different access accounts/methods

File TypesFile TypesIf no IFilter then properties onlyIf no IFilter then properties only

Server Name Mappings, Site Hit Frequency, etc.Server Name Mappings, Site Hit Frequency, etc.

Managing MetadataManaging Metadata

SPS discovers metadata while crawling content SPS discovers metadata while crawling content sourcessources

Easy to manage custom, inconsistent, overlapping Easy to manage custom, inconsistent, overlapping metadatametadata

Site administrators can:Site administrators can:View discovered metadataView discovered metadata

Drill to examples of indexed contentDrill to examples of indexed content

Enable/disable indexing/retrievalEnable/disable indexing/retrieval

Show custom metadata in search resultsShow custom metadata in search results

Change attributes such as display nameChange attributes such as display name

Note – metadata is per portal siteNote – metadata is per portal siteConsider team sites using same named columns in a list!Consider team sites using same named columns in a list!

Or the "About Me" property…Or the "About Me" property…

Managing KeywordsManaging Keywords

Improved keyword best bet managementImproved keyword best bet managementView best bets by keywordView best bets by keyword

SynonymsSynonyms

Separate Thesaurus also availableSeparate Thesaurus also availableWeighting, Replacement Sets, Expansion SetsWeighting, Replacement Sets, Expansion Sets

Humans make search betterHumans make search betterKeyword and best bets emphasize key resultsKeyword and best bets emphasize key results

Site administrators manage keyword hierarchySite administrators manage keyword hierarchyNew keyword view lists all best betsNew keyword view lists all best bets

Portal users can suggest best betsPortal users can suggest best bets

IIS logging of query keywords allows IIS logging of query keywords allows tracking of frequent queriestracking of frequent queries

Enhanced Search ResultsEnhanced Search Results

Investment in search results usabilityInvestment in search results usabilitySearch results are fast, simple and powerfulSearch results are fast, simple and powerful

Simple: Keyword, boolean OR, PhraseSimple: Keyword, boolean OR, PhraseAdvanced: Property, date range, boolean AND, filtering by item Advanced: Property, date range, boolean AND, filtering by item type etc.type etc.

Supports Unicode, patterns, proximity, stemming, etc.Supports Unicode, patterns, proximity, stemming, etc.

More than documentsMore than documentsPeople, Teams, Sites and AreasPeople, Teams, Sites and AreasReconciles people names and aliasesReconciles people names and aliasesPivot or link to document info, people, areas and team sitesPivot or link to document info, people, areas and team sites

Easy to sort by site, area, author, dateEasy to sort by site, area, author, dateCollapse by site to see broad resultsCollapse by site to see broad results

Save searches to My LinksSave searches to My LinksDiacritic insensitivityDiacritic insensitivity

Try searching for “Big” in Danish and you’ll find some fish Try searching for “Big” in Danish and you’ll find some fish

SELECT "DAV:href", "DAV:displayname", "DAV:contentclass", "DAV:getlastmodified", SELECT "DAV:href", "DAV:displayname", "DAV:contentclass", "DAV:getlastmodified", "DAV:getcontentlength", "DAV:iscollection", "urn:schemas-microsoft-"DAV:getcontentlength", "DAV:iscollection", "urn:schemas-microsoft-com:sharepoint:portal:profile:WorkPhone", "urn:schemas-microsoft-com:sharepoint:portal:profile:WorkPhone", "urn:schemas-microsoft-com:sharepoint:portal:profile:WorkEmail", "urn:schemas-microsoft-com:sharepoint:portal:profile:Title", com:sharepoint:portal:profile:WorkEmail", "urn:schemas-microsoft-com:sharepoint:portal:profile:Title", "urn:schemas-microsoft-com:sharepoint:portal:profile:Department", "urn:schemas-microsoft-com:sharepoint:portal:profile:Department", "urn:schemas.microsoft.com:fulltextqueryinfo:PictureURL", "urn:schemas-microsoft-"urn:schemas.microsoft.com:fulltextqueryinfo:PictureURL", "urn:schemas-microsoft-com:office:office#Author", "urn:schemas.microsoft.com:fulltextqueryinfo:description", com:office:office#Author", "urn:schemas.microsoft.com:fulltextqueryinfo:description", "urn:schemas.microsoft.com:fulltextqueryinfo:rank", "urn:schemas.microsoft.com:fulltextqueryinfo:rank", "urn:schemas.microsoft.com:fulltextqueryinfo:sitename", "urn:schemas.microsoft.com:fulltextqueryinfo:sitename", "urn:schemas.microsoft.com:fulltextqueryinfo:displaytitle", "urn:schemas-microsoft-"urn:schemas.microsoft.com:fulltextqueryinfo:displaytitle", "urn:schemas-microsoft-com:publishing:Category", "urn:schemas-microsoft-com:office:office#ows_CrawlType", "urn:schemas-com:publishing:Category", "urn:schemas-microsoft-com:office:office#ows_CrawlType", "urn:schemas-microsoft-com:office:office#ows_ListTemplate", "urn:schemas-microsoft-com:office:office#ows_SiteName", microsoft-com:office:office#ows_ListTemplate", "urn:schemas-microsoft-com:office:office#ows_SiteName", "urn:schemas-microsoft-com:office:office#ows_ImageWidth", "urn:schemas-microsoft-"urn:schemas-microsoft-com:office:office#ows_ImageWidth", "urn:schemas-microsoft-com:office:office#ows_ImageHeight", "DAV:getcontenttype", "urn:schemas-microsoft-com:office:office#ows_ImageHeight", "DAV:getcontenttype", "urn:schemas-microsoft-com:sharepoint:portal:area:Path", "urn:schemas-microsoft-com:sharepoint:portal:area:Path", "urn:schemas-microsoft-com:sharepoint:portal:area:CategoryUrlNavigation", "urn:schemas-microsoft-com:sharepoint:portal:area:CategoryUrlNavigation", "urn:schemas-microsoft-com:publishing:CategoryTitle", "urn:schemas.microsoft.com:fulltextqueryinfo:sdid", "urn:schemas-com:publishing:CategoryTitle", "urn:schemas.microsoft.com:fulltextqueryinfo:sdid", "urn:schemas-microsoft-com:sharepoint:portal:objectid" microsoft-com:sharepoint:portal:objectid" FROM ( TABLE Portal_Content..Scope() UNION ALL TABLE Non_Portal_Content..Scope() ) FROM ( TABLE Portal_Content..Scope() UNION ALL TABLE Non_Portal_Content..Scope() ) WHERE WITH WHERE WITH ("DAV:contentclass":0,"urn:schemas.microsoft.com:fulltextqueryinfo:description":0,"urn:schemas.microsoft("DAV:contentclass":0,"urn:schemas.microsoft.com:fulltextqueryinfo:description":0,"urn:schemas.microsoft.com:fulltextqueryinfo:sourcegroup":0,"urn:schemas.microsoft.com:fulltextqueryinfo:cataloggroup":0,"urn:.com:fulltextqueryinfo:sourcegroup":0,"urn:schemas.microsoft.com:fulltextqueryinfo:cataloggroup":0,"urn:schemas-microsoft-com:office:office#Keywords":1.0,"urn:schemas-microsoft-schemas-microsoft-com:office:office#Keywords":1.0,"urn:schemas-microsoft-com:office:office#Title":0.9,"DAV:displayname":0.9,"urn:schemas-microsoft-com:office:office#Title":0.9,"DAV:displayname":0.9,"urn:schemas-microsoft-com:publishing:Category":0.8,"urn:schemas-microsoft-com:office:office#Subject":0.8,"urn:schemas-com:publishing:Category":0.8,"urn:schemas-microsoft-com:office:office#Subject":0.8,"urn:schemas-microsoft-com:office:office#Author":0.7,"urn:schemas-microsoft-microsoft-com:office:office#Author":0.7,"urn:schemas-microsoft-com:office:office#Description":0.5,"urn:schemas-microsoft-com:office:office#Description":0.5,"urn:schemas-microsoft-com:sharepoint:portal:profile:PreferredName":0.2,contents:0.1,*:0.05) AS #WeightedProps com:sharepoint:portal:profile:PreferredName":0.2,contents:0.1,*:0.05) AS #WeightedProps (("urn:schemas-microsoft-com:publishing:HomeBestBetKeywords"= some array ['islay'] RANK BY (("urn:schemas-microsoft-com:publishing:HomeBestBetKeywords"= some array ['islay'] RANK BY COERCION(absolute, 999)) OR (FREETEXT("urn:schemas-microsoft-COERCION(absolute, 999)) OR (FREETEXT("urn:schemas-microsoft-com:sharepoint:portal:profile:PreferredName", 'islay') RANK BY COERCION(multiply, 0.01)) OR com:sharepoint:portal:profile:PreferredName", 'islay') RANK BY COERCION(multiply, 0.01)) OR FREETEXT(#WeightedProps, 'islay') ) FREETEXT(#WeightedProps, 'islay') ) ORDER BY "urn:schemas.microsoft.com:fulltextqueryinfo:rank" DESCORDER BY "urn:schemas.microsoft.com:fulltextqueryinfo:rank" DESC

““Simple” Query Simple” Query (For reference only) (For reference only)

SharePoint Portal Server SharePoint Portal Server Search and Managing MetadataSearch and Managing Metadata

Screenshot of SPS Search for Screenshot of SPS Search for Reference PurposesReference Purposes

Shared ServicesShared Services

Allows services to be shared across Allows services to be shared across multiple portal sitesmultiple portal sites

SNAPS (with thanks to Michael Herman!)SNAPS (with thanks to Michael Herman!)SearchSearch

Notifications (Alerts)Notifications (Alerts)

AudiencesAudiences

ProfilesProfiles

Single Sign-OnSingle Sign-On

So a single search index…"shared” So a single search index…"shared” across multiple portal sites…across multiple portal sites…

Shared Services ComponentsShared Services Components

Parent portal siteParent portal siteContent IndexesContent Indexes

Content SourcesContent Sources

All management options except search scopesAll management options except search scopes

Child portal sitesChild portal sitesSearch scopesSearch scopes

Execute qualified searches against parent portal siteExecute qualified searches against parent portal site

Qualified searches?Qualified searches?Using IDPath of child portalUsing IDPath of child portal

Must create content sources on parent for Must create content sources on parent for childchild

Portal AreasPortal Areas

““Sites to be crawled” listSites to be crawled” list

Creating a Search HierarchyCreating a Search Hierarchy

Parent and child sites have no topological Parent and child sites have no topological associationassociation

““Use search scope from another portal site”Use search scope from another portal site”Associates child with "parent" in desired Associates child with "parent" in desired hierarchyhierarchy

On crawl IDPath gets set with values for all On crawl IDPath gets set with values for all "associated" portals (those higher up the "associated" portals (those higher up the hierarchy)hierarchy)

Qualified search on child portals finds Qualified search on child portals finds content lower down the hierarchycontent lower down the hierarchy

SiblingsSiblings

Simplified ExampleSimplified Example

ParentIDPath = A

PDoc

Child 1IDPath = B

C1Doc

Child 2IDPath = C

C2Doc

Child 3IDPath = D

C3Doc

Child 4IDPath = E

C4Doc

Child 5IDPath = F

C5Doc

Child 6IDPath = G

C6Doc

ItemItemPDocPDoc

IDPathIDPathAA

C6Doc GC6Doc G

C3Doc DC3Doc D

C1Doc BC1Doc B

C4Doc EC4Doc EC5Doc FC5Doc F

C2Doc CC2Doc C

,B,B,C,C,D,D

,B,B

,B,B

One Small IssueOne Small Issue

IDPath *only* stamped on content held IDPath *only* stamped on content held in portal area or team sitein portal area or team site

Therefore external content is not Therefore external content is not returned on child portal using returned on child portal using “All sources” search scope“All sources” search scope

Since child site qualifies the search with Since child site qualifies the search with IDPathIDPath

Therefore must modify All sources Therefore must modify All sources scope on child portal to include “sites scope on child portal to include “sites to be crawled” content sourceto be crawled” content source

AgendaAgenda






Team Site IntegrationTeam Site Integration

Automatic indexing of team sitesAutomatic indexing of team sites

Intelligent detection of team and portal Intelligent detection of team and portal sitessites

Efficient indexing of team sitesEfficient indexing of team sitesIncremental, secure, fastIncremental, secure, fast

Alerts for lists or list itemsAlerts for lists or list itemsIntegrated with Microsoft Office Outlook alertsIntegrated with Microsoft Office Outlook alerts

Team sites can share portal searchTeam sites can share portal searchNeed to associate team site with portal siteNeed to associate team site with portal site

Thumbnails for image librariesThumbnails for image libraries

Issues: WSS and SPSIssues: WSS and SPS

Search experience vastly differentSearch experience vastly differentSo do you enable both?So do you enable both?

If so remember resources!If so remember resources!What if WSS is using same SQL server as SPS?What if WSS is using same SQL server as SPS?

WSS search must be enabled to get link to WSS search must be enabled to get link to portal searchportal search

So if you don’t enable WSS search, how will So if you don’t enable WSS search, how will users actually execute a portal search from a users actually execute a portal search from a team site?team site?

Select portal area for document, list or librarySelect portal area for document, list or libraryMerely adds a portal listing with no ongoing link Merely adds a portal listing with no ongoing link

Customizing WSS SearchCustomizing WSS Search

Replace Search Box with portal Replace Search Box with portal search?search?

http://msdn.microsoft.com/library/defauhttp://msdn.microsoft.com/library/default.asp?url=/library/en-us/odc_SP2003_talt.asp?url=/library/en-us/odc_SP2003_ta/html/ODC_SPSSearchBoxwithPortalSe/html/ODC_SPSSearchBoxwithPortalSearchResults.asparchResults.asp

http://www.gotdotnet.com/community/uhttp://www.gotdotnet.com/community/usersamples/details.aspx?sampleguidsersamples/details.aspx?sampleguid=ec88db5d-3ebd-4aa0-aaa5-01a10338d8=ec88db5d-3ebd-4aa0-aaa5-01a10338d80000

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odc_SP2003_ta/html/ODC_SPSSearchBoxwithPortalSearchResults.asp




http://www.gotdotnet.com/community/usersamples/details.aspx?sampleguid=ec88db5d-3ebd-4aa0-aaa5-01a10338d800




Common RequestCommon Request

Search associated team sites from Search associated team sites from portal areaportal area

Is this possible out of the box?Is this possible out of the box?Not easilyNot easily

Multiple content sources, scopes requiredMultiple content sources, scopes required

But it was so close…But it was so close…IDPath of portal site is stamped on content from IDPath of portal site is stamped on content from associated team sitesassociated team sites

But can set associated portal site to an area…But can set associated portal site to an area…

……so if IDPath of portal area had been stamped so if IDPath of portal area had been stamped then a then a “This Topic” search would include team “This Topic” search would include team site’s contentsite’s content

Metadata: IssuesMetadata: Issues

WSS does NOT propagate inbuilt Office WSS does NOT propagate inbuilt Office properties (Title is an exception)properties (Title is an exception)

Custom columns are promoted (except through Custom columns are promoted (except through Upload document option!)Upload document option!)

SPS Index engine DOES crawl propertiesSPS Index engine DOES crawl propertiesAnd shows them in search resultsAnd shows them in search results

(remember WSS does NOT crawl properties)(remember WSS does NOT crawl properties)

Therefore you can search using Office Therefore you can search using Office properties via SPSproperties via SPS

Properties returned in search resultsProperties returned in search results

But not visible in source document libraryBut not visible in source document library

AgendaAgenda






Search ExtensibilitySearch Extensibility

Re-use search page via powerful query stringsRe-use search page via powerful query stringsOr replace with another search engine?Or replace with another search engine?

Customization of search web partsCustomization of search web partsVersatile search results control is used across many SPS V2 Versatile search results control is used across many SPS V2 portal featuresportal features

Look and feel parameters are exposedLook and feel parameters are exposed

Or replace with another search engine?Or replace with another search engine?

.NET-enabled Search.NET-enabled SearchManaged-code query provider and admin APIManaged-code query provider and admin API

Search web service for remote applicationsSearch web service for remote applications

Simple keyword or full-text SQL supportSimple keyword or full-text SQL support

Filter interface enables indexing of custom document formatsFilter interface enables indexing of custom document formats

Protocol Handler interface enables access to additional content Protocol Handler interface enables access to additional content storesstores

Wordbreaker interface published in Platform SDKWordbreaker interface published in Platform SDK

Calling the Search PageCalling the Search Page

Can call the search page from your own Can call the search page from your own applications via a URLapplications via a URL

Pass parameters to search.aspxPass parameters to search.aspxLink to portal search from team site Link to portal search from team site does thisdoes this

SDK + Look at various SPS linksSDK + Look at various SPS linkshttp://<server>/Search.aspx?http://<server>/Search.aspx?db=Kevin+Laahs&wd=Recent+documentsdb=Kevin+Laahs&wd=Recent+documents+by+Kevin+Laahs+by+Kevin+Laahs

Save search to My LinksSave search to My Links

Customizing the Search PageCustomizing the Search Page

4 Web Parts:4 Web Parts:Search Box, Search Results, Advanced Search, Search Box, Search Results, Advanced Search, Search MenuSearch Menu

/search.aspx?Mode=Edit&PageView=Shared/search.aspx?Mode=Edit&PageView=SharedUse UI or export/import exposed Web PartsUse UI or export/import exposed Web Parts

Modify/add properties, e.g.:Modify/add properties, e.g.:EnableSQLCommandLogging, SortByList, EnableSQLCommandLogging, SortByList, ColumnDisplayNames, ColumnDisplayNames, OpenNewWindowForMatchingItems, etc.OpenNewWindowForMatchingItems, etc.

Subclass and override search results Subclass and override search results behaviorbehavior

Programming SearchProgramming Search

Managed-code query providerManaged-code query providerAdmin and SearchAdmin and Search

Keyword or SQL Full Text QueryKeyword or SQL Full Text Query

Use for many purposesUse for many purposesUse profile as lookup directoryUse profile as lookup directory

Last documents publishedLast documents published

Search.asmx Web ServiceSearch.asmx Web ServiceCan add as a service to Office Research Can add as a service to Office Research PanePane

Enables keyword searchingEnables keyword searching

Research Pane in ActionResearch Pane in Action

Customizing and Customizing and Re-using SearchRe-using Search

SummarySummary

WSS and SPS provide fit for purpose WSS and SPS provide fit for purpose search technologiessearch technologies

Careful planning is required to ensure Careful planning is required to ensure best end-user experiencebest end-user experience

SPS Search is a service that can easily SPS Search is a service that can easily be used in your own applications be used in your own applications

Your FeedbackYour Feedbackis Important!is Important!Please Fill Out a Survey forPlease Fill Out a Survey forThis Session on CommNetThis Session on CommNet

PRT376 SharePoint Search Technologies Kevin Laahs Principal Consultant Advanced Technology Group HP...

Documents

Transcript of PRT376 SharePoint Search Technologies Kevin Laahs Principal Consultant Advanced Technology Group HP...