Russ Houberg - SharePoint 2013 - Search Architecture
-
Upload
kartik-anand -
Category
Documents
-
view
33 -
download
0
description
Transcript of Russ Houberg - SharePoint 2013 - Search Architecture
-
Presentedby:
-
Introduction
RussHoubergSharePointMCSMKnowledgeLakeProfessionalServices
[email protected]/rhoubergtwitter/rhouberg
-
Up for discussion
Abriefhistory Concepts:SP2010 SP2013 Concepts:NewFunctionality FASTItsinthere ComponentTopologyOverhaul SchemaManagement PowerShellItsUnavoidable! ScalableArchitectureExamples ECMImpact Questions?
-
SharePoint2007 FASTESP
CustomConnector
SharePoint2010 FASTSearchforSharePoint2010
SearchSSAisFASTAware TheBypass:EnterpriseSearch FS4SP EnterpriseSearch
SharePoint2013 EnterpriseSearchw/FASTDNA
Hybridvigorbringsthebestofbothworlds
EnterpriseSearch(andECM)arestillgoingtobegenerallyonpremise(notcloud)
Hybridvigor
A brief history
-
Concepts: SP2010 SP2013Translating(orreplacing)thesyntax
SharePoint2010 SharePoint2013
ManagedProperty (Multiple) SearchSchemas
BestBets PromotedResults(QueryRule)
Scope andFederatedLocation ResultSource
ContentByQuery Content BySearch
Incremental Crawl ContinuousCrawl
Mirrored IndexPartition IndexReplica
MCM MCSM
-
Concepts: New FunctionalityNowwithbetterstuff
-
Concepts: New Functionality
CrawlingProcessingChanges
WenowhaveFull,IncrementalandContinuousCrawls
NomoreHostDistributionRules
NomoreFASTBypass
Nowwithbetterstuff
-
Concepts: New Functionality
ContinuousCrawlBenefits
Nomorewaitingforindexmerge
Doesnotwaitforothercrawlstocomplete
Canhavemultiplecontinuouscrawlsrunningsimultaneously
Continuouscrawlsignoreserrors
Continus CrawlFacts
Runsevery15minutesbydefault
DefaultintervalcanbechangedwithPowerShell
ShouldbeusedinsteadofincrementalcrawlsforSharePointcontentsources
Nowwithbetterstuff
-
Concepts: New Functionality
OtherBackendChanges
Schemamanagementatsitecollectionlevel
Crawldatabaserebalancing
ResultSources&RemoteSPIndex
Nowwithbetterstuff
-
Concepts: New Functionality
QueryRules QueryRuletoimplement
PromotedResults(bestbets) QueryRuletoaugmentsearch
resultswithentitymetadata(fromSPorBCS)
QueryRuletomanipulateresultsranking
Nowwithbetterstuff
-
Concepts: New Functionality
DEMOContentBySearch WebPart
Nowwithbetterstuff
-
FAST Its in thereFASTbitsarenowblendedin
-
FAST Its in there
ComponentsoftheFASTcrawlprocessing,indexstructure,relevanceandrankingmodels,entityextraction,andpipelineextensibilityoptions(throughwebservices)areallinfluencesofFASTtechnology.
FASTbitsarenowblendedin
-
Component Topology OverhaulBreakingoutoftheshell
-
Component Topology Overhaul
WhatitDoes
Crawlscontentsourcestopopulateindex
Deliverscrawlitems(binary)andmetadatatocontentprocessor
Invokesconnectorsorprotocolhandlerstointeractwithcontentsourcestoretrievedata
Usesoneormorecrawldatabasestostoreinfoaboutcrawlitemsandcrawlhistory
ImportantFacts
Wecanhavemultiplecrawlcomponents
MSRecommends:2CrawlComponentsperSearchServiceApplication
MSRecommends:8(4vm)CPU/8GBRAMperCrawlComponent
CrawlComponent
-
Component Topology Overhaul
WhatitDoes
Processescrawlitemsandfeedstoindexcomponent
Transformscrawlitemsintoartifactsthatcanbeincludedinsearchindex(Performsdocumentparsingandpropertymapping)
Writesinformationaboutlinksandurls inlinkdatabase(whichareanalyzedbyanalyticstocalculaterelevanceandcurrency Resultswrittenbacktosearchindexbycontentprocessingcomponent
Generatesphoneticnamevariationstoimprovepeoplesearch
ImportantFacts
Wemustonlyhaveone(1)crawlprocessingcomponentperserver morewillhurt,nothelpcrawlperformance
Maxof2persearchserviceapplication FeedingSessionsarescaledbasedonCPUcoresusingadefaultcoefficientof3
8(cores)*3=24feedingsessions4(cores)*3=12feedingsessions
MSRecommends:8(4vm)CPU/8GBRAMperContentProcessingComponent
FeedingsessionsrequireRAM MoreRAMisnecessarywhenmorecoresarepresent monitoringrequired
ContentProcessingComponent
-
Component Topology Overhaul
WhatitDoes Runsanalyticsjobsthatanalyzecrawlitemsanduserinteractionwithsearchresultstoperformbothsearchanalyticsandusageanalytics
AnalyzesLink&Anchortextanalysis,Cleardistance,SearchClicks,DeepLinks,SocialTags,SocialDistance,SearchReports,Recommendations,UsageCounts,ActivityRanking
Improvessearchrelevanceandcreatesearchresults
Outputincludedinsearchindexbycontentprocessor
ImportantFacts Maximumof6persearchserviceapplication AddmoreAnalyticsProcessingComponentstoimproveanalyticsperformance
MSRecommends:8(4vm)CPU/8GBRAM/300GBdiskspaceperAnalyticsProcessingComponent.
InteractswithAnalyticsReportingtostorestatisticalinformation
InteractswithLinkdatabasetostoreinformationaboutsearchesandcrawleddocuments
AnalyticsProcessingComponent
-
Component Topology Overhaul
WhatitDoes
Receivesprocesseditemsfromcontentprocessingcomponentandwritestheitemstotheindexfile
Receivesqueriesfromthequeryprocessingcomponentandreturnsresultsets
RedistributescontentamongindexpartitionswhenindexarchitectureischangedbySearchAdministrationComponent
ImportantFacts
Maximumof60indexpartitions(20indexpartitionsX3indexreplicas)persearchserviceapplication
MustprovisiononeIndexComponentforeachindexreplica.
MSRecommends:8(4vm)CPU/16GBRAM/500GBdiskspaceperIndexComponent.
IndexComponent
-
Component Topology Overhaul
Indexpartitionislogicalportionofentiresearchindex(sameasbefore)
Indexpartitionisservedbyoneormoreindexcomponents Indexcomponentscanbeprimary"replica"orsecondary"replica" PrimaryReplicaiscontactedbycontentprocessingcomponenttowritenewdataintheindex
SecondaryReplicaisreadonlycopythatgetupdatedwiththedata. Addingreplicasimprovesqueryperformanceunderload Addpartitionstohandleincreasedcontentcorpus Can'tremovepartitionafterithasbeenadded.
IndexArchitecture
IndexPartitionsandReplicas
-
Component Topology Overhaul
WhatitDoes
Analyzesandprocessesqueriesandresults
Afterreceivingaquery,itanalyzesandprocessesthequerytooptimizeprecision,recallandrelevance
Submitsprocessedqueriestotheindexcomponent
Processestheresultsetreturnedbytheindexcomponentbeforereturningtothequeryingentity.
ImportantFacts
Maximumof1perserver MSRecommends:8(4vm)CPU/8GBRAMperQueryProcessingComponent.
QueryProcessingComponent
-
Schema Management Schemachanges Granularconfiguration Canbeextendedatthesite
collectionlevelormanagedgloballybyfarmadmin
Fullcrawlneededtopickupchanges
ItsallaboutSchemanow
-
PowerShell Its unavoidableYournewbestfriend
MINIDEMOWhatyoucantdowiththeUI
-
PowerShell Its unavoidableGetSPEnterpriseSearchStatusNewSPEnterpriseSearchAdminComponentGetSPEnterpriseSearchCrawlContentSourceGetSPEnterpriseSearchCrawlCustomConnectorGetSPEnterpriseSearchCrawlDatabaseGetSPEnterpriseSearchCrawlExtensionGetSPEnterpriseSearchCrawlMappingGetSPEnterpriseSearchCrawlRuleNewSPEnterpriseSearchCrawlComponentNewSPEnterpriseSearchCrawlContentSourceNewSPEnterpriseSearchCrawlCustomConnectorNewSPEnterpriseSearchCrawlDatabaseNewSPEnterpriseSearchCrawlExtensionNewSPEnterpriseSearchCrawlMappingNewSPEnterpriseSearchCrawlRuleRemoveSPEnterpriseSearchCrawlContentSourceRemoveSPEnterpriseSearchCrawlCustomConnectorRemoveSPEnterpriseSearchCrawlDatabaseRemoveSPEnterpriseSearchCrawlExtensionRemoveSPEnterpriseSearchCrawlMappingRemoveSPEnterpriseSearchCrawlRuleSetSPEnterpriseSearchCrawlContentSourceSetSPEnterpriseSearchCrawlDatabaseSetSPEnterpriseSearchCrawlRuleSetSPEnterpriseSearchCrawlLogReadPermissionRemoveSPEnterpriseSearchCrawlLogReadPermissionRemoveSPEnterpriseSearchCrawlLogReadPermissionImportSPEnterpriseSearchTopologyExportSPEnterpriseSearchTopologySetSPEnterpriseSearchTopologyRemoveSPEnterpriseSearchTopologyRemoveSPEnterpriseSearchComponentNewSPEnterpriseSearchTopologyNewSPEnterpriseSearchQueryProcessingComponentNewSPEnterpriseSearchIndexComponent
NewSPEnterpriseSearchContentProcessingComponentGetSPEnterpriseSearchTopologyGetSPEnterpriseSearchServiceGetSPEnterpriseSearchServiceApplicationGetSPEnterpriseSearchServiceApplicationProxyGetSPEnterpriseSearchServiceInstanceNewSPEnterpriseSearchServiceApplicationNewSPEnterpriseSearchServiceApplicationProxyRemoveSPEnterpriseSearchServiceApplicationRemoveSPEnterpriseSearchServiceApplicationProxyRestoreSPEnterpriseSearchServiceApplicationResumeSPEnterpriseSearchServiceApplicationSetSPEnterpriseSearchServiceSetSPEnterpriseSearchServiceApplicationSetSPEnterpriseSearchServiceApplicationProxyStartSPEnterpriseSearchServiceInstanceStopSPEnterpriseSearchServiceInstanceSuspendSPEnterpriseSearchServiceApplicationUpgradeSPEnterpriseSearchServiceApplicationBackupSPEnterpriseSearchServiceApplicationIndexUpgradeSPEnterpriseSearchServiceApplicationSiteSettingsRestoreSPEnterpriseSearchServiceApplicationIndexRemoveSPEnterpriseSearchServiceApplicationSiteSettingsGetSPEnterpriseSearchOwnerSuspendSPEnterpriseSearchServiceApplicationSetSPEnterpriseSearchServiceInstanceGetSPEnterpriseSearchMetadataCategoryGetSPEnterpriseSearchMetadataCrawledPropertyGetSPEnterpriseSearchMetadataManagedPropertyGetSPEnterpriseSearchMetadataMappingNewSPEnterpriseSearchMetadataCategoryNewSPEnterpriseSearchMetadataCrawledPropertyNewSPEnterpriseSearchMetadataManagedPropertyNewSPEnterpriseSearchMetadataMappingRemoveSPEnterpriseSearchMetadataCategoryRemoveSPEnterpriseSearchMetadataManagedPropertyRemoveSPEnterpriseSearchMetadataMappingSetSPEnterpriseSearchMetadataCategory
SetSPEnterpriseSearchMetadataCrawledPropertySetSPEnterpriseSearchMetadataManagedPropertySetSPEnterpriseSearchMetadataMappingGetSPEnterpriseSearchPropertyRuleCollectionGetSPEnterpriseSearchPropertyRuleGetSPEnterpriseSearchQueryAndSiteSettingsServiceGetSPEnterpriseSearchQueryAndSiteSettingsServiceInstanceGetSPEnterpriseSearchQueryAndSiteSettingsServiceProxyGetSPEnterpriseSearchQueryAuthorityGetSPEnterpriseSearchQueryDemotedGetSPEnterpriseSearchQueryKeywordGetSPEnterpriseSearchQueryScopeGetSPEnterpriseSearchQueryScopeRuleGetSPEnterpriseSearchQuerySuggestionCandidatesGetSPEnterpriseSearchRankingModelGetSPEnterpriseSearchSecurityTrimmerNewSPEnterpriseSearchQueryAuthorityNewSPEnterpriseSearchQueryDemotedNewSPEnterpriseSearchQueryKeywordNewSPEnterpriseSearchQueryScopeNewSPEnterpriseSearchQueryScopeRuleNewSPEnterpriseSearchRankingModelNewSPEnterpriseSearchSecurityTrimmerRemoveSPEnterpriseSearchQueryAuthorityRemoveSPEnterpriseSearchQueryDemotedRemoveSPEnterpriseSearchQueryKeywordRemoveSPEnterpriseSearchQueryScopeRemoveSPEnterpriseSearchQueryScopeRuleRemoveSPEnterpriseSearchRankingModelRemoveSPEnterpriseSearchSecurityTrimmerSetSPEnterpriseSearchQueryAuthoritySetSPEnterpriseSearchQueryKeywordSetSPEnterpriseSearchQueryScopeSetSPEnterpriseSearchQueryScopeRuleSetSPEnterpriseSearchRankingModelStartSPEnterpriseSearchQueryAndSiteSettingsServiceInstanceStopSPEnterpriseSearchQueryAndSiteSettingsServiceInstanceImportSPEnterpriseSearchPopularQueries
SetSPEnterpriseSearchResultItemTypeSetSPEnterpriseSearchQuerySpellingCorrectionRemoveSPEnterpriseSearchResultItemTypeNewSPEnterpriseSearchResultItemTypeImportSPEnterpriseSearchThesaurusGetSPEnterpriseSearchResultItemTypeGetSPEnterpriseSearchQuerySpellingCorrectionGetSPEnterpriseSearchLanguageResourcePhraseGetSPEnterpriseSearchSiteHitRuleNewSPEnterpriseSearchLanguageResourcePhraseNewSPEnterpriseSearchSiteHitRuleRemoveSPEnterpriseSearchLanguageResourcePhraseRemoveSPEnterpriseSearchSiteHitRuleGetSPEnterpriseSearchVssDataPathGetSPEnterpriseSearchContentEnrichmentConfigurationSetSPEnterpriseSearchPrimaryHostControllerSetSPEnterpriseSearchLinguisticComponentsStatusSetSPEnterpriseSearchContentEnrichmentConfigurationRemoveSPEnterpriseSearchContentEnrichmentConfigurationNewSPEnterpriseSearchContentEnrichmentConfigurationGetSPEnterpriseSearchLinguisticComponentsStatusGetSPEnterpriseSearchHostControllerSetSPEnterpriseSearchLinksDatabaseRepartitionSPEnterpriseSearchLinksDatabasesMoveSPEnterpriseSearchLinksDatabasesRemoveSPEnterpriseSearchTenantSchemaRemoveSPEnterpriseSearchTenantConfigurationRemoveSPEnterpriseSearchLinksDatabaseRemoveSPEnterpriseSearchFileFormatNewSPEnterpriseSearchLinksDatabaseNewSPEnterpriseSearchFileFormatNewSPEnterpriseSearchAnalyticsProcessingComponentImportSPEnterpriseSearchCustomExtractionDictionaryGetSPEnterpriseSearchLinksDatabaseGetSPEnterpriseSearchFileFormatSetSPEnterpriseSearchFileFormatStateGetSPEnterpriseSearchComponentGetSPEnterpriseSearchServiceApplicationBackupStore
Yournewbestfriend
-
Scalable Architecture Examples
SmallSearchFarm(~10MillionItems)|4PhysicalServers|6VirtualServers
-
Scalable Architecture Examples
MediumSearchFarm(~40MillionItems)|8PhysicalServers|14VirtualServers
-
Scalable Architecture Examples
LargeSearchFarm(~100MillionItems)|18PhysicalServers|28VirtualServers
-
Scalable Architecture Examples
???
ExtremeSearchFarm(>100MillionItems)
-
ECM Impact
Schemacanbemanagedbysiteadmins,reducingtheloadonsearchadministrator
Schemaisconfiguredtoallowmoregranularity(query,retrieve,refine,sort,etc) Affectscontentindexsize
Remoteresultsourcescanbecrawledlocallyandthenqueriedbyremotefarms.HugeimpactongeodistributedsearchThirdpartysolutionsmaybeabletofacilitateblendedsearchresults!
Individualitemscanberecrawledeasily
AutomaticURLbalancingincrawldatabasesminimizeshostnamerestrictionsforlargearchiverepositories
Scalabilitylimitchangeswillhaveabigimpactonfarmdesignforlargearchivecontentrepositoriesinthenearfuture.
Whatdoesitallmean?
-
Questions?
StumptheChump(youprobablycan)
-
Thankyou!
The end
Search ArchitectureIntroductionUp for discussionA brief historyConcepts: SP2010 SP2013Concepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityFAST Its in thereFAST Its in thereComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulSchema ManagementPowerShell Its unavoidablePowerShell Its unavoidableScalable Architecture ExamplesScalable Architecture ExamplesScalable Architecture ExamplesScalable Architecture ExamplesECM ImpactQuestions?The end