Managing & Troubleshooting

SharePoint Search

#SPSJEThomas Vochten@thomasvochten

IT Pro - Level 200

About Me

Thomas Vochten SharePoint MVP. Platform architect. Speaker. Trainer. Involuntary DBA. Architect at Xylos. V-TSP at Microsoft.

@thomasvochtenhttp://thomasvochten.commail@thomasvochten.com

Objectives

• Understanding the search architecture• Creating a search service application• Managing the search topology• Troubleshooting the search topology

IT Pro - Level 200

Search ArchitectureBack to basics

Architecture

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Services & Processes

• SharePoint Search Host Controller Service

hostcontrollerservice.exe

Controls & checks other components

• SharePoint Server Search Service

mssearch.exe & mssdmn.exe

Crawl component

Services & Processes

• All other components

noderunner.exe

• Search Query and Site Settings ServiceWCF Service in IIS

searchservice.svc

Crawl Component

Crawl

ContentProcessing Index

Crawl

HTTP

File Shares

SharePoint

User Profiles

Exchange

Lotus Notes

Documentum

Custom (BCS)

...Admin

mssearch.exe

SearchAdmin

FAST Search Index

Crawl Component

ULS Category: CrawlerProcesses: mssearch.exe & mssdmn.exe

• Manages the crawl queue• Protocol handlers• State lives in the crawl database• Uses a local cache • Full/Incremental/Continuous

Crawl Database

Content Processing Component

Crawl

ContentProcessing Index

Crawl AdminAnalyticsProcessing

Link

Content EnrichmentWeb Service

SearchAdmin

FAST Search Index

Content Processing Component

ULS Category: Content ProcessingProcess: noderunner.exe

• Document parsing • Metadata extraction• Stateless• Extensible

Index Component

Replica

ReplicaReplica

Replica

Content Processing

Replica

Partition #1

Replica

Replica

Replica

Partition #2

Replica

Replica

Replica

Partition #3

Replica

Each replica is an index component

Index Component

ULS Category: IndexProcess: noderunner.exe

• Inserts content into binary index• Also stores security info & metadata• Indexes live on disk only• Uses partitions & replica’s

Query Component

UXQueryProcessing WFEIndex

SearchAdmin

Admin

FAST Search Index

API

Query Processing Component

ULS Category: QueryProcess: noderunner.exe

• Handles user queries• A lot of business logic• Stateless• Query Analyzer• Security Trimming

Query Path

• Web Server

• Search Query & Site Settings Service (HTTP endpoint)

• Query Processing Component (net.tcp endpoint)

• Index Component (net.tcp endpoint)

Analytics Component

Crawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Link

Analytics Reporting

Event Stream

Analytics Component

ULS Category: AnalyticsProcess: noderunner.exe

• Intensive computations• Links, social interactions, views, recommendations, …• Gets better over time• Also handles health statistics in central admin

Admin Component

SearchAdmin

Admin

Admin Component

ULS Category: AdminProcess: Administration

• Stores configuration objects• Web service for administering configuration objects• Management via PowerShell only!• Only one active but can be made redundant• Application Server Admin Service timer job

Components Placement

Search is probably the most resource intensive SharePoint service

Good Practice:

• CPC & APC together• Admin & Crawl together• Index & Query together

Start Here

Minimal highly available search architecture

Scale Out

Resource Requirements

Component CPU Network Disk Memory

Search administration * * *Crawler ** *** *** **Content processing (CPC) *** ** ***Analytics processing (APC) ** *** * **Index *** ** *** ***Query processing (QPC) * ** *

Databases Recap

• Administration DB• Crawl DB• Links DB• Analytics Reporting DB

SetupBuilding a Search Service Application

What you need

• A managed account• An application pool

Consider other service applications:• Managed Metadata• User Profiles• State Service• Usage & Health

PowerShell to the rescue

• Central Admin will give you nothing by GUIDs• New-SPEnterpriseSearchServiceApplication

Creation Process

• Check system requirements• Create Search Service Application• Create Search Topology• Create Components• Activate Search Topology

Default Topology

• 1 admin component• 1 crawl component• 1 content processing component• 1 index component• 1 query component• 1 analytics component

All components will be installed on a single server

Demo

Creating a search service applicationCreating a basic search topology

Manage the Search ComponentsGet to know your search topology

General

• Topology is stored in the Search Service Application• There is no user interface as in SharePoint 2010• You can deploy all components at once across the

network

Typical Operations

• Add search components• Move search components• Remove search components• Add additional search servers

New-SPEnterpriseSearch...ComponentRemove-SPEnterpriseSearchComponent

Modify the existing topology

• Make sure crawls are in idle state• Clone Search Topology• Add Components• Activate Search Topology (can take a while!)• Wait for a green light

About Cloning

Clone the active search topology• You cannot modify an active topology

Topology Activation Process• Check for valid topology first

Topology Activation in progress• Topology A is marked as Activating• Topology B is marked as Deactivating • The deactivating topology is the one serving search.• At this point your active current topology is still Topology B• Topology A is marked Active

(Re)moving Components

• Clone Search Topology• Add new components• Activate Search Topology• Wait for the green light• Clone Search Topology again• Remove components• Activate Search Topology• Wait for the green light

Index Operations

• Adding a replica to existing partition 0

New-SPEnterpriseSearchIndexComponent … -IndexPartition 0

• Adding a new partition (can’t remove!)

New-SPEnterpriseSearchIndexComponent … - IndexPartition 1

Demo Scenario

Server A has all search components.We want to make all components redundant

1. Clone the topology on server A2. Start search service instance on server B3. Modify the cloned topology4. Activate the cloned topology5. Verify the new topology

Demo

Make all search components redundantMove components to dedicated servers

Troubleshooting 101Topology & Component Failures

Search Validation

• Windows Services

Host Controller (on every server)Search Service (on crawl servers)

• Search Topology page

Verify everything is green

Search Validation

• Crawl Test

Add documentsFull crawl

• Query Test

Search CenterTry “*”

Search Validation

• Get-SPEnterpriseSearchStatus• Custom health scripts

Get-SPSearchTopologyState.ps1

http://thvo.me/spsearchstatus

Search Health

• Green (ok) – sit back and relax• Yellow (degraded) – sit back and wait• Red (failure) – manual intervention needed

Approach when it breaks

• Asking questions• Looking at the crawl/query logs• Looking at the ULS logs• Health Status in Central Administration• Validate processes & services• Diagnostics scripts

Useful tools

• ULSViewer http://thvo.me/ulsviewernew

• SharePoint 2013 Search Query Toolhttp://thvo.me/querytool

• Fiddlerhttp://thvo.me/getfiddler

Some Tips

• Don’t be overwhelmed

• Always make components redundant, especially Admin

• Admin component redundancy should be priority

• Pauze & Stop search before patching SharePoint!http://thvo.me/slowpatch

Some Tips

• Boosting logging levels to Verbose of VerboseEx

• You can follow crawled documents in the ULS

• You can follow issued queries in the ULS (single correlationid)

• Get to know logging categories

Importance of the Admin ComponentDown?

• No search administration possible• No search monitoring possible• No backup or restore• No topology changes possible• No new crawls will start• Queries will still work

Common Problems

• Disk space on the index servers (master merge)• Disk space on the analytics servers (temp space)• Disk space on the crawl servers (temp space)• Proxy servers preventing crawl• Domain Administrator service accounts• Missing Antivirus Exclusions• Not indexing the default zone• Stuck or long running crawls• Database problems (blocking, perf, space)

Common Problems

• Networking problems• Endpoints not being available• Problems with custom code (e.g. content enrichment

svc)• WCF endpoints

Search Data Locations

C:\Program Files\Microsoft Office Servers\15.0\Data

Analytics Temporary StorageCrawl Temporary StorageIndex files

Demo

Get-SPSearchTopologyState.ps1Something went wrong… when querying

Summary

• Get to know the individual search components• Start with a redundant implementation from the start• Learn the basic PowerShell commands• Keep your favorite tools around when troubleshooting• First things first: don’t neglect the logfiles• Rebuilding your service application should be your last

resort, not your first

Questions?

@thomasvochtenmail@thomasvochten.com

DisclaimerMost of the graphics in this presentation were created by Microsoft

Thank You!

Raffle Prizes at 16:00

1st #SharePint in the Hotel Bar sponsored by Cloudshare 16:30