Search Engine - Nguyen Ba Thai Binh Duong 0208A1 v1.1

8/3/2019 Search Engine - Nguyen Ba Thai Binh Duong 0208A1 v1.1

1/42

ti: tm kim thng tin trn website (Search Engine)

Nguyn B Thi Bnh Dng 0208A1 Trang 1/42

LI CM N

u tin, chng em xin gi li cm n n Thy, C khoa Cng ngh Thng tin

Vin i hc Mnhin tn tnh dy d, du dt chng em sut bn nm i hc.

Chng em cm n thy Thi Thanh Tng, ngi tn tnh hng dn, gip ,

ng vin chngem hon thnh lun vn ny.

Thng 12 nm 2011

Sinh vin

Nguyn B Thi Bnh Dng


2/42



NHN XT CA GIO VIN HNG DN

....

Ngy thngnm 2011

K tn


3/42



NHN XT CA GIO VIN PHN BIN

....

Ngy thngnm 2011

K tn


4/42


5/42


6/42



M U

Trong thi i ngy nay, thng tin l nhu cu thit yu i vi mingi trn

milnh vc. Mipht tri qua hng triu triu trang web c y ln nhm lm giu

ngun ti nguyn v tn ny. Khi lng thng tin qu ln con ngi d b ngp chm

trong thng tin, m khng bit ta phi tm ci g, nh th no.

Vy u l gii php, cn c mt b my chuyn thng k tin tc, nh ch mc

h tr ngi dng tm kim d liu mt cch tt hn. V y cng l mc tiu ca

bi tp nhm n. Mc d c kh nhiu tiu ch tm kim cng nh c nhiu cch

nh ch mc v phn loi ni dung. Nhng trong phm vi bi tp ny ch nh chmc da trn tiu trang, url, th meta keyword v description.


7/42



Phn 1 : TM HIU VN

Chng 1:TNG QUAN V H THNG SEARCH ENGINE

1.Cc b phn cu thnh h thng search engine1.1 B thu thp thng tin Robot

Robot l mt chng trnh t ng duyt qua cc cu trc siu lin kt thu

thp ti liu & mt cch quy n nhn v tt c ti liu c lin kt vi ti liu ny.

Robot c bit n di nhiu tn gi khc nhau: spider, web wanderer hoc

web worm Nhng tn gi ny i khi gy nhm ln, nh t spider, wanderer lm

ngi ta ngh rng robot t n di chuyn v t worm lm ngi ta lin tng n

virus. V bn cht robot ch l mt chng trnh duyt v thu thp thng tin t cc site

theo ng giao thc web. Nhng trnh duyt thng thng khng c xem l robot do

thiu tnh ch ng, chng ch duyt web khi c s tc ng ca con ngi.

1.2 B lp ch mc Index

H thng lp ch mc hay cn gi l h thng phn tch v x l d liu, thc

hin vic phn tch, trch chn nhng thng tin cn thit (thng l cc t n , t

ghp , cm t quan trng) t nhng d liu m robot thu thp c v t chc thnh cs d liu ring c th tm kim trn mt cch nhanh chng, hiu qu. H thng

ch mc l danh sch cc t kho, ch r cc t kho no xut hin trang no, a ch

no.


8/42



1.3 B tm kim thng tinSearch Engine

Search engine l cm t dng ch ton b h thng bao gm b thu thp thng

tin, b lp ch mc & b tm kim thng tin. Cc b ny hot ng lin tc t lc khi

ng h thng, chng ph thuc ln nhau v mt d liu nhng c lp vi nhau v

mt hot ng.

Search engine tng tc vi user thng qua giao din web, c nhim v tip

nhn & tr v nhng ti liu tho yu cu ca user.

Ni nm na, tm kim t l tm kim cc trang m nhng t trong cu truy vn(query) xut hin nhiu nht

2.Nguyn l hot ngSearch engine iu khin robot i thu thp thng tin trn mng thng qua cc

siu lin kt (hyperlink). Khi robot pht hin ra mt site mi, n gi ti liu (web page)

v cho server chnh to c s d liu ch mc phc v cho nhu cu tm kim thngtin.

Bi v thng tin trn mng lun thay i nn robot phi lin tc cp nht cc site

c. Mt cp nht ph thuc vo tng h thng search engine. Khi search engine

nhn cu truy vn t user, n s tin hnh phn tch, tm trong c s d liu ch mc &

tr v nhng ti liu tho yu cu.


9/42



Chng 2:B THU THP THNG TINROBOT1.ng dng ca Robot

Robot thng c s dng cho nhng mc ch sau :

1.1 Phn tch, thng k Statistical Analysis

Robot u tin c dng m s lng web server, s ti liu trung bnh

ca mt server, t l cc dng file khc nhau, kch thc trung bnh ca mt trang web,

kt dnh,

1.2 Duy tr siu lin k - Maintenance

Mt trong nhng kh khn ca vic duy tr mt siu lin kt l n lin kt vi

nhng trang b hng (dead links) khi nhng trang ny b thay i hoc thm ch b xa.

Tht khng may vn cha c c ch no cnh bo cc b duy tr v s thay i ny.Trn thc t khi cc tc gi nhn ra ti liu ca mnh cha nhng lin kt hng, h s

thng bo cho nhau, hoc thnh thong c gi thng bo cho h bng email.

Mt s robot, chng hn MOMspider c th tr gip tc gi pht hin cc lin

kt hng cng nh duy tr cc cu trc siu lin kt cng ni dung ca mt trang web.

Chc nng ny lp li lin tc Mi khi mt ti liu c cp nht, nh mivn

xy ra s c gii quyt nhanh chng.


10/42



1.3 nh x a ch web - Mirroring

Mirroring l mt k thut ph bin trong vic duy tr cc kho d liu ca FPT.

Mt nh x (mirror) s sao chp ton b cu trc cy th mc v thng xuyn cp

nht nhng file b thay i. iu ny cho php nhiu ngi cng truy xut mt ngun

d liu, gim s lin kt b tht bi, nhanh hn v t chi ph hn so vi truy cp trc

tip vo site thc s cha cc d liu ny.

1.4 Pht hin ti nguynResource Discovery

C l ng dng th v nht ca robot l dng n pht hin ti nguyn. Con

ngi khng th kim sot ni mt khi lng thng tin khng l trong mi trng

mng. Robot s gip thu thp ti liu, to v duy tr c s d liu, pht hin v xo b

cc lin kt hng nu c, kt hp vi cng c tm kim cung cp thng tin cn thit cho

con ngi.

2.Robot ch mc Robot IndexingTrong qu trnh thu thp thng tin phc v cho b lp ch mc, ta cn gii quyt

nhng vn sau :

Mt l : Trong mi trng mng, robot ly thng tin t cc site. Vy robot s

bt u t site no ? iu ny hon ton ph thuc vo robot. Mirobot khc nhau s

c nhng chin lc khc nhau. Thng th robot s ving thm cc site ph bin hoc

nhng site c nhiu lin kt dn n n.

Hai l: Ai s cung cp a ch ca cc site ny cho robot ?

C 2 ngun:


11/42


12/42



3.Chin thut thu thp d liuTrc khi cc trang web c nh ch mc, tt c cc trang web phi c ly

v my ca robot. ly c tt c cc trang web, robot phi c chin thut. T mt

s trang web c sn, robot lc ra danh sch cc lin kt, ri t d tm cc trang khc.

3.1 Chin thut tm kim

(1)Nhp thng tin site cn index(2)Xc nhn cho php index(3)Mi ln index ly ngu nhin 1 url cn index trong top 10 URL tr v

(c th ch nh site cn index)

(4)Gi request ly d liu(5)Lc thng tin d liu(6)Cp nht thng tin vo database(7)Trch xut danh sch lin kt trong ni dung tr v(8)Lu danh sch link trch xut c vo h thng. Nu link bt u

bng http:// hoc https://th trch xut ly domain v kim tra domain

c trong h thng cha.

a. Nu c to url vi siteid c trong h thng.b. Nu cha to site vo to url khi u /

(9)Kt thc chu trnh index, quay li bc 1


13/42



4.Nhng vn cn lu ca web robot

4.1 Chi ph v him ho

Vic s dng cc Robot tn kh nhiu chi ph, c bit l khi chng c iu

khin t xa trn internet. Phn ny chng ta s cng tho lun v nhng him ho do

robot gy ra.

4.1.1Qa ti mng v server Network resource and server loadSau mt khong thi gian di, thng l mt thng, robot s bt u hot ng

mt cch lin tc. tng tc nhiu robot c phng ra ng thi do cn c bng

thng ln. Ti nguyn mng b khai thc qu mc khi robot yu cu mt lng ln

thng tin trong khong thi gian qu ngn (rapid fire). Kt qu l thiu bng thng cho

nhng ng dng khc. Server va phi phc v yu cu ca robot va cung cp dch

v cho user, do yu cu ca robot tng ln bao nhiu th dch v s gim xung by

nhiu. Tc gi ca mt con robot th nghim bng cch cho thi hnh 20 lt truy

cp ng thi vo server ca anh ta. Nhng lc robot thu thp thng tin, server b chmli. Trong vng mt tun robot ving thm site ny vi mt yu cu kinh khng. Ch

sau 170 lt truy xut lin tc, th nghim tht bi do server b qu ti.

Rapid fire thc s l thm ho. Hiu qu truyn ti thng tin dng ny bng giao

thc web hay HTTP st gim thy r. Nhng giao thc mi ang c xem xt nhm

cu vn tnh th.

4.1.2Cp nht qu mc- Updating overheadNgi ta cho rng cc c s d liu do web robot to ra c th c cp nht t

ng nhng cho n thi im ny vn cha c c ch kim sot s thay i trn web


14/42



mt cch hiu qu. Cp nht thng tin rt quan trng nhng qa thng xuyn l iu

khng cn thit.

Xut pht t thc t HTTP a ra k thut if ModifiedSince gip cc

useragent xc nh c thi im ti liu thay i. Robot pht hin iu ny ch khi

n lu li cc thng tin c nhng s tn nhiu b nh & cn d liu phc tp.

Mt trong nhng c tnh ph bin ca robot l kh nng tip nhn cc t cn

tm trong khi vn thu thp d liu. Tuy nhin mt s ngi cho rng c tnh ny

khng ng hoan nghnh bi hai l do:

u tin, cc tc v tm kim ca ngi s dng cu i (end - user) gp phny server vo ch qu ti.

Th hai, khng c c s m bo c mi quan h gia cc t cn tm, ngchnh t v ti u i vi c s d liu. V d, nu b tm kim khng h tr

cc ton t boolean, mt user cn d liu v xe my mun c c thng tin

ng thay v nhp vo cm t Ford and garage phi nhp vo t car.

Nhng ngi khng h thc c iu ny.

Mt kha cnh nguy him na bt ngun t s nh hng sai lm ca end

user. Mt s ngi s dng cng c ca mnh rt tt nh d on c lng ti liu

ln nht c th c, bit chnh xc ni cn tm d liu, gii hn thi gian s dng robot,

trong khi mt s khc li lm dng kh nng ca robot mt cch v tnh hoc c .

V vy cc tc gi vit robot ngh ch nn phn pht sn phm ca mnh cho nhng

end-user hiuc web robot v nhng kh khn trong mi trng mng.

4.1.3Nhng tnh hung khng mong i Bad implementations


15/42



Thay v kim tra trn my cc b trc, mt s tc gi ln u tin vit robot

cho th ngay trn cc server thc s, iu ny lm au u khng t nh qun tr web

(web master).

Truy xut trng lp c th xy ra khi robot khng lu li du vt nhng ni n

i qua hoc n khng nhn din c cc URL mc d khc nhau v tn nhng li

cng dn n mt a ch, v d a ch DSN & IP.

i khi, robot lng ph thi gian v ti nguyn ch thu v nhng ti liu m

sau phi vt i. V d h thng ch quan tm n file vn bn (text file) nhng robot

li nhn c nhng loi file khc nh file hnh nh, file thc thi,

Chng 3:B LP CH MC INDEX

1.Khi qut v h thng lp ch mcCc trang Web sau khi thu thp v s c phn tch, trch chn nhng thng tin

cn thit (thng l cc t n, t ghp, cm t quan trng) lu tr trong c s d

liu nhm phc v cho nhu cu tm kim sau ny.


16/42



Chng 4:B TM KIM THNG TINSEARCH ENGINE

1.V sao ta cn mt cng c tm kim (SE) ?Tng tng ta mun tm vi quyn sch trong mt th vin rt ln. Vi sc

lc c nhn ta khng th xem qua ht tt csch, v vy ta cn mt danh mc sch.

Tng t, tn ti hng triu trang web trn th gii v Mipht tri qua s lng c

y ln cng nhiu hn, cho d ta c trong tay mt cng c lt web tuyt vi n ucng khng th duyt ht. Tuy nhin, vi s tr gip ca SE, ta c th thm ch xc

nh c v tr ca nhng t cn tm trong cc trang web khp ni trn th gii.

2.Cc phng thc tm kim2.1 Tm theo t kho Keyword searching

y l phng php c p dng vi hu ht cc search engine. Tr khitc gi

ca trang web xc nh t kha cho ti liu ca mnh, ngc li iu ny ph thuc vo

search engine. Nh vy cc search engine s t mnh chn v nh ch mc cho nhng

t m chng cho quan trng c th gip phn bit cc ti liu khc nhau. Cc t c

cp trong phn II chng II hoc cc t lp li nhiu ln u c ch . Mt s site

lp ch mc cho tt c cc t c trong mt trang web, mt s khc ch chn mt s

on vn bn.


17/42



Cc h thng nh ch mc trn ton vn bn (full -text indexing systems) m

s ln xut hin ca Mit trong ti liu ngoi tr cc t stopword. C nhng cng c

tm kim cn phn bit c ch hoa ln ch thng.

2.2 Nhng kh khn khi tm theo t kho

Search engine thng gp rc ri vi nhng t ng m khc ngha (v d hard

cider, hard stone, a hard exam, hard drive) hoc nhng t c cc bin th khc nhau do

c tin t v hu t nh big, bigger, student, students, . Bn cnh search engine

cng khng th tr v cc ti liu cha nhng t ng ngha vi cc t trong cu truy

vn.

2.3 Tm theo ng ngha Concept-based searching

Excite tng ni ting vi chin thut tm theo ng ngha nhng gi y chin

thut ny khng cn c s dng na. Khng ging cc h thng tm theo t kho, h

thng tm theo ng ngha s on mun ca ngi dng thng qua cu ch. Tm

theo ng ngha hot ng da trn hnh thc gom nhm ti liu, phc tp hn th da

vo ngn ng hc, cc thuyt v tr tu nhn to. Excite tip cn da vo phng php

tnh ton bng cch m s ln xut hin ca cc t quan trng. Khi nhiu t hoc

nhng cm t c ngha t gn nhau trong ti liu th Excite s cho rng chng ang

m ch mt ch no .

V d, khi t heart ng gn cc t nh attack (cn au tim), blood (s

sng), stroke (s say nng), th search engine s xp nhng trang cha cc t ny voch y hc v sc kho. Ngc li, khi t heart ng gn cc t flowers, candy,

th search engine s xp nhng trang cha cc t ny vo ch tr tnh.


18/42



3.Cc chin lc tm kimMingi u nhn xt rng web l ni m ta lun c c thng tin v bt k

ch g. Nhng kt qu cui cng thng l lng ph thi gian cho nhng URL v

ch. Do n lc ta ngh n cc chin lc tm kim.

Ta khi u vi mt ng thng tin trn mt ch kh rng? Hoc ta hnh

dung c c th nhng g cn tm? Hay ta mun tm a ch ca ai ?

Nu phm vi quan tm ca ta qu rng, ta nn xem xt cc th mc web (web

directory). Nu sau ta thu hp phm vi cn tm, hy xem xt vic la chn mt cngc tm kim thch hp.

3.1 Tm thng tin vi cc th mc ch

Ging nh tm sch trong th vin, cn nhc gia tm theo tc gi, tiu , ch

, ta thng chn ch c th bao qut mt vng thng tin rng hn.

V d : ta munto mt trang ch (home page) nhng khng bit cch vit mtfile HTML nh th no, thm ch cha tng to mt file nh, v cng khng bit cch

y mt trang ln mng. Tm li ta cn nhng thng tin cho mt ch kh rng -

xut bn mt trang web (web publishing).

Khi hon ton xc nh mnh cn tm nhng g ta nn bt u t mt th mc

web nh th mc ca Yahoo hoc Google,v th mc web tp trung nhiu vo ch

ang c quan tm hn l mt cng c tm kim.

Gn y cc web site thng kt hp thmc web v cc cng c tm vi nhau.

V d nu ta s dng Google tm thng tin v mt trong nhng kt qu ny nm


19/42



trong th mc web ca Google, Google s cung cp cho ta mt lin kt dn vo th

mc.

3.2 Tm thng tin vi cc cng c tm kim

Mt s cng c tm kim gp rc ri vi d liu u vo ca ngi dng. V d :

nhng t cha cc k t c bit nh C++, nhng t cha stopword nh to be or not

to be.Xt v d khc t r rng hn. Gi s ta l mt ngi rt thch tiu thuyt trinh

thm v mun tm nhng trang ni v cc tc gi yu thch. Nu n gin ch nhp vo

cc t mystery v writer, phn ln cc search engine s tr v cc lin kt dn n

cc trang cha mt trong 2 t trn hoc c 2. Nh vy c kh nng hng trm, thm ch

hng ngn URL khng mong mun. Tuy nhin nu ta nhp vo 1 cm t, kt qu s

kh quan hn.

3.3 Ti u cu truy vn

Rt nhiu search engine p dng cc ton t Boolean (Boolean operators) hoc

cc b nh v tr (proximai locators) ti u cu truy vn. l cc t kho sau:

STT T kha ngha

1 AND / php ton + Mit trong cu truy vn phi c trong ti liu

2 OR Ti liu cha t nht mt t cn tm


20/42



3 NOT / php ton - Ti liu khng cha [cc] t sau t kho

Bng 4.1 : Cc t kha gip ti u cu truy vn


21/42



Chng 5:MT S SEARCH ENGINETHNG DNG TRN THGII V VIT NAM

Vi nt v cc c trng ca mt s search engine thng dng trn th gii

Search

Engine

Google AlltheWeb AltaVista Teoma

Database google.com alltheweb.com altavista.com teoma.com

Kch thc(#

trang)

Khong 4 t (1

t khng nh

ch mc trn

ton vn bn)

Khong 3 t,

ch mc trn

ton vn bn.

Khong 1 t Khong 1 t

a phng

tin

(multimedia)

H tr H tr H tr Khng h tr

Ton t

Mc nh AND AND AND AND

Loi tr - - - -


22/42


23/42



filetype:

link:site:

Trong search

nng cao :

cache:info:

link.extension: anchor:applet: afterfate:

Cc c tnh

c bit

~ tm t ng

ngha

Gii hn bingn ng

Nhiu kiu file

: pdf, doc,

Caches : trang

web khi nh

ch mc

Duyt qua cc

URL

Trong tmnng cao :

gii hn bi

ngy, domain,

a ch iP

Gii hn bi

ngy, v tr,

ngn ng

Trong tm

nng cao : s

dng sortby

lc v sp xp

kt qu.

Dng refine

ti u kt qu.

Resource cc cc trang

v lin kt tp

trung trn ch

cn tm.

u im

u im

chnh

Rt tt vi

nhng trang c

ph bin

cao.

Cc trang tin

tc gn y

Tt nh

Google.

Khng c stop

word.

Dng nhiu

ton t

Boolean trong

tm kim.

Trong tm

nng cao h

tr hin th kt

Tnh ph

bin tt, da vo

s lng trang

web cng ch

vi cc trang

ang xt.

Thng t kt


24/42



qu theo

ph bin ca

t.

qu ng khch

l.

Search

Engine

Google AlltheWeb AltaVista Teoma

Bng 5.1 : Bng hng dn nhanh v cch s dng cc search engine ph bin

trn th gii

Search

engine

C s d liu Ton t Lc chn tm

kim

Linh tinh

Google

http://www.g

oogle.com

H tr tm

kim nng

cao

H thng th

mc ch

(Subject

Directory)

H thng th

Ton vn bn

ca cc trang

web, .pdf,

.doc, .xls, .ps,

.wpd

(4.3B, + 1B

mt phn ca

ch mc

URLs)Tin tc: cp

nht thng

AND (mc

nh)

OR (danh t

ring)

+ cho cc stop

word thng

dng, cho cc

URL hoc cc

trang c th (vd +edu)

- loi tr

Dng * rt

gn.

Dng tm cm

t.

fields : intitle:,

inurl:, link:, site:

Tm trn h

thng danh mc

cc ch trong

th mc web.

Tm cc trang

Kim li

chnh t.

Lu tr cc

trang lp

ch mc.

Tt cho tm

cc trang hay

b li 404.

Phin dch

n 5 ngn

ng.
http://www.google.com/http://www.google.com/http://www.google.com/http://www.google.com/http://www.google.com/


25/42



mc m

(Open

Directory)

xuyn (4500

ngun ).

Cc dng file

nh

Nhm :

Usenet t

1981 n nay

web tng t. ~ tm t ng

ngha.

AlltheWeb

http://allthew

eb.com

H tr tm

kim nng

cao

Ton b vn

bn cc trang

web, .pdf,

Flash,

(3.1B ton b

ch mc

URLs)

Tin tc : cp

nht thng

xuyn (3000

ngun)

Tranh nh

Video

Audio

AND (mc

nh)

OR, phi t

cc t trong

du .

ANDNOT,

RANK

- loi b

Khng rt gn.

Dng du cho

cm t.

Field intitle:inurl:

link:site:

Trong tm nngcao :

gii hn theo

ngy, ngn ng,

domain, file

format, a ch

iP.

Kim li

chnh t.

Tm nng cao

: tranh nh,

video.

H tr s

dng k thut

clusters

ti u cu

truy vn.
http://alltheweb.com/http://alltheweb.com/http://alltheweb.com/http://alltheweb.com/http://alltheweb.com/


26/42



FPT

AltaVistahttp://altavist

a.com

H tr tm

kim nng

cao

H thng thmc ch

(Subject

Directory )

H thng th

mc m

(Open

Directory)

Ton b vnbn cc trang

web (khong

1B) v file

.pdf.

Tin tc (3000

ngun), nh,

MP3/Audio,

Video.

AND (mcnh)

Trong tm nng

cao hoc danh

t ring trong

tm c bn :

AND, OR,

ANDNOT,

NEAR, du ()

lng nhau.

- cho loi tr.

Du * rt gn.Du cho cm

t.

Tm nng cao :

gii hn ngy,

ngn ng.

Kim lichnh t.

Phin dch : 8

ngn ng ca

Chu u &

cc ngn ng

ca Chu .

AltaVistaPri

ma : ti u

cu hi.

Teoma

http://teoma.c

om

H tr tm

kim nngcao

Ton b vn

bn trang web

(khong 1B)

AND (mc

nh)

OR (danh t

ring)

+ hoc chostopword

- loi b

Khng rt gn.

Dng du cho

cm t.

Field intitle:inurl:

site:geoloc:lang:l

ast:

afterdate:befored

Kim li

chnh t.

Gom nhm

kt qu

Refine tiu cu hi.

Resource
http://altavista.com/http://altavista.com/http://altavista.com/http://teoma.com/http://teoma.com/http://teoma.com/http://teoma.com/http://teoma.com/http://altavista.com/http://altavista.com/


27/42



ate:

betweendate:

Trong tm nng

cao :

gii hn theo

ngy, ngn ng,

domain, file

format, a ch

iP.

c cc trang

hoc lin kt

tp trung voch .

AskJeeves

www.ask.co

m

Nhn kt qu

t CSDL ca

Teoma.

Tm sn phm

:

PriceGrabber.

com,

Tm tranh nh

:

Picsearch.co

m

Tm tin tc :

Moreover.co

Ging Teoma.

i vi nhng

cu hi n

gin, xut hin

ca s i

thoi.

Ging Teoma.

Click vo

Remove Frame

thy URLs

ca cc trang.

Kim li

chnh t.
http://www.ask.com/http://www.ask.com/http://www.ask.com/http://www.ask.com/http://www.ask.com/


28/42



m.

AskJeeves forKids

www.ajkids.c

om

Tr li tt cccu hi n

gin.

Games cho

tr em,

Tin tc theo

tng nhm

tui.

Hi bng ngnng t nhin.

Khng s dng

cc ton t

Boolean.

Click voNoframes thy

URL ca trang

kt qu.

Dn n cctrang phc v

hc tp : t

in, vt l,

khoa hc, bn

, lch s,

Bng 5.2 : S lc v cc c trng ca mt s search engine thng dng trn

internet

Meta-searchengine

C s dliu

Ton t Lc chn tmkim

Linh tinh

Vivisimo

http://vivisi

mo.com

Netscape,

MSN,

Lycos,LookS

mart,

AND(mc

nh), OR, -

Tm trn ch :

tin tc, thng

mi, k thut,

th thao.

Gom nhm kt

qu.

Tt i vi ch

v cc s kin

& nhiu kha

cnh khc.
http://www.ajkids.com/http://www.ajkids.com/http://www.ajkids.com/http://vivisimo.com/http://vivisimo.com/http://vivisimo.com/http://vivisimo.com/http://vivisimo.com/http://www.ajkids.com/http://www.ajkids.com/


29/42



Dopgpile

http://dopgpi

le.com

Google,

Yahoo,

AltaVista,

Teoma/AskJe

eves,

About.com,

FAST,

FindWhat,

LookSmart

Tm nng cao :

AND, OR,

ANDNOT.

Sp xp theo kt

qu.

Xo cc kt qu

trng.

Kim li chnh

t.

Highlight t cn

tm trong kt

qu.

Gom nhm kt

qu.

Tt i vi ch

v ccs kin

& nhiu kha

cnh khc.

Bng 5.3 : Cc meta-search engine thng dng trn internet

Th mc ch

C s d

liu

Ton t Lc chn tm

kim

Linh tinh

Yahoo

http://dir.yaho

o.com

Xem xt cc

trang web

(khong

13K)

AND(mc

nh)

OR

-

Cm t :

Rt gn : *

Fields t: title,

u:URL

Nhiu dch v

trong Yahoo:

Tin tc : tng

gi.

Th thao :t

s,..
http://dopgpile.com/http://dopgpile.com/http://dopgpile.com/http://dir.yahoo.com/http://dir.yahoo.com/http://dir.yahoo.com/http://dir.yahoo.com/http://dir.yahoo.com/http://dopgpile.com/http://dopgpile.com/


30/42



Bn , thi

tit, mua sm.

Academic

info

http://academ

icinfo.net

Mc cao

ng hoc

nghin cu

(cng hu

ch cho

trung hc).

c chn

v ch thch

bi th th

Michael

Madin

(khong

25K)

AND, OR(mc

nh), NOT,

du () lng

nhau.

Dn n cc

chng trnh

mc cao

ng hoc cc

site, cc ngun

ti nguyn

khc hu chcho sinh vin.

Bng 5.4 : Cc h thng th mc theo ch thng dng trn internet

1.1 Th mc ca Yahoo, Google

V bn cht l cc danh mc ch . Sp xp cc trang theo mc quan trng ca chng. Tm theo ti hoc ch .
http://academicinfo.net/http://academicinfo.net/http://academicinfo.net/http://academicinfo.net/http://academicinfo.net/


31/42



Google l mt trong nhng cng c tm kim mi nhng nhanh chng c a

chung nh kh nng tm nhanh v chnh xc. tng chnh ca cng c ny l o

lng quan trng ca mt trang da vos lin kt n trang . Ni cch khc nunhiu webmaster cng quan tm n mt website th website xng ng c nh

gi cao. Yahoo tng dng Google nh mt chc nng ca mnh trong mt thi gian

di. K thut tm kim ca Microsoft v MSN.com da trn k thut inktomi. (inktomi

c p dng cho mt trong nhng search engine ni ting trong nhng nm 90 l

Hobot) Microsoft rt n lc trong vic tm ra mt k thut cho ring mnh nhng

vn cha thnh cng.

1.2 Alltheweb

Alltheweb cng l mt trong nhng cng c tm kim mi, c cho l cng c

d tm nhanh hn v hiu qu hn cc search engine khc nh mt lng ch mc rt

ln. Alltheweb c s dng bi Yahoo.

1.3 AltaVista

tng l mt trong nhng cng c tm kim c a chung nht nhng b

nh bi bi Google. Mc d vy n vn l mt search engine cho kt qu chnh xc v

tng c Yahoo s dng.

1.4 Lycos

c m t nh l nhng cng truy cp web (web portal) hay nhng trung tmtruy cp, l ni m ngi dng i vo ly thng tin cho milnh vc, k c tn gu,

gi th in t,


32/42



Phn 2 : THIT K V CI T

Ngn ng lp trnh : C#, HTML Cng c lp trnh : Visual Studio Web Server : IIS H qun tr CSDL : Mircrosoft SQL Server 2008

Chng 1:THIT K D LIUKhi thit k h thng tm kim thng tinvn kh khn nht phi i mt l t

chc cu trc d liu . V khi lng d liu phi lu tr ca h thng tm kim

thng tin l rt ln, v khi lng yu cu tm kim phi x l cng rt ln (trn mi

trng Web) nn cu trc d liu phi c t chc ti u cho vic p ng (v thi

gian) i vi yu cu tm kim ca ngi s dng.

1.C s d liu trong SQLCSDL trong SQL server phc v cho ton b h thng gm bng

Sites: cha thng tin v site

Bng

UrlSTT

Tn thuc tnh ngha Kiu d liu Min gi tr Ghi ch


33/42


34/42



10 ModifiedId Ngi cp

nht ln

cui

11 IndexDescription Thng tin

m t thm

trong qu

trnh index

12 TotalLike Tng sngi thch

13 TotalClick Tng s

click

Bng 1.0: Bng Sites

UrlIndexs : cha cc thng tin cn thit cho ng dng v 1 URL.

Bng

Url

STT

Tn thuc tnh ngha Kiu d liu Min gi tr Ghi ch

1 Id M duy

nht

xc nh

Char(32) Kha

chnh


35/42


36/42



11 Modified Thi gian

cp nht

ln cui

12 ModifiedId Ngi cp

nht ln

cui

13 IndexDescription Thng tin

m t thmtrong qu

trnh index

14 TotalLike Tng s

ngi thch

15 TotalClick Tng sclick

Bng 1.1: Bng UrlIndex


37/42



Chng 2:THU THP THNG TIN

1.Cu trc d liuVi mong mun khng ch m bo c cc nhim v ca web robot m cn

gip cho qun tr ch ng hn na trong cng vic ca mnh, module web robot s h

tr nhng chc nng sau:

URL bt u (StartURL) nh su lin kt Cc tu chn khi phn tch mt URL: cng site, cng th mc gc,

khc site.

Project Mt site c th c nhiu url. Mi Url trong site l duy nht Ch phn tch URL to nh index.

H thng Phin bn hin ti ch c mt spider index v l window application.

2.X l ca web robot Module chuyn i nh index va nh index va t ng ly danh sch

link lu vo hthng


38/42



Khng nh hng n hot ng ca cc module khc.Nhn chung quy trnh hot ng ca web robot u ging nhau. Tuy nhin, cc

ng dng h tr nhng tnh nng khc nhau s c s thay i v quy trnh hot ng.

Cc bc x l:

(1)Khi to.(2)Ly url cha c nh index c trng thi approved(3)Request ly thng tin(4)C bt c li no cp nht li v trng indexdescription(5)Phn tch ni dungly title, keyword, description, danh sch lin kt(6)Cp nht index(7)Lu danh sch link. Qu trnh ny c kim tra trng. Nu url c http

hoc https th t ng phn tch to site sau thm vo url

(8)Kt thc.

3.Cc thut ton phn tch cu trc file HTML3.1 Thut ton ly lin kt

to mt lin kt trong file HTML ngi ta thng dng mt trong cc

dng sau :

Tn th Thuc tnh kt hp


39/42


40/42



Chng 3:TM KIM THNG TIN

S dng Fulltext search ca SQL server tm kim thng tin

Bn cnh h tr cho qu trnh tm kim, h thng s dng b phn tch c php

tm kim nng cao tnh tin dng

Biu thc tm kim c nhp t cc t n l hoc cc cm t, cc cm t c t

trong du nhy kp ("). Cc t v cm t c phn cch nhau bi du cch.

ch nh t hoc cm t bt buc phi c trong kt qu, t du cng (+) pha

trc.

ch nh t hoc cm t bt buc s khng nm trong kt qu, t du tr (-) pha

trc.

ch nh site s dng t kha site:

V d: iu kinsite:fsviet.com cng ngh +"gii php" -nano

H thng s truy vn cc kt qu ca site: fsviet.com

c cha t cng

hoc cha t ngh,

ng thi phi cha cm gii php

v khng c cha t nano.


41/42



Chng 4:CC MODULE, PACKAGE, LP CHNH CACHNG TRNH

1.Cc module, package ca chng trnhSTT Module ngha

1 Dng.SearchEngine.Objects Cha cc object d liu h tr x l

data. Ngoi ra cn c thm mt s class

h tr mt s hm utility

2 Dng.SearchEngine.DataHelper Giao tip vi database

3 Dng.SearchEngine Giao tip vi ngi dng cui tm

kim thng tin

4 Dng.SearchEngine.Win Gi nhim v index d liu

5 Dng.SearchEngine.Service Phin bn window service cha hon

thin. d kin sau ny s dng window

service nh index ch k dng

application


42/42

Search Engine - Nguyen Ba Thai Binh Duong 0208A1 v1.1

Documents

Transcript of Search Engine - Nguyen Ba Thai Binh Duong 0208A1 v1.1