Human Language Technology

March, 2001

Feb, 2002

Human Language Human Language TechnologyTechnology

Gary Geunbae LeeGary Geunbae Lee

Intelligent Software Lab. POSTECHIntelligent Software Lab. POSTECH

Gary G. Lee, Postech

contentscontents

What is HLT? - definition/history/application?What is HLT? - definition/history/application? HLT workshop case study – acl/sigir/hlt conferencesHLT workshop case study – acl/sigir/hlt conferences Towards Technology synergy – 21c frontier projectTowards Technology synergy – 21c frontier project


Goals of the HLTGoals of the HLT

Computers would be a lot more useful if they could handle our Computers would be a lot more useful if they could handle our email, do our library research, talk to us …email, do our library research, talk to us …

But they are fazed by natural human language.But they are fazed by natural human language.

How can we make computers have abilities to handle human How can we make computers have abilities to handle human language? (Or help them learn it as kids do?)language? (Or help them learn it as kids do?)


A few applications of HLTA few applications of HLT

Spelling correction, grammar checking …Spelling correction, grammar checking … Better search enginesBetter search engines Information extraction, gistingInformation extraction, gisting Psychotherapy; Harlequin romances; etc.Psychotherapy; Harlequin romances; etc.

New interfaces:New interfaces: Speech recognition (and text-to-speech) Dialogue systems (USS Enterprise onboard computer) Machine translation; speech translation (the Babel tower??)

Trans-lingual summarization, detection, extraction …Trans-lingual summarization, detection, extraction …


Levels of LanguageLevels of Language

Phonetics/phonology/morphology:Phonetics/phonology/morphology: what words (or subwords) what words (or subwords) are we dealing with? are we dealing with?

Syntax:Syntax: What phrases are we dealing with? Which words mod What phrases are we dealing with? Which words modify one another?ify one another?

Semantics:Semantics: What’s the literal meaning? What’s the literal meaning? Pragmatics:Pragmatics: What should you conclude from the fact that I sai What should you conclude from the fact that I sai

d something? How should you react?d something? How should you react?


What’s hard – ambiguities, ambiguities, all What’s hard – ambiguities, ambiguities, all different levels of ambiguitiesdifferent levels of ambiguities

John stopped at the John stopped at the donut storedonut store on his way home on his way home from workfrom work. He . He thoughtthought a coffee was good a coffee was good every few hoursevery few hours. But . But itit turned out to be turned out to be too expensivetoo expensive there. [from J. Eisner] there. [from J. Eisner]

- donut: To get a donut (doughnut; spare tire) for his car?- donut: To get a donut (doughnut; spare tire) for his car?

- Donut store: store where donuts shop? or is run by donuts? or looks - Donut store: store where donuts shop? or is run by donuts? or looks like a big donut? or made of donut?like a big donut? or made of donut?

- From work: Well, actually, he stopped there from hunger and - From work: Well, actually, he stopped there from hunger and exhaustion, not just from work.exhaustion, not just from work.

- Every few hours: That’s how often he thought it? Or that’s for coffee?- Every few hours: That’s how often he thought it? Or that’s for coffee?

- it: the particular coffee that was good every few hours? the donut - it: the particular coffee that was good every few hours? the donut store? the situationstore? the situation

- Too expensive: too expensive for what? what are we supposed to - Too expensive: too expensive for what? what are we supposed to conclude about what John did?conclude about what John did?


Ubiquitous computingUbiquitous computing

Ubiquitous computingUbiquitous computing Pervasive computingPervasive computing Third paradigm computingThird paradigm computing Calm technologyCalm technology Computing everywhereComputing everywhere Invisible computingInvisible computing Irobot style interface – human language + hologram??Irobot style interface – human language + hologram??


Intelligent service robotIntelligent service robot

Remote speech input

reverberation Robot noiseEnvi noise


Telematics – Eye busy and hand busyTelematics – Eye busy and hand busy

GPS

Telematics devicePDACar Controller

CDMA

Voice Portal for

Email, VAD and

Internet Contents

PDA

Information Center


Smart HomeSmart Home

“대장금”하는 채널을 보여줘

네, 녹화할까요?

아줌마 : 요즘 이영애 나오는 인기있는 드라마가 뭐지 ?

DTV: MBC 에서 방영중인 대장금입니다 . 아줌마 : 대장군 재방송 어디서 해 ? DTV: 지금은 방송중이 아니고 , 채널 36 에서

오후 2 시에 방영예정입니다 . 아줌마 : 그럼 , 그거 녹화해 줘 . DTV: 네 , 알겠습니다 .


Research projectsResearch projects Communicator, TRIPS, COMIC, DIPPER, TRINDI, GALAXY, SMARTKOM, Verbmobil,

DUMAS, FASiL, EARS

Platforms for development and researchPlatforms for development and research CSLU, JASPIS, TRINDIKIT, SUEDE, WITAS, SpeechBuilder, …

Conferences and workshopsConferences and workshops ICSLP, Eurospeech, ICASSP, SigDial, …

JournalsJournals Computer speech and language, Speech Communication, IJHCS, IEEE Trans. SAP

(speech and audio processing)

ASR community (speech/ signal processing) [from McTear]


Research projectsResearch projects SAM/PAM, Pen treebank, TAG, GATE, MUC, TIPSTER, TDT, TIDES, etc

Platforms for development and researchPlatforms for development and research Alembic, Alvey, Gate, LingPipe, Collins parser, Jasen, postag/K, …(see NLP

software registry)

Conferences and workshopsConferences and workshops ACL, EACL, ANLP, COLING, IJCNLP, EMNLP…

JournalsJournals Computational Linguistics, Natural Language Engineering, ACM TALIP, IJCPOL,

Computers and Humanities…

NLP community (AI-NLP, Ling – CL)


Research projectsResearch projects SMART, Digital Libraries, TREC, NTCIR, etc

Platforms for development and researchPlatforms for development and research SMART, MG, Lemur, Z-PRIZE, etc

Conferences and workshopsConferences and workshops ACM SIGIR, AIRS, ACM CIKM, JCDL, ASIST,..

JournalsJournals IPM, JASIST, Information systems, …

IR community (Library science)


Long History of FundingLong History of Funding

long research history since 1960long research history since 1960’’ss significant research results due to constant funding (e.g. DAsignificant research results due to constant funding (e.g. DA

RPARPA’’s 20 years of funding) --- ready for practical solutions 20 years of funding) --- ready for practical solution five main desiderata for practical app?five main desiderata for practical app?

Integration at proper level of analysis/understanding Combination of appropriate modality Based on real examples (corpora) Towards multi-lingual applications Thorough evaluation (usability)


contentscontents

What is HLT? - definition/historyWhat is HLT? - definition/history

HLT workshop case study – darpa hlt examplesHLT workshop case study – darpa hlt examples Towards Technology synergy – 21c frontier projectTowards Technology synergy – 21c frontier project


History notes common technologiesHistory notes common technologies

HMM, SVM, CRF, MEMM, DBN for POS tagging, ASR, parsing, HMM, SVM, CRF, MEMM, DBN for POS tagging, ASR, parsing, prosody modeling, statistical MT, etcprosody modeling, statistical MT, etc

Tf/idf, n-gram, discounting/smoothing for sentence weighting, Tf/idf, n-gram, discounting/smoothing for sentence weighting, retrieval models, summarization, question answering, etcretrieval models, summarization, question answering, etc

Bayesian network, causal network, graphical models for inforBayesian network, causal network, graphical models for information retrieval, topic detection, dialog, task planning, etcmation retrieval, topic detection, dialog, task planning, etc

Trie indexing, tree indexing, caching for ASR pronunciation mTrie indexing, tree indexing, caching for ASR pronunciation modeling, morphological lexicon, IR indexing, TTS G2P, etcodeling, morphological lexicon, IR indexing, TTS G2P, etc

Common themes Common themes statistical language modeling and machin statistical language modeling and machine learning and empirical evaluation (glass box/black box)e learning and empirical evaluation (glass box/black box)


Written language vs. spoken Written language vs. spoken language?language?

CommonalitiesCommonalities Human languages

DifferencesDifferences Punctuation vs. prosodic cues Disfluencies vs. linguistic competency Recognition errors

I canned meat at eleven ten then okI canned meat at eleven ten then ok I can’t / meet at eleven ten then? / okI can’t / meet at eleven ten then? / ok


IT839: new technology for economy IT839: new technology for economy growth needs synergy?growth needs synergy?

8 new services 8 new services

- WiBro - DMB - - WiBro - DMB - Home Network Home Network

- Telematics - RFID application- Telematics - RFID application - W-CDMA - ground D - W-CDMA - ground DTV - internet telephony (VoIP) TV - internet telephony (VoIP)

3 new infrastructures 3 new infrastructures

- (BcN) - u-sensor network - IPv6 - (BcN) - u-sensor network - IPv6 9 new growth technology 9 new growth technology

- new mobile communication - new mobile communication - digital TV broadcasting- digital TV broadcasting

- Home Network- Home Network

- IT SOC - IT SOC - next generation PC- next generation PC - embedded SW- embedded SW

- digital contents(DC) - telematics - intelligent service robo- digital contents(DC) - telematics - intelligent service robotsts


NLP/IR/speech merge: ACL-05 NLP/IR/speech merge: ACL-05 conferencesconferences

The Association for Computational LinguisticsThe Association for Computational Linguistics invites the submissio invites the submission of papers for its 43rd Annual Meeting hosted jointly with the North n of papers for its 43rd Annual Meeting hosted jointly with the North American Chapter of the ACL. Papers are invited on substantial, origiAmerican Chapter of the ACL. Papers are invited on substantial, original, and unpublished research on all aspects of computational linguinal, and unpublished research on all aspects of computational linguistics, including, but not limited to: pragmatics, discourse, semantics, stics, including, but not limited to: pragmatics, discourse, semantics, syntax, grammars and the lexicon; syntax, grammars and the lexicon; phonetics, phonology and morphphonetics, phonology and morphologyology; lexical semantics and ontologies; word segmentation, tagging ; lexical semantics and ontologies; word segmentation, tagging and chunking; parsing, generation and summarization; language moand chunking; parsing, generation and summarization; language modeling, deling, spoken language recognition and understandingspoken language recognition and understanding; linguistic, p; linguistic, psychological and mathematical models of language; sychological and mathematical models of language; language-orientlanguage-oriented information retrieval, question answering, and information extracted information retrieval, question answering, and information extractionion; machine learning for natural language; corpus-based modeling ; machine learning for natural language; corpus-based modeling of language, discourse and dialogue; multi-lingual processing, machiof language, discourse and dialogue; multi-lingual processing, machine translation and translation aids; multi-modal and natural language ne translation and translation aids; multi-modal and natural language interfaces and interfaces and dialogue systemsdialogue systems; applications, tools and resources; ; applications, tools and resources; and evaluation of systems. and evaluation of systems.


NLP/IR/speech merge: ACM SIGIR-05 NLP/IR/speech merge: ACM SIGIR-05 conferencesconferences

SIGIR 2005SIGIR 2005 welcomes contributions related to any aspect of IR, but the major areas of interest are listed welcomes contributions related to any aspect of IR, but the major areas of interest are listed below. For each general area, two or more area coordinators will guide the reviewing process. below. For each general area, two or more area coordinators will guide the reviewing process.

Formal Models, Language Models, Fusion/CombinationFormal Models, Language Models, Fusion/Combination

Text Representation and Indexing, XML and MetadataText Representation and Indexing, XML and Metadata

Performance, Compression, Scalability, Architectures, Mobile ApplicationsPerformance, Compression, Scalability, Architectures, Mobile Applications

Web IR, Intranet/Enterprise Search, Citation and Link Analysis, Digital Libraries, Distributed IRWeb IR, Intranet/Enterprise Search, Citation and Link Analysis, Digital Libraries, Distributed IR

Cross-language Retrieval, Multilingual Retrieval, Machine Translation for IRCross-language Retrieval, Multilingual Retrieval, Machine Translation for IRVideo and Image Access, Audio and Speech Retrieval, Music RetrievalVideo and Image Access, Audio and Speech Retrieval, Music Retrieval

Text Data Mining and Machine Learning for IRText Data Mining and Machine Learning for IRText Categorization, ClusteringText Categorization, Clustering

Topic Detection and Tracking, Content-Based Filtering, Collaborative Filtering, AgentsTopic Detection and Tracking, Content-Based Filtering, Collaborative Filtering, Agents

Summarization, Question Answering, Natural Language Processing for IR, Information Extraction, Summarization, Question Answering, Natural Language Processing for IR, Information Extraction, Lexical AcquisitionLexical Acquisition

Interactive IR, User Interfaces, Visualization, User Studies, User ModelsInteractive IR, User Interfaces, Visualization, User Studies, User Models

Specialized Applications of IR, including Genomic IR, IR in Software Engineering, and IR for Chemical Specialized Applications of IR, including Genomic IR, IR in Software Engineering, and IR for Chemical StructuresStructures

Evaluation, Building Test Collections, Experimental Design and MetricsEvaluation, Building Test Collections, Experimental Design and Metrics


Speech/NLP/IR merge- recent HLT Speech/NLP/IR merge- recent HLT conference seriesconference series

HLT/NAACL2003 – Edmonton, CanadaHLT/NAACL2003 – Edmonton, Canada HLT/NAACL2004 – Boston, USAHLT/NAACL2004 – Boston, USA HLT/EMNLP2005 – Vancouver, CanadaHLT/EMNLP2005 – Vancouver, Canada The joint conference provides a unified forum for researchers The joint conference provides a unified forum for researchers

across a spectrumacross a spectrum of disciplines to present recent, high- of disciplines to present recent, high-quality, cutting-edge work, to exchange ideas, and to explore quality, cutting-edge work, to exchange ideas, and to explore emerging new research directions. The conference especially emerging new research directions. The conference especially encourages submissions that encourages submissions that discuss synergistic discuss synergistic combinations of language technologies (e.g., Speech with combinations of language technologies (e.g., Speech with Information Retrieval, Machine Translation with Speech, Information Retrieval, Machine Translation with Speech, Question Answering with Natural Language Processing, etc.).Question Answering with Natural Language Processing, etc.). Particular consideration will be given to papers addressingParticular consideration will be given to papers addressing novel learning tasks and evaluation metrics in speech, natural novel learning tasks and evaluation metrics in speech, natural language processing and information retrieval,language processing and information retrieval, including e.g.:including e.g.:


HLT/EMNLP2005 CFPHLT/EMNLP2005 CFP

learning taskslearning tasks insufficiently addressed in the past, e.g. collaborative learning, learning in the presence insufficiently addressed in the past, e.g. collaborative learning, learning in the presence of background knowledge, or finding anomalies in data; of background knowledge, or finding anomalies in data;

limits of standard evaluation methods on new tasks; limits of standard evaluation methods on new tasks; novel performance measures incorporating user preferences, competence, or relevance to a given probnovel performance measures incorporating user preferences, competence, or relevance to a given prob

lem; lem; learning and optimization algorithms addressing the above, e.g. novel statistical methods or cognitively learning and optimization algorithms addressing the above, e.g. novel statistical methods or cognitively

inspired solutions. inspired solutions. We are interested in papers from academia, government, and industry on all areas of traditional interest We are interested in papers from academia, government, and industry on all areas of traditional interest

to the HLT and SIGDAT communities, as well as aligned fields, to the HLT and SIGDAT communities, as well as aligned fields, including but not limited to:including but not limited to: Speech processing, Speech processing, including:including:

Speech recognition Speech generation Speech summarization Rich transcription: annotation of speech signals with metalinguistic information, such as speaker

identity, attitude, emotion, etc. Speech-based human-computer interfaces

Text summarization Text summarization Question answeringQuestion answering Paraphrasing Paraphrasing Computational analysis of phonology, morphology, prosody, syntax, semantics, pragmatics, discourse, Computational analysis of phonology, morphology, prosody, syntax, semantics, pragmatics, discourse,

style style Statistical techniques for language processing, Statistical techniques for language processing, includingincluding::

Corpus-based language modeling Lexical and knowledge acquisition


HLT/EMNLP2005 CFPHLT/EMNLP2005 CFP

Language generation and text planningLanguage generation and text planning Sentence parsing and discourse analysis Sentence parsing and discourse analysis Multilingual processingMultilingual processing, , including:including:

Machine translation of speech and text Cross-language information retrieval Multi-lingual speech recognition and language identification

Evaluation, Evaluation, including:including: Glass-box evaluation of HLT systems and system components Back-box evaluation of HLT systems in application settings

Development of language resources, Development of language resources, including:including: Lexicons and ontologies Treebanks, proposition banks, and frame banks

Understanding of human communication, Understanding of human communication, including:including: Natural language interfaces Dialogue structure and dialogue systems Message and narrative understanding systems

Information extraction from multiple media Information extraction from multiple media Information retrieval, Information retrieval, including:including:

Formal models, clustering and classification Web mining for IR Natural language processing for IR Spoken IR Metadata annotation and XML IR


contentscontents

What is HLT? - definition/historyWhat is HLT? - definition/history HLT workshop case study – darpa hlt examplesHLT workshop case study – darpa hlt examples

Towards Technology synergy – 21c frontier projectTowards Technology synergy – 21c frontier project


Technology cross-over: some Technology cross-over: some examplesexamples

Spoken language understanding needs information extraction Spoken language understanding needs information extraction technologytechnology

Language modeling (adaptation) for ASR needs information reLanguage modeling (adaptation) for ASR needs information retrieval for corpus expansion for a specific domaintrieval for corpus expansion for a specific domain

Statistical MT needs fast-viterbi decodingStatistical MT needs fast-viterbi decoding ASR, SMT, speech error correctionASR, SMT, speech error correction use exactly same HMM use exactly same HMM

modeling processmodeling process Language modeling for ASR needs parsing/structural analysisLanguage modeling for ASR needs parsing/structural analysis And more and more…And more and more…


Some scenarios from 21c frontier Some scenarios from 21c frontier projectproject

33 단계단계 ::노인 노인 : : 꾀돌아꾀돌아 , , 파리의 연인 재방송은 언제하지파리의 연인 재방송은 언제하지 ?? 로봇 로봇 : : 파리의 연인 재방송은 파리의 연인 재방송은 SBS SBS 드라마 채널에서 월요일 아침 드라마 채널에서 월요일 아침 1010 시에 시에 합니다 합니다 . . 노인 노인 : : 그거 예약 녹화 좀 해 놔라그거 예약 녹화 좀 해 놔라 .. 다음 주에도 같은 시간에 재방송이니다음 주에도 같은 시간에 재방송이니 ??

로봇 로봇 : : 네네 , , 예약하겠습니다예약하겠습니다 . . 다음 주는 올림픽 중계로 재방송이 없습니다다음 주는 올림픽 중계로 재방송이 없습니다 . . 노인 노인 : : 참참 , , 이번 일요일 강영순 집사가 온다고 했는데 몇 시에 오니이번 일요일 강영순 집사가 온다고 했는데 몇 시에 오니 ?? (domain switching)(domain switching)

로봇 로봇 : : 강영순 집사와 월요일 약속은 오후 강영순 집사와 월요일 약속은 오후 11 시 입니다시 입니다 ..노인 노인 : : 알았다알았다 . . 그날 그날 11 시간 전에 다시 알려다오시간 전에 다시 알려다오 .. 그리고 냉장고에서 마실 것 좀 가져와라그리고 냉장고에서 마실 것 좀 가져와라 .. (domain switching)(domain switching)로봇 로봇 : : 네네 , , 알겠습니다알겠습니다 . . 마실 것은 무엇으로 가져올까요마실 것은 무엇으로 가져올까요 ? ? (mixed mode convc(mixed mode convceration)eration)노인 노인 : : 시원한 냉수가 좋겠다시원한 냉수가 좋겠다 .. 로봇 로봇 : : 네네 , , 냉수 냉수 11 잔을 가져다 드리겠습니다잔을 가져다 드리겠습니다 . . 노인 노인 : : (( 거실 구석의 책상을 가리키며거실 구석의 책상을 가리키며 )) 그리고 그 위에 있는 노란 책 좀 그리고 그 위에 있는 노란 책 좀

가져와라가져와라 .. (multi-modal gesture)(multi-modal gesture)로봇 로봇 : : 네네 , , 책상 위에 있는 노란 책을 가져다 드리겠습니다책상 위에 있는 노란 책을 가져다 드리겠습니다 ..


노인 그 그래 ? 어 대장금 시작하믄 알려줘

그래 ? 대장균 삭히면 알려줘

그래 ? 대장금 시작하면 알려줘

음성인식결과

대화 현상을 반영한 음성인식 간투어 : 어 / 반복 / 수정발화 : 그 / 그래 ? 발음변이 : 시작하믄 ( 시작하면 )(CSR itself)

영역 지식 (TV 가이드 ) 및 구문 / 의미 /문맥 지식을 이용한 인식 오류 수정(post error correction)

로봇 예 . 대장금 시작할 때 TV 를 켜고

알려 드리겠습니다 .

대화 모델과 영역 지식을이용한 음성대화 진행(dialog understanding)

Conversational SDS-integrated Conversational SDS-integrated approachapproach


HLT- speech/language synergyHLT- speech/language synergy

언어 이해언어 이해UnderstandingUnderstanding

음성 처리음성 처리RecognitionRecognition

&&CorrectionCorrection

대화대화 및 및

질의 응답질의 응답

음성 인식 시스템음성 인식 시스템음성 인식 시스템음성 인식 시스템

HTK 기반의 제한 - 실용적 음성 인식

음성 후처리음성 후처리음성 후처리음성 후처리음성 오류 수정

음성 언어 이해음성 언어 이해음성 언어 이해음성 언어 이해음성 오류에 강인한

음성 언어 이해

대화 시스템대화 시스템대화 시스템대화 시스템대화모델링 및 대화를 통한

음성 인식 수정

질의 응답 시스템질의 응답 시스템질의 응답 시스템질의 응답 시스템

DataBase 질의 및 결과 응답 시스템

미래의 통합정보 DB


From signal to dialog/knowledgeFrom signal to dialog/knowledge

노인 : 짱구야 , 장금이 언제 하지 ? 로봇 : 드라마 대장금은 월요일 , 화요일 밤 9 시 55 분에 합니다 . 노인 : 가만 , 오늘이 무슨 요일이지 ? 로봇 : 오늘은 월요일입니다 . 노인 : 그래 ? 대장금 시작하면 알려줘 . 로봇 : 예 , 드라마 시작할 때 TV 를 켜고 알려 드리겠습니다 .

노인 : 짱구야 , 장금이 언제 하지 ? 로봇 : 드라마 대장금은 월요일 , 화요일 밤 9 시 55 분에 합니다 . 노인 : 가만 , 오늘이 무슨 요일이지 ? 로봇 : 오늘은 월요일입니다 . 노인 : 그래 ? 대장금 시작하면 알려줘 . 로봇 : 예 , 드라마 시작할 때 TV 를 켜고 알려 드리겠습니다 .

9시 55분 입니다 .

9시 55분 입니다 .

대장금 언제 하지 ?대장금

언제 하지 ?

대화음성인터페이스

대화음성인식기 성능 향상 노년층 대화현상을 반영한 음성인식 문맥정보를 활용한 음성인식 오류 수정 대화모델을 이용한 체감인식률 향상

음성 전처리 성능향상 음원 분리 및 음원 분류 원격 마이크 환경보상 로봇 잡음 및 배경잡음 보상

실버 메이트용 대화음성 인터페이스응용 시나리오 예제 실버 메이트용 대화음성 인터페이스응용 시나리오 예제

Human Language Technology

Documents

Transcript of Human Language Technology