Post on 22-Dec-2021
All Rights Reserved, Copyright © NHK
Research on Universal
Broadcasting at NHK STRL
Masahiro Shibata
NHK Science & Technology Research Labs.
All Rights Reserved, Copyright © NHK
Background
2
All Rights Reserved, Copyright © NHK
Terminology
3
Universal Broadcasting
Broadcasting which can deliver information to
everyone including visually or auditorily impaired
people, elder people, and foreigners.
Accessible Technology
Technology to realize universal broadcasting or
human-friendly information presentation.
All Rights Reserved, Copyright © NHK
Objectives of Universal Broadcasting
4
1. To deliver information “extensively in Japan”
2. To deliver information to “everyone equally”
regardless of impairment/age or language
difference
Two dimensional
“extensiveness”
Three dimensional
“extensiveness”
= Universal Broadcasting
Eve
ryo
ne
-equ
ally
All Rights Reserved, Copyright © NHK
History of closed caption (CC)
and audio description (AD) services
5
Analog broadcasting Oct. 1983 Character multiplex broadcast (test service)
Oct. 1984 Closed caption (CC) started. (morning drama “Oshin”)
1982 Audio multiplex broadcast
Apr. 1991 Audio description (AD) started.
(morning drama “Rinrin-to”)
Digital broadcasting ISDB-T standard
Any digital receiver has decoders for CC and AD
⇒ Hardware issues solved
All Rights Reserved, Copyright © NHK
Accessible-service guideline
6
Service guideline set by Japanese government
Guideline for CC
(set in 1997) Guideline for CC and AD
(set in 2007)
Target year 2007 2017
Scope Recorded
programs
All programs
from 7AM to 12PM
CC 100% 100%
AD - 10% (15% for NHK 2 (Ed.)) Status in 2010 NHK 1 NHK 2 (Ed.)
CC 62.2% 52.5%
AD 7.6% 11.2%
Completed
(percentage per 7AM to 12PM)
All Rights Reserved, Copyright © NHK
Research on accessible technology
7
CG
sign language
Accessible technology for the impaired
CC by speech
recognition
AD script
editor
Accessible technology for the elderly, foreigners
答え:後に同じことが書いてあるので省略しましょう
夏に電力が足りなくなるかもしれない問題で、人々の節電への関心が高くなったため、扇風機が良
く売れています。質問:「夏に...高くなったため」
の部分はいりますか?
現在の難しさ 2500
“Easy Japanese ” for foreigners
Multi-modal
display
Speech processing for the elderly
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the impaired
[Closed Caption (CC) Service]
8
News show, general news etc.
Relayed keyboards method Stenographic keyboard method
Direct speech recognition
http://www.speed-wp.co.jp/laboratory/
Re-spoken speech recognition
Music programs etc.
Sports, information programs etc. Short news, American baseball, etc.
Methods of producing live CC in NHK
All Rights Reserved, Copyright © NHK
10
Non-live CC Drama, Documentary, etc
With
CC
Keyboard Part of general news
Music show etc.
Current status
of NHK 1
Re-spoken/Direct
Speech rec. Sumo, baseball etc
With
ou
t C
C
Hybrid-type
Speech rec. Short news
Talk shows, etc.
Recognition of
Spontaneous speech
R
&
D
Speech recognition for CC
All Rights Reserved, Copyright © NHK
text
Speech rec. Error correct CC on-air
voice
Speech of announcer
or reporter
Direct
Interview
Re-speak 1-2 operators
Male-female recognition, Auto. dictionary update
Hybrid speech recognition for CC
11
Recognize speech of announcer and reporter directly, and re-spoken speech for interview Recognition accuracy of over 95%
Higher recognition rate and lower operation cost
R&D issue: Spontaneous speech in news show programs Need to overcome unclear or rapid speech and phrasing in speech
Recognition accuracy of less than 90% for spontaneous conversations
Speech recognition
Operators for error correction
Output
display
Latest CC system by speech recognition
for 4 o’clock News Mar. 14, 2012〜
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the impaired
[Audio Description (AD)
Service]
13
All Rights Reserved, Copyright © NHK
AD production
Current AD should be considered as a kind of “special radio drama” and its
production is labor intensive.
Script writers and narrators have to be highly trained.
Finished
program Writing
AD script
Recording
AD voice
(in studio)
Mixing down
to sound track
Script writer
Script writer
Program director
Narrator
On Air
Program director
Narrator
Engineer
14
Time-consuming!
AD script editor
Detected periods
with stage
direction
Input script
Script exceeding
the estimation is
indicated by red
Automatically
estimated syllable
counts
Supplemental
direction for narrator
Automatically
generated
script draft
Current script
window
Wave form of
program sound
Automatically
detected periods of
voice and silence
All Rights Reserved, Copyright © NHK
Evaluation of the system
Average of three trials by MS Office Operators
0
20
40
60
80
100
120
140
160
180
手作業
(1人の結果)
電子台本なし 電子台本アリ
システム利用
165
101
77
Manual input Without
data of
original script
With
data of original
script
Using AD script editor system
Work time of
Experienced
AD Writers
Tim
e to
write
scrip
t(m
in)
16
(Duration of the
program was 15 min)
All Rights Reserved, Copyright © NHK
Evaluation of the system
17
Opinion of Novice Writers • The system is useful.
• Indicating silent portion is convenient.
• Estimation of syllable count is a good help.
Opinion of Experienced Writers • The system does not help much.
• AD should be inserted to right positions regardless of the presence of main program voices. This should be done by writers’ skill.
Conclusion
The system can extend the variation of AD programs without intensively training new writers while AD can be supplied only for very popular programs with the current production style.
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the impaired
[Sign Language (SL) Service]
18
All Rights Reserved, Copyright © NHK
SL service
19
For deaf people by nature or those who lost hearing in early childhood, sign language is their first language. Sign language service (rather than Closed Caption (CC) ) is requested.
Grammar of Japanese Sign Language (JSL) is different from that of the aural Japanese.
Difficulty to keep sign language interpreters for regular service Especially for emergency news
TV
program broadcasting
internet SL CG
Station Home Translate CC to SL
CG sign language service by automatic translation
All Rights Reserved, Copyright © NHK
Overview of automatic translation system
Japanese - JSL
dictionary
Japanese - JSL
corpus
Input Japanese sentence: 私の名前は金子です。 (“My name is Kaneko.”)
Convert Japanese sentence into sequence of
symbols representing JSL word
Generate signing motion for each symbol
(including hand and finger motions, facial
expressions, and so on).
Generate seamless motion transitions between
each symbol by using motion interpolation.
Output computer-animated JSL
[ 私, 名前, 金子, 言う, N ]
( I, name, Kaneko, say, nodding )
20
All Rights Reserved, Copyright © NHK
Japanese – JSL CG dictionary system
Found Japanese entry
Confidence measure
Thumbnail image of
CG animation
Click entry to display
CG animation of JSL word
Entries of synonyms
21
Input Japanese word
All Rights Reserved, Copyright © NHK
Evaluation of
Japanese-JSL CG dictionary system
22
■Natural
■Some part is unnatural but omittable
■Some part is unnatural but not
bothering
■Strange
■Very unnatural
Naturalness
■Easy to read
■Some part is unreadable but omittable
■Some part is unreadable but not
bothering
■Bothering
■Unreadable
Readability
Based on questionnaire survey at Tokyo Federation of Deaf meeting on May 5-6, 2011
All Rights Reserved, Copyright © NHK
Japanese-JSL CG translation
prototype for weather information
23
Screen shot of Japanese-JSL example-based translation prototype.
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the impaired
[Multi-modal Display]
24
All Rights Reserved, Copyright © NHK
Digital TV receiver for visually impaired
25
Visual information and graphical user interface (GUI) are everywhere! PC, TV, smart-phone, remote controller, ticketing machine, ATM, etc
But they can be big obstacle for visually impaired people
Example of data broadcasting screen
DTV receiver
for visually
impaired people
•Reading-out menu and other
visual information
•Magnifying characters on screen
•(Braille expression of texts)
All Rights Reserved, Copyright © NHK
Also applicable for guide terminals in public places such as
railway stations, ATMs in the bank.
Interactive tactile display
26
Two dimensional tactile display + touch position sensor
As a GUI device for visually impaired people
Hierarchical expression Menu expression by Braille
All Rights Reserved, Copyright © NHK
Information-barrier-free for visually
impaired people
27
Voice and Braille
Tactile and force feedback
3D objects
Interactive tactile display
2D graphics (+ GUI)
texts
Universality
for
impaired/non-impaired
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the elderly
28
All Rights Reserved, Copyright © NHK
Aging of the society
29
Aging rate and its speed of Japan is highest in the world.
Compensation to auditory and visual ability is social requirement.
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
日本
イタリア
ドイツ
アメリカ合衆国
韓国
シンガポール
Japan
Italy
Germany
U.S.A.
Korea
Singapore
Redrawn from “Annual
Report on the Aging
Society: 2009”
(by Cabinet Office Japan)
Agin
g r
ate
(≧
65
)
All Rights Reserved, Copyright © NHK
Problems in information perception of
the elderly
30
Visual information
Degraded accommodation function (presbyopia)
Low vision
Lower sensitivity of bluish color
Auditory information
Less sensitive to higher tone
Difficult to understand speech with high speed
Difficult to understand speech under background noise
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the elderly
[Speech Rate Conversion]
31
All Rights Reserved, Copyright © NHK
Evaluation experiment TV and radio set with “Slow button”
32
Started research by responding a claim from an elderly person; “Recently, speech on broadcasting is too fast for me to understand”
Slow down speech without degrading sound quality
Speech rate conversion for the elderly
All Rights Reserved, Copyright © NHK
Speech rate conversion without
changing voice pitch
33
Temporal duration of waveform is changed to keep fundamental period
×
Slower time
① ② ③ ③ ④ ⑤ ⑥ ⑦ ⑧ ⑧ ⑨ ⑩
Faster time
① ② ④ ⑤ ⑥ ⑦ ⑨ ⑩
Original time
time
① ② ③ ④ ⑤ ⑥ ⑦ ⑧ ⑨ ⑩
stop
Fundamental period is extended and pitch becomes lower
Conventional
×
① ② ③ ④ ⑤ ⑥ ⑦ ⑧ ⑨ ⑩
All Rights Reserved, Copyright © NHK
Speech rate conversion without
changing length
34
Start is coincided at
blue line positions
Start is not coincided but… Again it coincides
Original
Converted
All Rights Reserved, Copyright © NHK
Original
(n times) BGM speech silent
Important part(speech)
Speech rate conversion for visually
impaired people
35
Understandable high speed speech
Visually impaired people usually understand PC screen
by using screen reader ⇒ they want to hear like quick
reading of sighted people
Make this part easier to understand
Converted (same length)
Make slower Make slower
time
All Rights Reserved, Copyright © NHK
Applications of speech rate conversion
36
Mobile terminal
iPhone app.
For language learning and audio book
For PC
Plug-in for Windows media player
Variable speed with rip synchronization
Variable speech for audio and visual content
on the internet
DAISY Player for visually impaired
people
Applicable both to CD player and
software
※DAISY:Digital Accessible Information SYstem
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the elderly
[Speech Processing]
37
All Rights Reserved, Copyright © NHK
Speech processing for the elderly
38
Home
Speech
enhance
Deriving
speech
output
Loudness
control
Developing as receiver’s
functionalities
Estimated
speech
Estimated
background sounds
input
Speech (narration etc.)
Background
sounds
Mixed
sounds
Mixing balance meter
Studio
Calculating
loudness &
Estimating
favorability of
mixing
balance
All Rights Reserved, Copyright © NHK
Research on accessible
technology for foreigners
[“Easy Japanese”]
39
All Rights Reserved, Copyright © NHK
Research on “Easy Japanese” for
foreigners living in Japan
40
Registered foreigners : 1.7% of pop. in 2009.
Increase of foreigners population
Difficulty of enriching multi-lingual services
Developing Web news services by
“Easy Japanese”:understandable
for foreigners in Japan
Research issues:
・Establish methods to re-write “Easy
Japanese” from news manuscripts in
usual Japanese.
・Develop authoring tools for “Easy
Japanese”
(Press-release from the ministry of justice)
All Rights Reserved, Copyright © NHK
An experimental service started on
2nd
April 2012
41
http://www3.nhk.or.jp/news/easy/
Uses short sentences and
easy words
All Kanji characters have
Furigana (phonetic characters)
Underlined words have
dictionary explanation
All Rights Reserved, Copyright © NHK
Authoring tool for “Easy Japanese”
42
Rewriting is done by a team of a
Japanese language teacher and
a reporter.
The tool assists the team work.
Authoring tool can measure the
text difficulty and show it in
index figures and color.
All Rights Reserved, Copyright © NHK
Other aspects of accessibility
for broadcasting
[Emergency Warning
Broadcasting System (EWBS)]
43
All Rights Reserved, Copyright © NHK
What is EWBS ?
44
Remote activation of Radio and TV receivers ready for EWBS
AM, FM Radio and TV : Control signal and Alert Sound
Digital TV Broadcasting “ISDB-T” :
EWBS Control (Activation Flag)
In Japan, EWBS has been operated since September 1985.
EWBS on ISDB-T has been operated since December 2003.
EWBS test signals are monthly broadcast in Japan.
Broadcasting Station
Transmitter Broadcasting Service Area
Automatic Activation
Meteorological Agency
All Rights Reserved, Copyright © NHK
Mechanism of EWBS for transmitting
EWBS signal sent by on air signal
start signal ; switches on receivers
end signal ; turns off receivers
Analog system for Radio &TV ; 640Hz & 1024Hz frequency
Digital system TV ; digital data as flag on ISDB-T
45
All Rights Reserved, Copyright © NHK
Functions of EWBS in disaster relief
1. Gathering/receiving disaster information from
authorized organizations
2. Delivering disaster information to the public general
Advantages of EWBS
1. Always connected to everybody, send signal ASAP
There are no congestions like telephone lines
2. Most cost efficient system
One way “Delivery” system is enough.
(No two-way radio communication system is required.)
46
All Rights Reserved, Copyright © NHK
Towards/beyond
Universal Broadcasting
47
All Rights Reserved, Copyright © NHK
Towards Universal Broadcasting
48
media
se
rvic
e
/fu
nctio
na
lity
Closed caption
Read-out
….
Sign language
Audio description
Speech rate conversion
Broadcasting
To support variety of services/functionalities:
Hybridcast ®
(a platform of collaboration of broadcasting
and communication)
All Rights Reserved, Copyright © NHK
Enriched services
Broadcasting media (Simultaneous, High quality, Trustworthy)
Program
49
Hybridcast ® receiver
Program-related inf.
Internet service (Cooperation with
broadcasting)
Communication media (Applicable to personal request)
DRM technology
Synchronization of
various information
Coop. with mobile term.
®
Demand, SNS input
Concept of Hybridcast ®
Broadcaster
Service provider
All Rights Reserved, Copyright © NHK
Towards Universal Broadcasting
50
Technology for viewer adaptation
Context
Who?
Demand?
Interest?
Situation?
Broadcaster
Content provider
Context-aware
•Content provision
•AV signal adaptation
CG
sign language
Accessible technology for the impaired
CC by speech
recognition
AD script
editor
Accessible technology for the elderly, foreigners
答え:後に同じことが書いてあるので省略しましょう
夏に電力が足りなくなるかもしれない問題で、人々の節電への関心が高くなったため、扇風機が良
く売れています。質問:「夏に...高くなったため」
の部分はいりますか?
現在の難しさ 2500
“Easy Japanese ” for foreigners
Multi-modal
display
Speech processing for the elderly
All Rights Reserved, Copyright © NHK
Beyond Universal Broadcasting
51
media
se
rvic
e
/fu
nctio
na
lity
Closed caption
Read-out
…
Sign language
Speech rate conversion
Broadcasting Web, VOD, BB DVD
For media-independent data sharing: Standardization (ITU, SMPTE, W3C, etc.)
Universal media services
⇒Society free from information barrier
Mobile
Audio description
All Rights Reserved, Copyright © NHK
Closing remarks
52
Introduced researches on accessible technologies
Technologies for impaired people
Technologies for the elderly, foreigners
Other aspects of accessibility for broadcasting
In order to realize the society free from information
barrier
Towards Universal Broadcasting
Based on accessible technologies
Hybridcast ® as a platform
Viewer adaptation
Beyond Universal Broadcasting
Expansion to other media
All Rights Reserved, Copyright © NHK
53
Thank you very much!!