US007479949B2

(12) United States Patent (10) Patent N0.2 US 7,479,949 B2 Jobs et al. (45) Date of Patent: *Jan. 20, 2009

(54) TOUCH SCREEN DEVICE, METHOD, AND (65) Prior Publication Data GRAPHICAL USER INTERFACE FOR DETERMINING COMMANDS BY APPLYING Us Zoos/0174570 A1 Jul’ 24’ 2008

HEURISTICS Related US. Application Data

(75) Inventors: Steven P. Jobs, P810 A110, CA (Us); (63) Continuation of application No. 11/850,635, ?led on Scott Forstall, Mountain View, CA Sep 5, 2007'

(sltleslgl’lglrgg iil?iglle’siinpizjgisccé (52)’ (60) Provisional application No. 60/937,993, ?led on Jun. (Us), Scott Herz S’an Jose CA (U’S) 29, 2007, provisional application No. 60/937,991, Maréel van ()5 S’an Francisco CA ’ ?led on Jun. 29, 2007, provisional application No. (Us), Bas ordi’ng San FranCiS’CO CA 60/879,469, ?led on Jan. 8, 2007, provisional applica (US; Gregory No’vick Santa cla’ra CA tion No. 60/879,253, ?led on Jan. 7, 2007, provisional (Us); Wayne 0 Weste’rman’ San ’ application No. 60/824,769, ?led on Sep. 6, 2006.

518828182‘,8216882311153? 61> 0, Coffman, Menlo Park, CA (US); G06F 3/048 E2006'01; Kenneth Kocienda, Sunnyvale, CA ' (Us); Nitin K_ Ganatra’ San Jose’ CA (52) US. Cl. ..................... .. 345/173; 345/169; 715/786; (US); Freddy Allen Anzures, San _ _ _ 715/784 Francisco, CA (Us); Jeremy A_ Wyld, (58) Field of Classi?cation Search ............... .. 345/156,

San Jose, CA (US); Jeffrey Bush, San _ _ 345M573 173’181 Jose, CA (Us); Michael Matas, San See application ?le for complete search history. Francisco, CA (US); Paul D. Marcos, - Los Altos, CA (US); Charles J. Pisula, (56) References Clted San Jose, CA (US); Virgil Scott King, U.S. PATENT DOCUMENTS Mountain VieW, CA (US); Chris

W19; 282833 2: 2122? 81111 ~~~~~~~~~~~~~~~~~~~ ~~ 312/322 g:168559153:grgoaz?gisg?p?gno’ 5,805,161 A * 9/1998 Tiphane .................... .. 715/786 Gatos CA Andre M- J- Bioule 6,278,443 B1 8/2001 Amro et a1. . 345/173 sunny’vale CA Henri C- ’ 6,466,203 B2 10/2002 Van Ee ..... .. 345/173 Lamiraux’ San carl’os CA (Us) 6,559,869 B1 * 5/2003 Lui et a1. .................. .. 715/785

’ ’ 6,597,345 B2* 7/2003 Hirshberg ................. .. 345/168

(73) Assigneei Apple Ine, Cupenine, CA (US) 6,657,615 B2 12/2003 Harada ........... .. 345/173 6,683,628 B1 * l/2004 Nakagawa et a1. 715/799

( * ) Notice: Subject to any disclaimer, the term ofthis 6,690,387 B2 2/2004 Zimmerman et a1, __ 345/6g4 patent is extended or adjusted under 35 7,088,344 B2 8/2006 Maezawa 61:11. .... .. 345/173 U.S.C. 154(b) by 0days. 7,093,203 B2 8/2006 Mugura 61:11. 715/864

2002/0158838 A1 10/2002 Smith et a1. 345/156 This patent is subject to a terminal dis- 2003/0184593 A1 10/2003 Dunlop ........ .. 345/810 claimer. 2004/0012572 A1 l/2004 SoWden et a1. . 345/173

2004/0021676 A1 2/2004 Chen et a1. 345/684 (21) Appl. No.: 12/101,832 2004/0160420 A1 8/2004 Baharav ................... .. 345/173

2005/0012723 A1 l/2005 Pallakoff .................. .. 345/173

(22) Filed: Apr. 11, 2008 2005/0193351 A1 9/2005 Huoviala .................. .. 715/840

41:42: Current Time M

Welcome m

http://wwwmumpanymom/startm { y i am

{3351) Block 4 1914A

(1916;. Block 5 m

Block 6 19143 ‘13225)

A

1m 3131

m

3222

US 7,479,949 B2 Page 2

2006/0001652 A1 1/2006 Chiu et al. ................ .. 345/173

2006/0007178 A1 1/2006 Davis ....... .. . 345/173

2006/0022955 A1 2/2006 Kennedy .................. .. 345/173

2006/0028428 A1 2/2006 Daiet al. .................. .. 345/156

2006/0031786 A1 2/2006 Hillis et a1. ..... .. 715/863

2006/0044259 A1 3/2006 Hotelling et al. .......... .. 345/156

2006/0049920 A1 3/2006 Sadler et a1. ........... .. 340/407.1

2006/0101354 A1 5/2006 Hashimoto et a1. ........ .. 715/863

2006/0132460 A1 6/2006 Kolmykov-Zotov et al. . 345/173 2006/0164399 A1 7/2006 Cheston et al. ........... .. 245/173

2006/0181519 A1 8/2006 Vernier et a1. 345/173 2006/0253793 A1 11/2006 Zhaiet a1. ................. .. 715/773

2006/0294472 A1 12/2006 Cheng et a1. .............. .. 715/771

2007/0040812 A1 2/2007 Tang et al. 345/173 2007/0061126 A1 3/2007 Russo et al. ................ .. 713/24

2007/0118400 A1 5/2007 Morita et al. ................ .. 705/2

2007/0120834 A1 2007/0130532 A1 2007/0150826 A1 2007/0152984 A1 2007/0177803 A1

5/2007 Boillot ........ ..

6/2007 Fuller et al. .

6/2007 AnZures et a1. 7/2007 Ording et al. 8/2007 Elias et al.

FOREIGN PATENT DOCUMENTS

EP 0 827 064 A2 3/1998 EP 0 827 094 A2 3/1998 EP 1 517 228 A2 3/2005 GB 2 347 200 A 8/2000 W0 WO 02/01338 A1 1/2002 W0 WO 2004/111816 A2 12/2004 W0 WO 2005/074268 A1 8/2005 W0 WO 2006/020305 A2 2/2006 W0 WO 2006/126055 A2 11/2006

OTHER PUBLICATIONS

Examiner’ s Report on Australian Innovation Patent No. 2008100179, dated Apr. 30, 2008, Which application claims priority to US. Appl. No. 60/824,726, the same priority application of the instant applica tion. Baguley, R., “Nokia Handlelds & Palmtops Internet Tablet 770, Nokia’s Small, Svelte, Internet-Savvy PDA,” Jan. 31, 2006, http:.// WWW.pcWorld.com/printable/ article/ id, 124456/printable.html. Bordovsky et al., “Interpreting Commands from a Graphical User Interface,” reproduced from International Technology Disclosures, vol. 9, No. 6, Jun. 25, 1991, 1 page. Cheng et al., “Navigation Control and Gesture Recognition Input Device for Smaill, Portable User Interfaces,” Synaptics Inc. of San Jose, Callifornia, pp. 1-13, 2004. Computergram International, “Next-Generation Sharp Organiser to carry Pen Interface,” No. 1955, Jul. 2, 1992. Electronic Engineering Times, “Screen Can Tell Finger From Sty lus,” No. 858, Jul. 24, 1995, p. 67. Gillespie, D., “Novel Touch Screens for Hand-Held Devices,” Infor mation Display, vol. 18, No. 2, Feb. 2002, 5 pages.

Hoover, J .N., “Computer GUI Revolution Continues With Microsoft Surface’s Touch Screen, Object Recognition,” Information Week, May 30, 2007, http://WWW.informationWeek.com/story/shoWArticle. jhtml?articleID:199703468. IBM, “Method to Disable and Enable aTouch Pad Pointing Device or Tablet Input Device Using Gestures,” Jun. 11, 2002, pp. 1-3. Johnson, R.C., “Gestures Rede?ne Computer Interface,” Electronic Engingeering Times, No. 924, p. 42(1), Oct. 21, 1996. Korpela, J ., “Using Inline Frames (iframe elements) to Embed Docu ments into HTML Documents,” (Online), Sep. 25, 2006, http://Web. archive.org/Web/2006092511355l/http://WWW.cs.tut.?/{jkorpela/ htm/iframe.html. NarayanasWamy, et al., “User Interface for a PCS Smart Phone,” Multimedia Computing and Systems, IEEE Conference 1999, Pub lished Jun. 7-11, 1999, vol. 1, pp. 777-781. Poon et al., “Gestural User Interface Technique for Controlling the Playback of Sequential Media,” Xerox Disclosure Journal, vol. 19, No. 2, Mar/Apr. 1994, pp. 187-190. PR NeWsWire, “FingerWorks Announces a Gestrue Keyboard for Apple PoWerBooks,” Jan. 27, 2004, 2 pages. PR NeWsWire, “FingerWorks Announces the ZeroForce iGesture Pad,” Feb. 18, 2003, 2 pages. International Search Report and Written Opinion for International Application PCT/US2007/088885, mailed Apr. 24, 2008. Examiner’s Report on Australian Innovation Patent No. 2008100179 dated Apr. 30, 2008. “Sprint Power Vision Smart Device TreoTM 700p by Palm,” Sprint Nextel, 432 pages, 2006. “Google Maps APIiGoogle Code,” Google Inc., http://WWW. google.com/apis/maps, 1 page, printed Apr. 10, 2008.

* cited by examiner

Primary ExamineriDuc Q Dinh (74) Attorney, Agent, or FirmiMorgan, LeWis & Bockius LLP

(57) ABSTRACT

A computer-implemented method for use in conjunction With a computing device With a touch screen display comprises: detecting one or more ?nger contacts With the touch screen display, applying one or more heuristics to the one or more

?nger contacts to determine a command for the device, and processing the command. The one or more heuristics com prise: a heuristic for determining that the one or more ?nger contacts correspond to a one-dimensional vertical screen scrolling command, a heuristic for determining that the one or more ?nger contacts correspond to a two-dimensional screen translation command, and a heuristic for determining that the one or more ?nger contacts correspond to a command to transition from displaying a respective item in a set of items to displaying a next item in the set of items.

20 Claims, 293 Drawing Sheets

US. Patent Jan. 20, 2009 Sheet 1 of 293 US 7,479,949 B2

Memory Portable Multifunction Device 102 — \ m

. 136 Operatmg System I 126 Applications (continued) I 148 Communication Module f 128 Calendar Module f 149

Contact/Motion Module Widget Modules f 149 1

Graphics Module f Weather Widget f 134 _ 149-2

Text Input Module .1’ 135 Stocks W|dget f 149 3

GPS Module I136 Calculator Widget f 149 4 Applications I137 Alarm Clock Widget f 149 5

Contacts Module f 138 Dictionary Widget f Telephone Module f 3

_ 139 ' _ 149-6

V|deo Conference Module f 140 User-Created W|dget(s) f 150

E-mail Client Module f 141 Widget Creator Module f 151

Instant Messaging Module f 142 Search Module f

Blogging Module I143 5 Camera Module f 162

144 Power f Image Management Module f

_ 145 System V|deo Player Module f

. 146 Muslc Player Module f External I124

- 147 " Port Browslng Module f

11 118 103 RF Circuitry ‘A 103 \ ‘A 103 >1 m Speaker

104-f " \ " l 103 111

— Controller 5 u '0 122 -/ Circuitry

[I _ =O\I\ V = L) : Perlpherals 103 Microphone ll Interface 5 Proximity 113 v

Sensor \-'\166 103

120 _/" Processor(s) H ‘ Accelerometer(s)

M ‘@103

IIO Subsystem 153 106 1 Optical /

Display Other Input 160 Controller Sensor(s) Controller(s) ‘f

_ Controller

156 J 11 11 11

‘r103 ‘r103 ‘r103

. . Optical Other Input Touch-Sens|t|ve f116 112 _f . Sensor(s) Control

Dlsplay System m Devices

Figure 1A

US. Patent Jan. 20, 2009 Sheet 2 of 293 US 7,479,949 B2

/ Memory M Portable Multifunction Device m 136 Operating System f 126 Applications (continued) I 148 Communication Module .1’ 128 Calendar Module f 149

Contact/Motion Module Widget Modules I149 1 Graphics Module f Weather Widget(s) f 149 2 Text Input Module 13’4 Stocks Widget f

135 149-3 GPS Module f 136 Calculator Widget f 149 4

Applications 1.137 Alarm Clock Widget f 1495 Contacts Module f 138 Dictionary Widget f Telephone Module f i

139 ' _ 149-6

Video Conference Module f 140 User-Created Widget(s) f 150

E-mail Client Module f 141 Widget Creator Module f 151

Instant Messaging Module f 142 Search Module f 155

Blogging Module I143 Online Video Module f Camera Module I144 5 Image Management Module I152 Video and Music Player Module f Power J’ 162

153 System Notes Module f

Map Module f 1 + External f T Browsing Module I147 Port

it 118 103 RF Circuitry V‘ 103 \ ‘A 103 >1 108 Speaker

104 I ll \ ll — I 111

- c t ii i103 mam 122 -/ °" '° e’ Circuitry

110 =[ ) < ‘7103 a Peripherals 103 Micro hone

ll ll

1: ' Interface 5 Proximity 113 103 Sensor \-'\ 156

120 r‘- Processor(s)

‘L Accelerometer(s) “ 168 ‘7" 103 —

IIO Subsystem 153 106 _f' o . /

. ptlcal Display Sensor(s) Other Input /. 150

Controller Controller(s) ‘ __ Controller

1 56 —/ A it it

‘$103 ‘,“103 ‘$103

. . Optical Other Input Touch-Sensitive 116

112 j Display System Sensor(s) Control I m Devices

Figure 1B

US. Patent Jan. 20, 2009 Sheet 3 of 293 US 7,479,949 B2

Portable Multifunction Device 100

E m \4 m f

Optical Proximity 0o [Speaker m) ( Sensor m) [ Sensor E 0| 200

5 ,

210 is SIM card slot 212 is headphone jack

E

Touch Screen l

c J

Microphone Home Acce|erometer(s) m m m

External Port m

Figure 2

US. Patent Jan. 20, 2009 Sheet 4 of 293 US 7,479,949 B2

Portable Multifunction Device 206 100 \‘

Optical Proximity [speaker m] (Sensor m J [Sensor m 300A 5 E \ )

Current Time ~\ 308

310 ~'\ Day and Date“ 312 E

Wallpaper image m

Slide to unlock J 3212 3056

Touch Screen 1 1

Microphone Home Accelerometer(s) m y m

Figure 3A

US. Patent Jan. 20, 2009 Sheet 5 of 293 US 7,479,949 B2

Portable Multifunction Device f E 100 \

Optical Proximity [speaker m] [Sensor 164 J [ Sensor E 3005

g S NI f >: D m3

§| Current Time \A 308 310 V\ Day and Date“ 312

5 318

5 316

w .

—) SIIde to unlock S 5 302 306

Touch Screen i k J

Microphone Home Acceler0meter(s) m 204 E

Figure 3B

US. Patent Jan. 20, 2009 Sheet 6 of 293 US 7,479,949 B2

Portable Multifunction Device & 109 \n

f \

Optical Proximity [speaker m) [ Sensor E] [ Sensor E 3000

a A \ N A 2 Current Time m I:- m

E Enter Passcode

QQQQ / \/ \/—\

1 2 3

r \/ \Ki 4 5 6

\ GHI /\ JKL /\ MNO , K \/ \fi

7 8 9

/i

C3" +

Acce|er0meter(s) E

Figure 3C

US. Patent Jan. 20, 2009 Sheet 7 of 293 US 7,479,949 B2

Portable Multifunction Device m 100 \‘

f \

Optical Proximity [Speaker m) ( Sensor m J [ Sensor E J 400A

°° 5 al - \

A E E Current Tlme M II m

§| m y o o o Q

Text Photos Camera Videos

m M E E

Jan 1 5

Weather Stocks Calendar M M E m

- ~_ User

.‘ ABC Created Widget

Calculator Alarm Dictionary Widget 149-3 149-4 149-5 _149-6

@410 [1] Phone Mail Browser Music

E m m E 2

Touch Screen l \ )

Microphone Home Accelerometer(s) m m m

k J

Figure 4A

US. Patent Jan. 20, 2009 Sheet 8 of 293 US 7,479,949 B2

Portable Multifunction Device E 100 \

f \

Optical Proximity [SpeaKermJ [Sensor?] [Sensor? 400B

00 5 8| / A . \ A m /» Current Tlme & >K ) '3' @

£ 00 2| © © @

Text Calendar Photos Camera

m E M E

Online Weather Video M E

Clock 149-4

414 410

Phone Mail Browser iPod

E E M E E

Touch Screen l k j

Home Accelerometer(s) m 204 E

k

Figure 4B

US. Patent Jan. 20, 2009 Sheet 9 of 293 US 7,479,949 B2

Portable Multifunction Device 206 100 \

Optical Proximity 500 [Speaker m) [ Sensor m J [ Sensor M S

r . \

A 2 Current Tlme M I]! m E

@ 512 Instant Messages ~0 502 (3

Jane Doe v‘ 504-1 5508-1 510-1 Hola v~ 506-1 Today 9

E

JOhn ‘A 504-2 I 5062 508-2 510-2

Are you going tonight? January 58, 2006 ; 408-123-4567“ 504-3 5508's 510-3 Bye “ 500-3 January 27,2006 2

Bob Adams “3 504-4 508-4 510-4

CU later“ 506-4 January §5, 2006 g

516

\ Touch Screen l )

Microphone Home Accelerometer(s) m 204 E

Figure 5

US. Patent

E

E

206

Jan. 20, 2009

Portable Multifunction Device

Sheet 10 of 293

100

US 7,479,949 B2

\

Optical Proximity [speakermj [Sens-mm] [sensor? 600A 5

f ' \

A @ Current Tlme M '3' 406

< Instant Messages ~'\ 602 Jane Doe“ 504-1

604-1 January 29, 2006 11:35 AM

R U there? A , 608-1

_ I 604 2 M

@ : January 30,2006 V 8:302AM

608-2

(Text box i l ( Send >~a614

0000000000

@ mm 0 l

Microphone Home Acceler0meter(s) m _2 04 E

Figure 6A

US. Patent Jan. 20, 2009 Sheet 11 of 293 US 7,479,949 B2

Portable Multifunction Device 206 1 0o \

Optical Proximity [speaker m) [ Sensor 164 J [ Sensor 166 6008 S

f E 4; O 6’) A m Current Time M .1

< Jane Doe“

R U there? 606-1

’ 604-1 IM is the best.

E

January 30, 2006 630

604-2 830 606_2 H 608-2

Q-low was your vacatl w l CSlDEM

@acant) @acate) @acation) w @t)

9009009090 mm eemm

A .?123

Microphone Home Acceler0meter(s) m _2 04 E

Figure 6B

US. Patent Jan. 20, 2009 Sheet 12 of 293 US 7,479,949 B2

Portable Multifunction Device E 100 \‘

r

Optical Proximity [Speaker m) [Sensor 164 J [ Sensor m 600C 5

r m A o 0D A m Current Time m n»

< "‘602 Jane Doe - 504-1

R U there? 6064 I .

604-1 IM IS the best. \

606-2 9%} January 30, 2006

604-2 8:30 AM? 6082 Hola

Q-low was your vacat| w l ( Send >314

@acant) @acate) @acation E @

628

24

Microphone Home Accelerometer(s) m _2 04 E

Figure 6C

E

k

US. Patent Jan. 20, 2009 Sheet 13 of 293 US 7,479,949 B2

206 100

Optical Proximity [Speaker m) [Sensor m J [ Sensor E 600D S

,

Portable Multifunction Device x

A @ Current Time m

Instant Messages 602 Jane Doe v\ 504-1

R U there? 606*

’ 604-1 IM is the best. \

@ January 30, 2006 6062 _ 8230 AM 604 2 2 Hola

608-2

Q-low was your vacat| i l (1% 4

@acant) @acate) @acation) w @

628

39 \ J

Microphone Home Acceler0meter(s) m 204 E

Figure 6D

E

US. Patent Jan. 20, 2009 Sheet 14 of 293 US 7,479,949 B2

Portable Multifunction Device m 100 \

,

Optical Proximity [Speaker m) [ Sensor 164 J [ Sensor E 600E

8 5 “I r A Q Current Time % .1 £9

§| < 602 Jane Doe“ 504-1

9% January 30, 2006 606'2 604-2 8:30 AM2

608-2 606-3

5

/@-|ow was your vacation? ‘How was your vacation?

Q-low was your vacation? m l @1314

oaoaom

Microphone Home Accelerometer(s) m M m

Figure 6E

US. Patent Jan. 20, 2009 Sheet 15 of 293 US 7,479,949 B2

Portable Multifunction Device E 100 x

,

Optical Proximity [Speaker m) [Sensor m) [Sensor @ 600F

w s O r . W

“I A & Current Tlme m I:- @

8| New Message 636% Cancel ) N

T02 Q 634 ~@

(Text box 2 l @14

90960900 9

@ mm .?123 Space Return Km E

Microphone Home Accelerometer(s) m 204 E

Figure 6F

US. Patent Jan. 20, 2009 Sheet 16 of 293 US 7,479,949 B2

Portable Multifunction Device 206 100 \

Optical Proximity [Speaker m) [ Sensor m [ Sensor E

6006 S

r M A Q Current Time M I1 £9

New Message 636@

634 "@ E

To: v| Q

Mike Van OS 38 650-123-2332

Venu A 712-666-3466 iv‘

Vick Tom g 642 872-739-2322 640

' 712-342-3455

@ @mm 2&3 [ Space W Reiurn

m

Microphone Home Acce|erometer(s) m 204 E

Figure 6G

Vivan

US. Patent Jan. 20, 2009 Sheet 17 of 293 US 7,479,949 B2

Portable Multifunction Device E 100 \

(

Optical Proximity [Speaker m) [Sensor M) [Sensor E 600H 5

F . X A Q Current Time M D 406

New Message 636% Cancel 1

To: Mike Van Os i 634@

E

E

dinner “644

@an you make dinI“ 656 wk @1314

\

.?123 dinner m Return \ m E

16 \

Microphone Home Accelerometer(s) m E E

Figure 6H

US. Patent Jan. 20, 2009 Sheet 18 of 293 US 7,479,949 B2

Portable Multifunction Device 206 100 \

OPtical Proximity 600' [Speaker m) ( Sensor m J [ Sensor M s

r . \

I A 2 Current Time m I1 E

New Message e36~@ To: Mike Van Os @ 634"@

208

E

656 5

{Can you make dinner'i'hl will - M 648-1

pick Dan from the day care. m @14

.?123 Space m Return m E

m

Microphone Home Acceler0meter(s) m 2 04 E

Figure 6|