1

Application-specific constraints for multimedia presentation generation

Joost Geurts, Jacco van Ossenbruggen and Lynda Hardman

CWI Amsterdam

email: Joost.Geurts@cwi.nl

2

Talk overview

•Generating multimedia automatically

•Cuypers multimedia generation engine

•Multimedia and constraints–Quantitative constraints

–Qualitative constraints

•Cuypers demo

•Conclusion, future directions

3

Generating adaptive multimedia

•Content–Large multimedia database

•System profile–PC, PDA, WAP

•Network profile–Modem, Gigabit

•User profile–Language, Interests, Abilities, Preferences

Too costly to author manually

4

Cuypers multimedia generation engine

5

Automatic multimedia generation

•Designer does not specify complete presentation……but only specifies requirements

•System automatically finds a solution which meets requirements

•How should the requirements be specified?–Declarative constraints

6

Traditional use of constraints

•Constraint solving used for problems with:–Many variables–Large domains

•Based on domain reduction paradigm•Quantitative constraints

– Integer domain–Reduction by arithmetic relations

• Greater than (>)• Less than (<)• Equals (=)

7

Drawbacks of quantitative constraints

•Too many (trivial) solutions that differ by:–1 pixel position, or–1 milliseconds in timing

•Not sufficiently expressivee.g. cannot specify “no overlap” constraint

•Too low levele.g. A.X2 B.X1

8

Solution: qualitative constraints

•Example “A not overlap B”, “B after C”•Advantages:

–More intuitive–More expressive–Smaller domains

•Qualitative solutions translate automatically to lower level quantitative constraints

•New problem: What if constraints are insoluble?

9

Solution: Constraint Logic Programming

•Combine Prolog unification and backtracking with constraint solving

•Use Prolog rules to generate constraints•Backtrack when constraints are insoluble

10

Cuypers generation engine

•Multiple layers:

–Prolog rules to generate

constraints

–Qualitative constraints

translate to quantitative

constraints

–Solution of both constraints

provides sufficient

information for final

presentation

11

Cuypers demo: scenario

•Client: User is interested in Rembrandt and wants to know about about the “chiaroscuro” technique

•Server: Query database•Server: Generate constraints according to:

–System profile–User profile–Network profile

•Server: Solve constraints / revise constraints

•Server: Generate SMIL presentation•Client: Play presentation

12

Conclusions

•Quantitative constraintsare insufficient for automatic multimediapresentation generation. Also need

•Qualitative constraintsto allow intuitive and effectivehigh level specification, and

•Backtrackingfor revising specific constraintsthat cause the entire set to fail

13

Future directions

•Best-first instead of depth-first–Choose “best” among possible solutions–Needs evaluation criteria

•Improve knowledge management–Make design knowledge declarative and explicit

–Preserve metadata in final presentation–Use standardized and reusable profiles

14

15

Need to make trade-offs

•Semantics–Convey message

•Aesthetics–Clear / nice layout

•Resources–Screen size, bandwidth

•Dimension may result in conflicting goals

16

Quantitative Constraints% csp(+Ids, -Boxes)csp([IdA,IdB],[box(IdA,[x1:AX1, …]), box(IdB,[x1:BX1,…])]) :-

% get valuesmaxX(MaxX), maxY(MaxY),height(IdA,HeightA),widtht(IdA,WidthA),…% define domains

[AX1,AX2,BX1,BX2]::[0..MaxX], [AY1,AY2,BY1,BY2]::[0..MaxY],

% set width & heightAX2 – AX1 #= WidthA,AY2 – AY1 #= HeightA,…% constraintsAX2 #< BX1, % A left-of BAY1 #= BY1, % A top-align B,…true.

17

Multimedia and Constraints

•Constraint Logic Programming–Domain reduction–Backtracking–Unification (matching rules)

•Qualitative Constraints–Non-integer domain–Allen’s 13 temporal interval relations in three dimensions

18

Qualitative Constraints

•Example:Two images, A,BA left or right of BA not above or

below B

19

Qualitative Constraints

% csp(+Ids, -Graph)csp([IdA, Idb], [edge(IdA,IdB,x,NoOverlap),…]) :-

% define domainsNoOverlap :: [b,b-,m,m-],Overlap :: [d,d-,s,s-,f,f-,e],

% constraintsedge(IdA,IdB,x,NoOverlap), % B not-overlap Aedge(IdA,IdB,y,Overlap), % B overlap Atrue.

20

Qualitative Constraints

•Reasoning– Inverse:

edge(A,B,D,Value) <=> inverse(Value,RValue),edge(B,A,D,RValue).

– Equalityedge(A,B,D,V1), edge(A,B,D,V2) => V1 #= V2

– Transitiveedge(A,B,D,VAB), edge(B,C,D,VBC) =>

tr(VAB,VBC,VAC), % rule generation algorithm

edge(A,C,D,VAC).

•Translation rules to quantitative domainedge(A,B,D,b) => node(A,D/2,V2), node(B,D/1,V1)

V1 #< V2.

21

Problems in generating multimedia

•Text documents are flexible–Add page, scrollbar,

–Template models

–Wrap text around images

•Multimedia documents are less flexible–No pages or scrollbars, no line-breaking or hyphenation

–Not based on text-flow–Feedback needed

•Linear process model does not work for multimedia

22

Quantitative Constraints

•Example:Two images,

A,BA left-of BA top-align B

23

Cuypers generation engine

•Rhetoric/Semantic–Sequence, Example

•Communicative devices–Bookshelf, Slideshow

•Qualitative Constraints–A before B

•Quantitative Constraints–A.X2 < B.X1

•Presentation–SMIL