Automated Staging for Virtual CinematographyAutomated Staging for Virtual Cinematography Amaury...
Transcript of Automated Staging for Virtual CinematographyAutomated Staging for Virtual Cinematography Amaury...
Automated Staging for Virtual
Cinematography
Amaury Louarn Marc Christie Fabrice Lamarche
2018-11-08
IRISA / Inria Rennes Atlantique
MimeTIC team
1
What is Staging?
Staging is the process of correctly
positioning actors, lights and viewpoints
in order to have a pleasing aesthetic
view for the spectator.
Viewpoint:
Theatre: Fixed (public)
Movies: Cameras in environment,
Image composition
Hamilton, by Lin-Manuel Miranda
Back to the future, by Robert Zemeckis
2
Staging in video games
In video games, staging concepts are mostly used for:
• Cutscenes
Figure 1: Battlefield 1,
by Electronic Arts
• Interactive drama
Figure 2: The Walking Dead,
by Telltale Games
3
Automated staging?
4
Automated staging?
• 7 DoF per camera (Position, Orientation, Focal Length)
4
Automated staging?
• 7 DoF per camera (Position, Orientation, Focal Length)
• 6+ DoF per actor (Position, Orientation, Rig joints)
4
Automated staging?
• 7 DoF per camera (Position, Orientation, Focal Length)
• 6+ DoF per actor (Position, Orientation, Rig joints)
• Environment constraints (Obstacles, occlusions, etc.)
4
Automated staging?
• 7 DoF per camera (Position, Orientation, Focal Length)
• 6+ DoF per actor (Position, Orientation, Rig joints)
• Environment constraints (Obstacles, occlusions, etc.)
High complexity:
Everything is interdependent
4
Staging in litterature - Camera placement
Multiple approaches to camera placement:
• Constraint solving (Bares et al. 2000)
• Numerical optimization (Drucker et al. 1992; Olivier et al.
1999; Ranon et al. 2014)
• Geometrical approach (Lino et al. 2010)
5
Staging in litterature - Actor placement (1)
Talbot 2015:
Actor placement for theatre plays
• Fixed viewpoint (public)
• Simple environment (theatre stage)
• Spring-mass system
Fig: Spring-mass
staging system
6
Staging in letterature - Actor placement (2)
Elson et al. 2007:
Actor and camera placement for machinimas
• Free viewpoint
• Precomputed library of actor-camera
configurations
Fig: Staging library
Fig: Stage placement
7
Limits of current approaches
Talbot 2015:
• No cameras
• Simple environments
Elson et al. 2007:
• Finite library of stagings
• Simple scenes
Common hypothesis:
Staging can be done in 2D with satisfactory results.
8
Limits of current approaches
Talbot 2015:
• No cameras
• Simple environments
Elson et al. 2007:
• Finite library of stagings
• Simple scenes
Common hypothesis:
Staging can be done in 2D with satisfactory results.
Goals of our contribution:
• No a priori knowledge
• Arbitrary complex scenes
• High-level specification
8
Overview
Two-fold contribution:
1. A staging description language
• Based on Prose Storyboard Language by Ronfard et al. 2015
• Staging constraints: distance, orientation, visibility, etc.
2. A staging resolution engine
9
Pre-process - Environment
2D Topological map extraction from lightly informed
3D environment:
10
Pre-process - Staging Description
Geometric constraints identification from staging description:
• Distance• (Entity) is close to (Element)
• (Entity) is at [least|most] (Value) from
(Element)
Actor is close to Bar
11
Pre-process - Staging Description
Geometric constraints identification from staging description:
• Distance
• Orientation• (Entity) is (facing | left of | ... ) (Entity)
• (Entity) looks at (Element)
A looks at B. B looks at A
11
Pre-process - Staging Description
Geometric constraints identification from staging description:
• Distance
• Orientation
• Visibility• (Entity) is [not] seeing (Element)
A is seeing bar
11
Pre-process - Staging Description
Geometric constraints identification from staging description:
• Distance
• Orientation
• Visibility
• Framing• (Camera), (Size) on (Entity) [(Profile)]
[(Screen position)] {and (Size) on (Entity)
[(Profile)] [(Screen Position)]}
Camera 1, MS on A and MS on B
11
Engine - Overview
Algorithm in 2 Steps:
1. Search-space pruning
with constraints
2. Progressive sampling
via elicitation
12
13
Engine - Search space pruning
Toy example:
• A near bar
• B near A
• B in chair
Fixed-point process:
1. Compute geometric regions where constraints are satisfied
2. Deduce allowed regions for entities from it
3. Propagate the new entity regions to the constraints
4. Repeat until all entities are at their smallest allowed regions
14
Engine - Search space pruning
Toy example:
• A near bar
• B near A
• B in chair
Fixed-point process:
1. Compute geometric regions where constraints are satisfied
2. Deduce allowed regions for entities from it
3. Propagate the new entity regions to the constraints
4. Repeat until all entities are at their smallest allowed regions
14
Engine - Search space pruning
Toy example:
• A near bar
• B near A
• B in chair
Fixed-point process:
1. Compute geometric regions where constraints are satisfied
2. Deduce allowed regions for entities from it
3. Propagate the new entity regions to the constraints
4. Repeat until all entities are at their smallest allowed regions
14
Engine - Search space pruning
Toy example:
• A near bar
• B near A
• B in chair
Fixed-point process:
1. Compute geometric regions where constraints are satisfied
2. Deduce allowed regions for entities from it
3. Propagate the new entity regions to the constraints
4. Repeat until all entities are at their smallest allowed regions
14
Engine - Search space pruning
Toy example:
• A near bar
• B near A
• B in chair
Fixed-point process:
1. Compute geometric regions where constraints are satisfied
2. Deduce allowed regions for entities from it
3. Propagate the new entity regions to the constraints
4. Repeat until all entities are at their smallest allowed regions
14
15
Engine - Sampling
Toy example:
• A near bar
• B near A
• B in chair
1. Choose an entity (using a heuristic)
2. Choose a sample using a uniform distribution on its region
3. Apply a fixed-point process to propagate its new position and
orientation on other entities
4. If this yields an unsolvable situation, discard everything and
start the sampling process again
5. Else, continue with another entity
16
Engine - Sampling
Toy example:
• A near bar
• B near A
• B in chair
1. Choose an entity (using a heuristic)
2. Choose a sample using a uniform distribution on its region
3. Apply a fixed-point process to propagate its new position and
orientation on other entities
4. If this yields an unsolvable situation, discard everything and
start the sampling process again
5. Else, continue with another entity
16
Engine - Sampling
Toy example:
• A near bar
• B near A
• B in chair
1. Choose an entity (using a heuristic)
2. Choose a sample using a uniform distribution on its region
3. Apply a fixed-point process to propagate its new position and
orientation on other entities
4. If this yields an unsolvable situation, discard everything and
start the sampling process again
5. Else, continue with another entity
16
Engine - Sampling
Toy example:
• A near bar
• B near A
• B in chair
1. Choose an entity (using a heuristic)
2. Choose a sample using a uniform distribution on its region
3. Apply a fixed-point process to propagate its new position and
orientation on other entities
4. If this yields an unsolvable situation, discard everything and
start the sampling process again
5. Else, continue with another entity
16
Results
17
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A.
18
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and
in chair.
18
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and
in chair. A and B and Camera 1 and Camera 2 are not seeing C.
18
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and
in chair. A and B and Camera 1 and Camera 2 are not seeing C.
Camera 1, CU on B front screenright and BCU on A back screenleft.
18
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and
in chair. A and B and Camera 1 and Camera 2 are not seeing C.
Camera 1, CU on B front screenright and BCU on A back screenleft.
Camera 2, MCU on A right screenleft and B left screenright.
18
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and
in chair. A and B and Camera 1 and Camera 2 are not seeing C.
Camera 1, CU on B front screenright and BCU on A back screenleft.
Camera 2, MCU on A right screenleft and B left screenright. Camera
3, FS on A and B and C.
18
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and
in chair. A and B and Camera 1 and Camera 2 are not seeing C.
Camera 1, CU on B front screenright and BCU on A back screenleft.
Camera 2, MCU on A right screenleft and B left screenright. Camera
3, FS on A and B and C. Camera 4, FS on C. Camera 4 is not seeing A
and B.
18
Results - complex scenario (1)
Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and
in chair. A and B and Camera 1 and Camera 2 are not seeing C.
Camera 1, CU on B front screenright and BCU on A back screenleft.
Camera 2, MCU on A right screenleft and B left screenright. Camera
3, FS on A and B and C. Camera 4, FS on C. Camera 4 is not seeing A
and B.
Scene 2: A and B are in same position as in scene 1. C is at
most 4 meters behind A and facing A. Camera 1, same position and
orientation as in scene 1. Camera 2, MLS on B screenleft and A
screencenter and C screenright. Camera 3, FS on A and B and C.
Camera 4, MS on C. Camera 4 is not seeing A and B.
18
Results - complex scenario (2)
Video
19
Results - complex scenario (3)
Scene 1:
CU on B front screenright and
BCU on A back screenleft
FS on A and B and C
MCU on A right screenleft and
B left screenright
FS on C. Camera 4 is not
seeing A and B
20
Results - complex scenario (2)
Scene 2:
Same position and orientation
as in scene 1
FS on A and B and C.
MLS on B screenleft and A
screencenter and C screenright
MS on C. Camera 4 is not
seeing A and B.
21
Results - Real-life comparison (1)
Comparison with Back to the future by Robert Zemeckis:
Scene 1: George is in chair
facing the bar. Marty is in
chair facing George. Camera
1, CU on Marty 3/4 left
screencenter and George left
screenright.
Scene 2: Marty is in same
position and orientation
as in Scene 1. George is
facing Marty. Camera 1, MCU
on George front screencenter
and Marty 3/4 backright
screenleft.
Scene 3: Marty and George
are in same position as in
Scene 2. Goldie is on the
left of Marty and on the
right of George. Camera
1, MLS on Goldie front
screencenter.
22
Results - Real-life comparison (1)
Comparison with Back to the future by Robert Zemeckis:
Scene 1: George is in chair
facing the bar. Marty is in
chair facing George. Camera
1, CU on Marty 3/4 left
screencenter and George left
screenright.
Scene 2: Marty is in same
position and orientation
as in Scene 1. George is
facing Marty. Camera 1, MCU
on George front screencenter
and Marty 3/4 backright
screenleft.
Scene 3: Marty and George
are in same position as in
Scene 2. Goldie is on the
left of Marty and on the
right of George. Camera
1, MLS on Goldie front
screencenter.
22
Results - Real-life comparison (2)
Scene 1: Scene 2: Scene 3:
23
Summary
Our contribution is:
• a high-level staging description language
• a resolution engine implementing our language
The resolution engine:
• is not limited to a precomputed set of actor-camera
configurations
• works on any arbitrary scene
• can propose alternative stagings
24
Future works
Example: Wes Anderson’s symmetry
Fantastic Mr. Fox Moonrise Kingdom The Grand Budapest Hotel
Example: Quentin Tarantino’s trunk shot
Reservoir Dogs Pulp Fiction Kill Bill vol. 1
• How can we extend our language to more aesthetic shots?
• How can we model a movie style and extract it?
• How can we apply a movie style to new scripts?
25
References
Bares, W., McDermott, S., Boudreaux, C., & Thainimit, S. (2000).
Virtual 3D Camera Composition from Frame Constraints. In
ACM International Conference on Multimedia.
Drucker, S., Galyean, T., & Zeltzer, D. (1992). CINEMA: A System
for Procedural Camera Movements. In SI3D ’92: Proceedings
of the 1992 symposium on Interactive 3D graphics
(pp. 67–70). Cambridge, Massachusetts, United States: ACM
Press.
26
Elson, D., & Riedl, M. (2007). A Lightweight Intelligent Virtual
Cinematography System for Machinima Production. In
Proceedings of the Third AAAI Conference on Artificial
Intelligence and Interactive Digital Entertainment (pp. 8–13).
AIIDE’07. Stanford, California: AAAI Press.
Lino, C., Christie, M., Lamarche, F., Schofield, G., & Olivier, P.
(2010). A Real-time Cinematography System for Interactive
3D Environments. In ACM SIGGRAPH/Eurographics
Symposium on Computer Animation.
Olivier, P., Halper, N., Pickering, J. H., & Luna, P. (1999). Visual
Composition as Optimisation. In AISB Symposium on on AI
and Creativity in Entertainment and Virsual Art (pp. 22–30).
Ranon, R., & Urli, T. (2014). Improving the Efficiency of Viewpoint
Composition. IEEE Transactions on Visualization and
Computer Graphics, 20(5), 795–807.
27
Ronfard, R., Gandhi, V., & Boiron, L. (2015). The Prose
Storyboard Language: A Tool for Annotating and Directing
Movies. CoRR, abs/1508.07593.
Talbot, C. (2015). Directing virtual humans using play-script
spatiotemporal reasoning. (Doctoral dissertation, University
of North Carolina at Charlotte).
28