Demonstrating and Testing the BML Compliance of BML Realizers

26
Demonstrating and Testing the BML Compliance of BML Realizers Herwin van Welbergen, Yuyu Xu, Marcus Thiebaux, Wei-Wen Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro

description

Demonstrating and Testing the BML Compliance of BML Realizers. Herwin van Welbergen, Yuyu Xu , Marcus Thiebaux, Wei-Wen Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro. What is this talk about. Testing the adherence to the BML standard by realizers - PowerPoint PPT Presentation

Transcript of Demonstrating and Testing the BML Compliance of BML Realizers

Page 1: Demonstrating and Testing the BML Compliance of BML  Realizers

Demonstrating and Testing the BML Compliance of BML

RealizersHerwin van Welbergen, Yuyu Xu, Marcus Thiebaux, Wei-Wen Feng,

Jingqiao Fu, Dennis Reidsma, Ari Shapiro

Page 2: Demonstrating and Testing the BML Compliance of BML  Realizers

Testing the adherence to the BML standard by realizers

(and, by this, also testing for weaknesses in the standard)

Better software engineering practices for virtual humans◦ Daily automated testing (regression, acceptance,

…)

What is this talk about

Page 3: Demonstrating and Testing the BML Compliance of BML  Realizers

BML Realizer Interface:SAIBA Framework

Page 4: Demonstrating and Testing the BML Compliance of BML  Realizers

BML Realizer Interface: BML goes in

Page 5: Demonstrating and Testing the BML Compliance of BML  Realizers

bml1 starts (globaltime=10) gaze1 starts (localtime=0) gaze1 ready (localtime=1) speech1 start (localtime=1) … speech1 end (locatime=10) bml1 ends (globaltime=20)

BML Realizer Interface: Feedback comes out

Page 6: Demonstrating and Testing the BML Compliance of BML  Realizers

BML Realizers are complex software components

They form the backbone of virtual human applications by research groups

Yet their testing has so far been limited to time consuming manual inspection of the execution of BML scripts

Why automatic testing of realizers?

Page 7: Demonstrating and Testing the BML Compliance of BML  Realizers

Automatic tests are automatic◦ No wasted developer time

Automatic tests can be run often◦ Early error detection

Why automatic testing of realizers?

Page 8: Demonstrating and Testing the BML Compliance of BML  Realizers

A realizer provides standardized interface◦ BML goes in◦ Feedback comes out

Several realizers implement this interface We can make automatic tests for this

interface These tests can be used for all realizers

implementing the interface

How can we test a realizer?

Page 9: Demonstrating and Testing the BML Compliance of BML  Realizers

Message flow and behavior execution Adherence to time constraints Error handling Acceptance testing

What can be tested?

Page 10: Demonstrating and Testing the BML Compliance of BML  Realizers

Send BML to a realizer, capture all feedback Wait until the realizer has executed the BML Verify assertions on the received feedback

Typical test case

Page 11: Demonstrating and Testing the BML Compliance of BML  Realizers

<bml id="bml1"><speech id="speech1" start="6"><text>Hey punk <sync id="s1" />what do ya want?</text></speech><head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

What can we assert about the expected feedback?- All default sync points remain in order for all behaviors- The head nod will start within |ε| seconds of the start of “what”- Speech1 starts within |ε| seconds of t=6- There will be no exceptions or warnings- The block will end

……

Example test case

Page 12: Demonstrating and Testing the BML Compliance of BML  Realizers

<bml id="bml1"><speech id="speech1" start="6"><text>Hey punk <sync id="s1" />what do ya want?</text></speech><head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

@Test public void testSpeechNodTimedToSync() {realizerPort.performBML(readTestFile("testspeech_nodtimedtosync.xml"));waitForBMLEndFeedback("bml1");assertSyncsInOrder("bml1", "speech1", "start", "ready", "stroke_start", "stroke", "s1", "stroke_end", "relax", "end");assertAllBMLSyncsInBMLOrder("bml1", "nod1");assertBlockStartAndStopFeedbacks("bml1");assertRelativeSyncTime("bml1", "speech1", "start", 6);assertLinkedSyncs("bml1", "speech1", "s1", "bml1", "nod1", "start");assertNoExceptions();assertNoWarnings();

}

Example test case

Page 13: Demonstrating and Testing the BML Compliance of BML  Realizers

A generic testing framework that can test all BML Realizers

Provides a set of tests + assertions And functionality to easily author new tests

◦ Utility methods (e.g. waitForBMLEndFeedback)◦ Custom assertions (e.g. assertLinkedSyncs)

Code and tests released on SF, together with a nice set of example movies to showcase various realizers

RealizerTester

Page 14: Demonstrating and Testing the BML Compliance of BML  Realizers

Using RealizerTester◦ Additional tests for Elckerlyc specific functionality◦ Regression testing◦ Acceptance testing

Running all tests takes some time (~3 min.)◦ This might discourage frequent testing by

developers◦ => Run tests automatically on a continuous

integration server Notify developers that commit a build that fails tests Keep track of test result history

Testing Elckerlyc

Page 15: Demonstrating and Testing the BML Compliance of BML  Realizers

RealizerTester does not help in identifying the exact location of errors◦ It tests a Realizer as a black box◦ Elckerlyc uses additional white box testing at a

smaller granularity to identify the exact location of errors >1000 unit + mid range tests, running in <10s RealizerTester helps in identifying locations that

require more unit testing Regular visual inspection is still valuable!

Testing Elckerlyc

Page 16: Demonstrating and Testing the BML Compliance of BML  Realizers

Visual regression testing◦ Record a baseline◦ Does it (approximately) look

like the baseline?◦ Automatically check

difference with baseline◦ If different

Bug: fix New functionality: create new

baseline

Testing in SmartBody

Page 17: Demonstrating and Testing the BML Compliance of BML  Realizers

Testing in SmartBody: example

Page 18: Demonstrating and Testing the BML Compliance of BML  Realizers

Visual regression testing:◦ checks whether the correct motion is generated

rather than whether the correct signals are sent◦ is Realizer and character dependent◦ does not provide acceptance testing

RealizerTester is complementary to visual regression testing

Testing in SmartBody

Page 19: Demonstrating and Testing the BML Compliance of BML  Realizers

Dealing with BML versions Took 1 day to implement Found some unclarities in BML (e.g.

feedback ordering) now addressed in 1.0 Found some minor implementation issues in

SmartBody Did not find any interpretation differences in

the constraint satisfaction

Testing SmartBody using RealizerTester

Page 20: Demonstrating and Testing the BML Compliance of BML  Realizers

Observations: au 1

Page 21: Demonstrating and Testing the BML Compliance of BML  Realizers

Observations: gaze

Page 22: Demonstrating and Testing the BML Compliance of BML  Realizers

Observations: longer performance

Page 23: Demonstrating and Testing the BML Compliance of BML  Realizers

Designing test cases + assertions◦ Highlighted several cases in which the current BML

specification lacked detail or was unclear Setting up a video corpus

◦ Highlighted some expressivity issues in the BML standard

◦ Shows that several behaviors are executed in a semantically equivalent manner

◦ Created a healthy competition between the developers

◦ Motivated developers to move to better BML compliance

Observations

Page 24: Demonstrating and Testing the BML Compliance of BML  Realizers

The modularity proposed by SAIBA enables the reuse of testing functionality

Designing a generic testing framework and a shared video repository helps move the standard forward

BMLRealizerTester provides a starting point for a BML compliance test suite

Conclusions

Page 25: Demonstrating and Testing the BML Compliance of BML  Realizers

XML format to author tests using BML and assertions expressed?

Add more realizers (PLEASE CONTACT US!)◦ Full BML compliance is not required◦ Support for feedback is required

Include newest version of BML standard discussed last Wednesday

Continue extending our repository of nice example movies

Further work

Page 26: Demonstrating and Testing the BML Compliance of BML  Realizers

http://realizertester.sourceforge.net/

Thanks for your attention

contact: [email protected]