BX Benchmarks

Working Group ResultsBIRS Center, Dec. 3, 2013

Addressed Questions

• Which categories of bx benchmarks do exist?• What´s the difference between a bx example

and a bx benchmark?• What are bx benchmarks expected to measure?• What about formats for bx benchmark

descriptions?• [ Discussion of meaningfulness of

„incrementality“ in state-based scenario ]

Categories of bx Benchmarks

• Functional bx benchmarks address selected features of bx approaches:– Properties of consistency relations– Types of transformation scenarios– Laws for bx approaches– [ Appropriateness for categories of data ? ]– [ Platform dependency ? ]

• Non-functional bx benchmarks: …

Categories of bx Benchmarks

• … Non-functional bx benchmarks address bx transformation characteristics like:– Runtime / memory consumption– Scaleability w.r.t. to size of data, spec. …– Size of propagated deltas– [ Loss of information after update propagation ]– …

bx Benchmark Properties

A bx benchmark is a bx example (with an optional additional characterization w.r.t. to a bx taxonomy) that has a• precise and executable definition of a binary consistency

relation on source and target data (that can be used as an oracle for a bx application)

• an explicit definition of or a generator for input data elements (for a bx application)

• set of precisely defined update scenarios for certain input data elements

• set of executable metrics definitions (with an SE interpretation of the word „metrics“)

Further bx Benchmark Properties

• A „useful“ bx benchmark should capture the essence of certain aspects of a class of real-world bx application scenarios

• bx descriptions may exist on different levels of abstraction with varying degrees of dependence on a specific execution platform (PIM bx benchmark description, … )