Data centric Design & Operation: A data-driven and scientific approach for game business
Collaborative Infrastructure for Test-Driven Scientific ...comar/sciunit-icse14.pdf · Test-Driven...
Transcript of Collaborative Infrastructure for Test-Driven Scientific ...comar/sciunit-icse14.pdf · Test-Driven...
Collaborative Infrastructure for Test-Driven Scientific Model Validation
Cyrus Omar + Jonathan Aldrich carnegie mellon university !Richard Gerkin arizona state university
Model Prediction
2
Empirical Observations
Observable Quantity
3
Science is Dynamic
4
Science is Dynamic
5
Science is Dynamic
Empirical Observations
Models Goodness-of-Fit
7
8
The state of the art?
Other [models] were examined and found promising but were not considered due to our inability to test the techniques”
“
9
10
papers ~ component documentationimplementations
models ~ software componentsscientific community ~ software development team
conformance testsinterfaces
11
models ~ software componentsmodel capabilities ~ interfaces
observable quantities ~ typesgoodness-of-fit tests ~ unit tests
empirical observations ~ test parameters
interface SunspotNumberPredictor predict : TimeSeries<SunspotNumber> -‐> TimeSeries<SunspotNumber>
SunspotTest : TimeSeries<SunspotNumber> -‐> Test<SunspotNumberPredictor, Metric> = ...
TestSuite([SunspotTest(cycle_data) for cycle_data in all_cycle_data]) .judge([OhlsMethod, FeynmansMethod, ThompsonsMethod]).view()
OhlsMethod FeynmansMethod ThompsonsMethod
summary tables ~ test result summaries
: SunspotNumberPredictor = ... : SunspotNumberPredictor = ... : SunspotNumberPredictor = ...
12
interface SunspotNumberPredictor predict : TimeSeries<SunspotNumber> -‐> TimeSeries<SunspotNumber>
SunspotTest : TimeSeries<SunspotNumber> -‐> Test<SunspotNumberPredictor, Metric> = ...
TestSuite([SunspotTest(cycle_data) for cycle_data in all_cycle_data]) .judge([OhlsMethod, FeynmansMethod, ThompsonsMethod]).view()
OhlsMethod FeynmansMethod ThompsonsMethod
: SunspotNumberPredictor = ... : SunspotNumberPredictor = ... : SunspotNumberPredictor = ...
which programming language? a DSL?Paper: SciUnit, a Python library (widely adopted, good FFI support, simple syntax, IPython notebook)
collaborative workflow?Paper: SciDash, a wiki pointing to IPython notebooks hosted on Github
adoption?Paper: people who organize modeling competitions (e.g. a neural modeling competition we are helping re-start)
Ongoing work: automatic test and model generation from existing repositoriesContact Me: [email protected] or @neurocy
Model Prediction
13
Empirical Observations
Observable Quantity
14
Test
Data Type
Model Prediction
Unit Tests
Models Goodness-of-Fit