Robert Meyer- pypet
-
Upload
pydata -
Category
Data & Analytics
-
view
229 -
download
0
Transcript of Robert Meyer- pypet
![Page 1: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/1.jpg)
pypet PyData 2016
Robert Meyer
Neural Information Processing, TU-Berlin
![Page 2: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/2.jpg)
Motivation
The cycle of simulations and proto-typing
Idea
CodingSimulations
![Page 3: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/3.jpg)
Motivation
The cycle of simulations (ground truth)
Idea
Spaghetticode
Parameter exploration bash scripts
Hacky I/Ofunctionalityto store data
Data Analysisthat only works with
my current I/O
![Page 4: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/4.jpg)
Motivation
The cycle of simulations (ground truth)
Uhm, maybe I shouldtry something else
Spaghetticode
Parameterexplorationbash scripts
Hacky I/Ofunctionalityto store data
Data Analysisthat only works with
my current I/O
![Page 5: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/5.jpg)
Motivation
The cycle of simulations (ground truth)
Spaghetticode
Parameterexplorationbash scripts
Hacky I/Ofunctionalityto store data
Data Analysisthat only works with
my current I/OLet's repeat an old simulation
Uhm, maybe I shouldtry something else
![Page 6: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/6.jpg)
Motivation
The cycle of simulations (ground truth)
Spaghetticode
Parameterexplorationbash scripts
Hacky I/Ofunctionalityto store data
Data Analysisthat only works with
my current I/OLet's repeat an old simulation
Uhm, maybe I shouldtry something else
![Page 7: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/7.jpg)
Motivation
The cycle of simulations (ground truth)
Uhm, maybe I shouldtry something else
Spaghetticode
Parameterexplorationbash scripts
Hacky I/Ofunctionalityto store data
Data Analysisthat only works with
my current I/O ???
![Page 8: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/8.jpg)
Motivation
The cycle of simulations (ground truth)
Uhm, maybe I shouldtry something else
Spaghetticode
Parameterexplorationbash scripts
Hacky I/Ofunctionalityto store data
Data Analysisthat only works with
my current I/O
pypet
![Page 9: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/9.jpg)
pypet
python parameter exploration toolbox Suitable for any numerical simulation in python
![Page 10: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/10.jpg)
pypet
python parameter exploration toolbox Suitable for any numerical simulation in python
Main objectives Allow easy exploration of the parameter space
Easy storage and retrieval of computed resultsand the underlying set of parameters
![Page 11: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/11.jpg)
pypet
Evolves around the trajectory container Tree Data Structure to manage parameters
and results Group your parameters according to meaningful categories
>>> traj.f_add_parameter('traffic.ncars', 100, comment = 'Number of vehicles')>>> traj.f_add_parameter('traffic.cyclists', 42, comment='Number of cyclists')>>> traj.f_add_parameter('simulation.dt', 0.01, comment='Timestep')
![Page 12: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/12.jpg)
pypet
Evolves around the trajectory container Tree Data Structure to manage parameters
and results Group your parameters according to meaningful categories
>>> traj.f_add_parameter('traffic.ncars', 100, comment = 'Number of vehicles')>>> traj.f_add_parameter('traffic.cyclists', 42, comment='Number of cyclists')>>> traj.f_add_parameter('simulation.dt', 0.01, comment='Timestep')
Access your data with Natural Naming >>> traj.traffic.ncars
100
Or cut short through the tree >>> traj.cyclists
42
![Page 13: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/13.jpg)
pypet
Evolves around the trajectory container Handles a variety of data formats
Python Natives (bool, int, float, complex, str) Python Containers (dict, list, tuple)
NumPy Arrays and Matrices, Scipy Sparse Matrices Pandas DataFrames BRIAN(2)_ Quantities and Monitors (Everything you can pickle)
_BRIAN(2): http://briansimulator.org/
![Page 14: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/14.jpg)
pypet
Evolves around the trajectory container Handles a variety of data formats
Python Natives (bool, int, float, complex, str) Python Containers (dict, list, tuple)
NumPy Arrays and Matrices, Scipy Sparse Matrices Pandas DataFrames BRIAN(2)_ Quantities and Monitors (Everything you can pickle)
Your work with something else? Easily extendible for other parameters and results
_BRIAN(2): http://briansimulator.org/
![Page 15: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/15.jpg)
pypet
Evolves around the trajectory container Tree Data Structure to manage parameters
and results
Allows arbitrary exploration of the parameterspace Beyond grids (Cartesian product space) and
straight lines
![Page 16: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/16.jpg)
pypet
Evolves around the trajectory container Tree Data Structure to manage parameters
and results
Allows arbitrary exploration of the parameterspace beyond grids (cartesian product lines
ncars
cyclists
![Page 17: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/17.jpg)
pypet
Evolves around the trajectory container Tree Data Structure to manage parameters
and results
Allows arbitrary exploration of the parameterspace beyond grids (cartesian pr lines
ncars
cyclists
![Page 18: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/18.jpg)
pypet
Evolves around the trajectory container Tree Data Structure to manage parameters
and results
Allows arbitrary exploration of the parameterspace beyond grids (cartesian produ lines
cyclists
ncars
![Page 19: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/19.jpg)
pypet
HDF5_ (Hierarchical Data Format) Portable file format for managing and storing data
Can handle large amounts of data Human readable data
You can browse within your data as in a regular file system
Your data is automatically stored using the PyTables_ interface
_HDF5: http://www.hdfgroup.org/HDF5_PyTables: http://www.pytables.org/
![Page 20: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/20.jpg)
pypet
Standing on the shoulder of giants HDF5_ (Hierarchical Data Format)
Portable file format for managing and storing data You can browse within your data as in a regular file system
![Page 21: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/21.jpg)
pypet Provides and environment to disentangle
your actual scientific simulation from I/O management and execution
Manages logging
![Page 22: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/22.jpg)
pypet Provides and environment to disentangle
your actual scientific simulation from I/O management and execution
Manages logging
Automatic integration with git version control
![Page 23: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/23.jpg)
pypet Provides and environment to disentangle
your actual scientific simulation from I/O management and execution
Manages logging
Automatic integration with git version control
Allows adaptive exploration (optimization with DEAP_)
_DEAP: http://deap.readthedocs.org
![Page 24: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/24.jpg)
pypet Provides and environment to disentangle
your actual scientific simulation from I/O management and execution
Manages logging
Automatic integration with git version control
Allows adaptive exploration (optimization with DEAP_)
Allows parallel simulations and multiprocessing(on multiple serves or clusters like SGE/qsub using SCOOP_)
_DEAP: http://deap.readthedocs.org _SCOOP: http://scoop.readthedocs.org
![Page 25: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/25.jpg)
pypet
Live Demo (although you shouldn't do any)!
>>> from pypet import Environment
![Page 26: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/26.jpg)
Even more important! Open source: project available on github_
Fully documented on readthedocs_
pypet
_github: https://github.com/SmokinCaterpillar/pypet
_readthedocs: https://pypet.readthedocs.org/en/latest/
![Page 27: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/27.jpg)
Even more important! Open source: project available on github_
Fully documented on readthedocs_
Thoroughly tested More than 1000 tests, 91% code coverage Continuous integration testing with Travis_ (Linux) and
Appveyor_ (Windows)
pypet
_github: https://github.com/SmokinCaterpillar/pypet
_readthedocs: https://pypet.readthedocs.io
_Travis: https://travis-ci.org/SmokinCaterpillar/pypet
_Appveyor:https://ci.appveyor.com/project/SmokinCaterpillar/pypet
![Page 28: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/28.jpg)
pypet
Where to get pypet (for python 2.6 till 3.5) ? https://pypi.python.org/pypi/pypet Or >>> pip install pypet Or https://github.com/SmokinCaterpillar/pypet
Documentation?! https://pypet.readthedocs.io Tutorial More than 20 comprehensive examples
![Page 29: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/29.jpg)
Bonus Track
I am looking for a job!
Soon to be PhD with lack of motivation/self-delusion to stay in academia
![Page 30: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/30.jpg)
![Page 31: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/31.jpg)
Bonus Track
I am looking for a job!
Soon to be PhD with lack of motivation/self-delusion to stay in academia
Will do (Python) Coding, Data Science, and Machine Learning for food!
![Page 32: Robert Meyer- pypet](https://reader031.fdocuments.us/reader031/viewer/2022030305/587186631a28ab2c198b4ff5/html5/thumbnails/32.jpg)
Thank You!
from xkcd.com
And thanks to the BCCN Berlin,my supervisor Prof. Klaus Obermayer,
Robert Pröpper, Phillip Meier, Owen Mackwood, and Mehmet Nevvaf Timur