niprov DocumentationRelease 0.5
Jasper J.F. van den Bosch
December 20, 2016
Contents
1 Commandline Usage 3
2 Code Examples 5
3 Configuration 7
4 Provenance Fields 94.1 path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.2 hostname . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.3 user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.4 location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.5 added . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.6 version-added . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.7 size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.8 hash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.9 created . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.10 transient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.11 modality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.12 acquired . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.13 duration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.14 subject . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.15 dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.16 project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.17 protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.18 subject-position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.19 water-fat-shift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.20 transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.21 parent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.22 code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.23 logtext . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.24 script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.25 args . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.26 kwargs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.27 sampling-frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.28 fif-type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.29 lowpass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.30 highpass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.31 bad-channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
i
4.32 projection-description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.33 seriesuid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.34 filesInSeries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.35 technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.36 repetition-time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.37 field-of-view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.38 epi-factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.39 magnetization-transfer-contrast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.40 diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.41 slice-thickness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.42 slice-orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.43 echo-time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.44 flip-angle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.45 inversion-time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5 MongoDB setup 15
6 niprov package 176.1 Submodules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
6.1.1 niprov.adding module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.1.2 niprov.approval module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.1.3 niprov.basefile module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.1.4 niprov.camera module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.1.5 niprov.capturing module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.1.6 niprov.clock module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.1.7 niprov.cnt module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.1.8 niprov.commandline module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.1.9 niprov.comparing module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206.1.10 niprov.config module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206.1.11 niprov.context module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.1.12 niprov.dcm module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226.1.13 niprov.dependencies module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226.1.14 niprov.diff module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236.1.15 niprov.discovery module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.1.16 niprov.exceptions module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.1.17 niprov.exporting module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.1.18 niprov.externals module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.1.19 niprov.fif module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.1.20 niprov.filefilter module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.1.21 niprov.files module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.1.22 niprov.filesystem module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.1.23 niprov.format module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.1.24 niprov.formatdict module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.1.25 niprov.formatfactory module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.1.26 niprov.formatjson module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.1.27 niprov.formatnarrated module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.1.28 niprov.formatobject module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.1.29 niprov.formatsimple module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.1.30 niprov.formatxml module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306.1.31 niprov.hashing module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306.1.32 niprov.importing module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306.1.33 niprov.inheriting module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306.1.34 niprov.inspection module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306.1.35 niprov.jsonfile module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
ii
6.1.36 niprov.libraries module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316.1.37 niprov.location module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316.1.38 niprov.locationfactory module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326.1.39 niprov.mediumdirect module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326.1.40 niprov.mediumfactory module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326.1.41 niprov.mediumfile module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326.1.42 niprov.mediumstdout module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326.1.43 niprov.mediumviewer module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.1.44 niprov.mnefun module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.1.45 niprov.mongo module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.1.46 niprov.nifti module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346.1.47 niprov.parrec module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346.1.48 niprov.pictures module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346.1.49 niprov.pipeline module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.1.50 niprov.pipelinefactory module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.1.51 niprov.plogging module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.1.52 niprov.querying module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366.1.53 niprov.recording module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366.1.54 niprov.renaming module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376.1.55 niprov.reporting module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376.1.56 niprov.repository module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376.1.57 niprov.searching module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396.1.58 niprov.users module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396.1.59 niprov.views module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396.1.60 niprov.webapp module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.2 Module contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
7 Indices and tables 41
Python Module Index 43
iii
iv
niprov Documentation, Release 0.5
Ever encountered a file of which you weren’t sure what analysis steps it had gone through? Ever wanted to know whattypes of data you have available for a subject in one overview? Automatically document an analysis pipeline?
Provenance is meta-data that tracks the ‘history’ of a file, and niprov is a python program to create, store and publishprovenance for brain imaging files.
To get started, install niprov using pip:
pip install niprov
Keep in mind that to open image files you may need other libraries, such as mne-python, nibabel or pydicom.
The next step is to look through your directories for image files:
provenance discover /my/data/dir
After which you can start looking at what you’ve collected:
provenance report
You can do the same in python code:
import niprovniprov.discover('.')print(niprov.report()) #this will return a list of dictionaries
Read on to see other features and options.
Get in touch if you have questions or suggestions, by submitting an issue on http://github.com/ilogue/niprov or viaemail to [email protected], or at tweet to @niprovenance
Contents:
Contents 1
niprov Documentation, Release 0.5
2 Contents
CHAPTER 1
Commandline Usage
Look for image files below the current directory, inspect them and store the obtained provenance metadata.
provenance discover .
Run a transformation command and log it as provenance for the new file:
provenance record mcflirt -in t1flip_all_orig -out t1all_reg -refvol 0
Alternatively, log the provenance after running the command:
provenance log 'motion correction' --new fmri-3dmc.nii --parent fmri.nii
Publish provenance of known files for subject ‘John Doe’ as an html file.
provenance report --subject "John Doe" --html
Register a file to the provenance database without inspecting it:
provenance add 'motionvars.mat'
Rename dicom files that don’t have an extension (or niprov ignores them):
provenance rename /raw_data/dicoms/
List files marked for approval:
provenance todo
Approve a file:
provenance approve /path/to/myfile.img
3
niprov Documentation, Release 0.5
4 Chapter 1. Commandline Usage
CHAPTER 2
Code Examples
Discover and retrieve provenance:
import niprovniprov.discover('.')files = niprov.report(forSubject='John Doe')
5
niprov Documentation, Release 0.5
6 Chapter 2. Code Examples
CHAPTER 3
Configuration
Niprov can be configured through a text file with configuration directives. This file is located at ~/niprov.cfg. Alterna-tively, the settings can be changed in code:
from niprov.config import Configurationconf = Configurationconf.verbose = Trueniprov.log(x,y,z, opts=conf)
See niprov.config.Configuration for details on the individual settings.
7
niprov Documentation, Release 0.5
8 Chapter 3. Configuration
CHAPTER 4
Provenance Fields
Overview of provenance attributes collected:
Field superficial transformed PAR-REC DICOM FIF CNTpath yes yes yes yes yes yeshostname yes yes yes yes yes yesuser yes yes yes yes yes yeslocation yes yes yes yes yes yesadded yes yes yes yes yes yesversion-added yes yes yes yes yes yessize yes yes yes yes yes yeshash yes yes yes yes yes yescreated yes yes yes yes yes yestransient yes yes yes yes yes yesmodality yes inherited yes yes yes yesacquired inherited yes yes yes yesduration inherited yes yes yes yessubject inherited yes yes yes yesdimensions yes maybe yes yesproject inherited yesprotocol inherited yes yessubject-position inherited yes yeswater-fat-shift inherited yes yestransformation yesparent yescode yeslogtext yesscript yesargs yeskwargs yessampling-frequency inherited yes yesfif-type yeslowpass yeshighpass yesbad-channels yesprojection-description yesseriesuid yesfilesInSeries yes
Continued on next page
9
niprov Documentation, Release 0.5
Table 4.1 – continued from previous pageField superficial transformed PAR-REC DICOM FIF CNTtechnique inherited yesrepetition-time inherited yesfield-of-view yesepi-factor inherited yesmagnetization-transfer-contrast inherited yesdiffusion inherited yesslice-thickness yesslice-orientation yesecho-time inherited yesflip-angle inherited yesinversion-time inherited yes
4.1 path
The last known path to the file.
4.2 hostname
Hostname of the computer on which the file resides.
4.3 user
Name of the user that created the file and/or registered its provenance.
4.4 location
A string that combines computer and filesystem path.
4.5 added
Date and time that the provenance for this file was registered.
4.6 version-added
A floating point number reflecting the niprov version used to create the provenance record.
4.7 size
File size.
10 Chapter 4. Provenance Fields
niprov Documentation, Release 0.5
4.8 hash
An MD5 hash of the file’s binary contents.
4.9 created
Last known modified date of the file as reported by the OS.
4.10 transient
Whether the file is deemed temporary.
4.11 modality
Type of data; MRI, DWI, MEG, EEG or other.
4.12 acquired
When the data was collected.
4.13 duration
Duration of the acquisition in seconds.
4.14 subject
The participant whose brain was imaged.
4.15 dimensions
Dimensions of the image. Order of dimensions dependent on format; in principle follows (where applicable); in-slice,number of slices, time. E.g. x,z,y,t.
4.16 project
The name of the research project.
4.8. hash 11
niprov Documentation, Release 0.5
4.17 protocol
The name of the pulse sequence used.
4.18 subject-position
The position and orientation of the subject during during the scan. E.g. head first supine.
4.19 water-fat-shift
Water fat shift value.
4.20 transformation
The name of the transformation applied to generate this derivative image.
4.21 parent
The file that this file is a transformed version of.
4.22 code
The command used to generate this image.
4.23 logtext
The commandline output of the transformation.
4.24 script
The path to the code file containing the transformation routine.
4.25 args
The positional arguments passed to a python-based transformation command.
4.26 kwargs
The keyword arguments passed to a python-based transformation command.
12 Chapter 4. Provenance Fields
niprov Documentation, Release 0.5
4.27 sampling-frequency
How many samples were acquired per second.
4.28 fif-type
Type of MNE fiff file. One of ‘raw’, ‘ave’, ‘epo’, ‘cov’, ‘fwd’ or ‘trans’.
4.29 lowpass
Cutoff frequency of the lowpass filter applied, in Hz.
4.30 highpass
Cutoff frequency of the highpass filter applied, in Hz.
4.31 bad-channels
Names of channels/sensors marked as bad.
4.32 projection-description
Description of projection vectors in the data file.
4.33 seriesuid
A unique identifier for files in a series. Corresponds to SeriesInstanceUID in the DICOM format.
4.34 filesInSeries
A list of paths to files that are part of this series.
4.35 technique
The imaging technique that the protocol uses.
4.36 repetition-time
Time to scan one volume
4.27. sampling-frequency 13
niprov Documentation, Release 0.5
4.37 field-of-view
The extent of the observable world that is seen by the image. This is a list of three values.
4.38 epi-factor
Echo-planar-imaging factor.
4.39 magnetization-transfer-contrast
A technique that uses the transfer of nuclear spin polarization and/or spin coherence from one population of nuclei toanother population of nuclei.
4.40 diffusion
Whether this is a diffusion image. Boolean.
4.41 slice-thickness
Thickness in mm of the slices. In case the file contains multiple images, this applies to the first image.
4.42 slice-orientation
Spatial orientation of the slices. In case the file contains multiple images, this applies to the first image.
4.43 echo-time
The time in milliseconds between the application of the 90° pulse and the peak of the echo signal in Spin Echo andInversion Recovery pulse sequences.
4.44 flip-angle
The angle to which the net magnetization is rotated or tipped relative to the main magnetic field direction via theapplication of an RF excitation pulse.
4.45 inversion-time
The time period between the 180° inversion pulse and the 90° excitation pulse in an Inversion Recovery pulse sequence.
14 Chapter 4. Provenance Fields
CHAPTER 5
MongoDB setup
This quickstart guide explains how to setup a MongoDB backend for niprov on Ubuntu.
...incomplete..
Requires 2.6
Follow installation instructions at
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
Configure mongodb:
sudo nano /etc/mongod.conf
# network interfacesnet:
port: 27017bindIp: 0.0.0.0
sudo service mongod restart
Allow access to mongodb from IP addresses in the range 123.45.67.0/24:
sudo ufw allow from 123.45.67.0/24 to any port 27017
15
niprov Documentation, Release 0.5
16 Chapter 5. MongoDB setup
CHAPTER 6
niprov package
6.1 Submodules
6.1.1 niprov.adding module
niprov.adding.add(filepath, transient=False, provenance=None, dependen-cies=<niprov.dependencies.Dependencies object>)
Simply register the file.
Inspects the file and makes it known to the provenance data, such that image files can be logged that have beencreated using this file. Useful also for temporary files.
Example
(provenance, status) = niprov.add(‘/path/to/my.nii’)
Parameters
• filepath (str) – Path to the newly created file.
• transient (bool, optional) – Set this to True to indicate that the file is only tem-porary and future checks should not expect it to be physically present. Defaults to False,assuming that the file remains.
• provenance (dict, optional) – Add the key-value pairs in this dictionary to theprovenance record for the new file.
Returns Tuple of new provenance and status. Status is a string with one of the following values:‘new’: File was not known yet and has been added. ‘series’: The file was deemed part of aseries and has been added. ‘failed’: There was an error inspecting the file. ‘known’: The filewas already known to niprov, nothing happened. ‘dryrun’: Function called with config.dryrun,database not touched.
Return type tuple
6.1.2 niprov.approval module
niprov.approval.approve(filepath, dependencies=<niprov.dependencies.Dependencies object>)Mark this file as approved.
Parameters filepath (str) – Path to the tracked file that has been found valid.
17
niprov Documentation, Release 0.5
niprov.approval.markForApproval(files, reset=False, dependen-cies=<niprov.dependencies.Dependencies object>)
Mark a list of files for approval by a human.
Parameters
• files (list) – List of paths of files tracked by niprov to mark for approval.
• reset (bool) – Also mark files that have already been approved. False by default.
niprov.approval.markedForApproval(dependencies=<niprov.dependencies.Dependencies ob-ject>)
List files marked for approval by a human.
niprov.approval.selectApproved(files, dependencies=<niprov.dependencies.Dependencies ob-ject>)
Return only files that have approval status ‘granted’.
Parameters files (list) – List of paths of files to check for approval status.
6.1.3 niprov.basefile module
class niprov.basefile.BaseFile(location, provenance=None, dependen-cies=<niprov.dependencies.Dependencies object>)
Bases: object
attach(form=’json’)Not implemented for BaseFile parent class.
Parameters form (str) – Data format in which to serialize provenance. Defaults to ‘json’.
compare(other)
getProtocolFields()
getProvenance(form=’dict’)
getSeriesId()
getSnapshotFilepath()
inspect()
keepVersionsFromPrevious(previous)
parents
versions
viewSnapshot()
6.1.4 niprov.camera module
class niprov.camera.Camera(dependencies)Bases: object
saveSnapshot(data, for_)Plot an overview of the image and store it.
Uses PictureCache as service that provides a file-like handle to save the plotted picture to.
Calls takeSnapshot() to do the actual plotting.
Parameters data (numpy.ndarray) – Array of 2, 3 or 4 dimensions with image data.
18 Chapter 6. niprov package
niprov Documentation, Release 0.5
takeSnapshot(data, on)Plot an overview of the image using matplotlib.pyplot.
Parameters
• data (numpy.ndarray) – Array of 2, 3 or 4 dimensions with image data.
• on (str or file-like object) – Where to save figure to.
6.1.5 niprov.capturing module
class niprov.capturing.OutputCaptureBases: object
Context manager which records stdout writes.
Used when recording python-based transformations.
6.1.6 niprov.clock module
class niprov.clock.ClockBases: object
getNowString()
6.1.7 niprov.cnt module
class niprov.cnt.NeuroscanFile(location, **kwargs)Bases: niprov.basefile.BaseFile
Support for the Neuroscan .cnt format.
inspect()
6.1.8 niprov.commandline module
class niprov.commandline.Commandline(dependencies=<niprov.dependencies.Dependencies ob-ject>)
Bases: object
addUnknownParent(fpath)
discoveryFinished(nnew, nadded, nfailed, ntotal)
exportedToFile(fname)
fileAdded(image)
fileError(fpath)
filesMarkedForApproval(images)
interpretedRecording(new, transform, parents)
log(level, message, exceptionClass=None)
missingDependencyForImage(lib, fpath)
mnefunEventReceived(operationName)
6.1. Submodules 19
niprov Documentation, Release 0.5
receivedBashCommand(command)
renamedDicom(fpath)
usingCopyAsParent(copy)
vlevels = [’debug’, ‘info’, ‘warning’, ‘error’]
niprov.commandline.ordinal(num)
6.1.9 niprov.comparing module
niprov.comparing.compare(file1, file2, dependencies=None)Compare the provenance of two files.
This creates a niprov Diff object that can be further interrogated or displayed for differences between the twofiles.
Example
Chain calls with methods of Diff or print the Diff object:
compare(file1, file2).assertEqual(ignore='path')print(compare(file1, file2))
Parameters
• file1 (BaseFile) – One of two niprov BaseFile objects to compare.
• file2 (BaseFile) – As file1
Returns A niprov Diff object which reflects differences between the two files.
Return type niprov.diff.Diff
6.1.10 niprov.config module
class niprov.config.Configuration(configFilePath=’~/niprov.cfg’)Bases: object
General settings for niprov.
Individual settings are documented as follows;
setting = default_value type - Explanation.
The settings can be changed in the configuration file, or in code.
All settings:
attach = Falsebool: Attach provenance to image files. For nifti files for instance, this means inserting a header extensionwith serialized provenance. See ‘attach_format’ to configure which data format is used.
attach_format = ‘json’string: Format in which to attach provenance to the file. One of ‘json’, or ‘xml’. For example, if set to‘json’ and the ‘attach’ option is True, this will add a header extension to nifti files created with the relevantprovenance data in json format.
20 Chapter 6. niprov package
niprov Documentation, Release 0.5
database_type = ‘file’str: Type of backend in which to store provenance. Currently only ‘file’ or ‘MongoDB’
database_url = ‘~/provenance.json’str: URL of the database. If database-type is file, this is the path to the file.
discover_file_extensions = [’.PAR’, ‘.dcm’, ‘.fif’, ‘.cnt’]list: Discover uses this to determine which files to include. Not strictly extensions, can be any string thatappears in the file name. Use comma’s to separate items.
dryrun = Falsebool: Do not execute commands or make lasting changes to the provenance database.
user = ‘’string: Name of the user creating provenance. If not provided, will be determined based on OS informationor as passed as an argument to the provenance operation. See also niprov.users
verbosity = ‘info’string: Level of information to report. One of ‘debug’,’info’,’warning’, ‘error’. Any level includes higherlevels, i.e. ‘info’ will log messages of that are deemed ‘info’, ‘warning’ or ‘error’.
6.1.11 niprov.context module
class niprov.context.ProvenanceContextBases: object
add(filepath, transient=False, provenance=None)See niprov.adding
approve(filepath)See niprov.approval
backup()See niprov.exporting
compare(file1, file2)See niprov.comparing
discover(root)See niprov.discovery
export(images, medium, form, pipeline=False)See niprov.exporting
get()See niprov.querying
importp(fpath)See niprov.importing
inspect(location)See niprov.inspection
log(new, transformation, parents, code=None, logtext=None, transient=False, script=None,user=None, provenance=None, opts=None)See niprov.plogging
markForApproval(files)See niprov.approval
markedForApproval()See niprov.approval
6.1. Submodules 21
niprov Documentation, Release 0.5
print_(images, pipeline=False)See niprov.exporting
record(command, new=None, parents=None, transient=False, args=None, kwargs=None, user=None,opts=None)
See niprov.recording
renameDicoms(dicomdir)See niprov.renaming
search(text)See niprov.searching
selectApproved(files)See niprov.approval
view(images, pipeline=False)See niprov.exporting
6.1.12 niprov.dcm module
class niprov.dcm.DicomFile(location, **kwargs)Bases: niprov.basefile.BaseFile
getSeriesId()Return the DICOM “SeriesInstanceUID” that all files in this series have in common.
Returns A string uniquely identifying files belonging to this series.
Return type str
hasFile(other)
inspect()Inspect the DICOM file attributes.
If a general AcquisitionDateTime attribute is not present, the SeriesDate and SeriesTime will be used toset the acquired provenance field.
Returns Provenance for the inspected file.
Return type dict
mergeWith(img)Add a single DICOM file object to this series.
The file will be stored in provenance in the ‘filesInSeries’ list.
6.1.13 niprov.dependencies module
class niprov.dependencies.Dependencies(config=None)Bases: object
getCamera()
getClock()
getConfiguration()
getExternals()
getFileFactory()
22 Chapter 6. niprov package
niprov Documentation, Release 0.5
getFileFilter()
getFilesystem()
getFormatFactory()
getHasher()
getLibraries()
getListener()
getLocationFactory()
getMediumFactory()
getPictureCache()
getPipelineFactory()
getQuery()
getRepository()
getSerializer()
getUsers()
reconfigureOrGetConfiguration(newConfiguration)
6.1.14 niprov.diff module
class niprov.diff.Diff(file1, file2)Bases: object
Difference between two files.
This represents differences in provenance between two files.
See niprov.comparing
Parameters
• file1 (BaseFile) – One of two niprov BaseFile objects to compare.
• file2 (BaseFile) – As file1
NCHARSCOL = 20
areEqual(ignore=None, select=None)Whether there are any differences between the files.
Parameters
• ignore (list) – Optional. List of fields not to evaluate when determining differences.
• select (list) – Optional. List of fields that should be specifically evaluated. All otherfields will be ignored.
Returns True if no differences, False otherwise.
Return type bool
areEqualProtocol()Whether there are any differences for protocol fields.
Each BaseFile subtype has a getProtocolFields() method that is used here to selectively see if any ofthese are different.
6.1. Submodules 23
niprov Documentation, Release 0.5
Returns True if no differences, False otherwise.
Return type bool
assertEqual(ignore=None, select=None)Raises exception if there are differences.
Parameters
• ignore (list) – Optional. List of fields not to evaluate when determining differences.
• select (list) – Optional. List of fields that should be specifically evaluated. All otherfields will be ignored.
Raises AssertionError – Message with differences in a table.
assertEqualProtocol()Raises exception if there are differences in protocol fields.
Each BaseFile subtype has a getProtocolFields() method that is used here to selectively see if any ofthese are different.
Raises AssertionError – Message with protocol differences in a table.
defaultIgnore = [’_id’]
getDifferenceString(ignore=None, select=None)Get table of differences as string.
Parameters
• ignore (list) – Optional. List of fields not to evaluate when determining differences.
• select (list) – Optional. List of fields that should be specifically evaluated. All otherfields will be ignored.
Returns A three-columns table listing provenance fields and their respective values for the twofiles.
Return type str
getDifferences(ignore=None, select=None)Get dictionary with fields that differ and how they differ.
Parameters
• ignore (list) – Optional. List of fields not to evaluate when determining differences.
• select (list) – Optional. List of fields that should be specifically evaluated. All otherfields will be ignored.
Returns A dictionary with provenance fields as keys and strings indicating how they differ.
Return type dict
getSame()Get dictionary with fields that have equal values.
Returns A dictionary with provenance fields as keys the string ‘same’ as value.
Return type dict
getSameString()Get table of values that are the same for the compared files.
Returns A three-columns table listing provenance fields and their respective values for the twofiles.
24 Chapter 6. niprov package
niprov Documentation, Release 0.5
Return type str
6.1.15 niprov.discovery module
niprov.discovery.discover(root, dependencies=<niprov.dependencies.Dependencies object>)Search a directory for image files, and add them to your provenance collection.
Files are only included if they match the filters in the ‘discover_file_extensions’ settings. Refer to niprov.addfor details on what happens to individual files.
Parameters root (str) – The top directory in which to look for new files.
6.1.16 niprov.exceptions module
exception niprov.exceptions.NiprovErrorBases: exceptions.Exception
exception niprov.exceptions.UnknownFileErrorBases: niprov.exceptions.NiprovError
6.1.17 niprov.exporting module
niprov.exporting.backup(dependencies=<niprov.dependencies.Dependencies object>)Shortcut for export(medium=’file’, form=’json’) for all provenance.
niprov.exporting.export(provenance, medium, form, pipeline=False, dependen-cies=<niprov.dependencies.Dependencies object>)
Publish or simply return provenance for selected files.
To get provenance on one specific file, pass its path as the ‘forFile’ argument. Alternatively, to get all filesassociated with a certain subject, use the ‘forSubject’ argument. If none of these is used, provenance for themost recently registered files is reported.
Parameters
• provenance – Niprov BaseFile object or list of such.
• medium (str) – The medium in which to publish the provenance. One of:
‘stdout’ (print the provenance to the terminal), ‘direct’ (return object to caller), ‘file’(write to a text file), ‘viewer’ (open in the system image viewer).
• form (str) – The format in which to serialize the provenance. One of‘json’,’xml’,’narrated’,’simple’,’dict’,’picture’.
Returns Depends on medium selected.
niprov.exporting.print_(provenance, pipeline=False, dependen-cies=<niprov.dependencies.Dependencies object>)
Shortcut for export(medium=’stdout’, form=’simple’).
niprov.exporting.view(provenance, pipeline=False, dependencies=<niprov.dependencies.Dependenciesobject>)
Shortcut for export(medium=’viewer’, form=’picture’).
6.1. Submodules 25
niprov Documentation, Release 0.5
6.1.18 niprov.externals module
class niprov.externals.ExternalsBases: object
Utility that wraps the python subprocess module to start other applications
run(command)Start a subprocess with the command provided.
Parameters command (list) – A list of command elements.
class niprov.externals.Result(succesful, output)Bases: object
6.1.19 niprov.fif module
class niprov.fif.FifFile(location, **kwargs)Bases: niprov.basefile.BaseFile
attach(form=’json’)Attach the current provenance to the file by appending it as a json-encoded string to the ‘description’header field.
This is only attempted if the file has been inspect()-ed and has been determined to be a raw fif file.
Parameters form (str) – Data format in which to serialize provenance. Defaults to ‘json’.
inspect()
6.1.20 niprov.filefilter module
class niprov.filefilter.FileFilter(dependencies=<niprov.dependencies.Dependencies object>)Bases: object
include(filepath)Whether the file is to be included in discovery.
Parameters filepath (str) – The full path of the file.
Returns True if the file should be included.
Return type bool
6.1.21 niprov.files module
6.1.22 niprov.filesystem module
class niprov.filesystem.FilesystemBases: object
Wrapper of filesystem access functionality such as that implemented by the os package in the standard library.
fileExists(path)
getctime(path)Get the creation time for the file at path.
Parameters path – Path to the file to read.
26 Chapter 6. niprov package
niprov Documentation, Release 0.5
Returns Time when the file was last changed
Return type datetime
getsize(path)
read(path)Read the contents of a textfile.
Parameters path – Path to the file to read.
Returns Contents of the file
Return type str
Raises IOError – [Errno 2] No such file or directory: ‘xyz’
readlines(path)
walk(path)
write(path, content)Write string content to a textfile.
Parameters
• path – Path to the file to read.
• content (str) – What to fill the file with
6.1.23 niprov.format module
class niprov.format.Format(dependencies=<niprov.dependencies.Dependencies object>)Bases: object
Parent Format class from which specific formats are derived.
serialize(provenance)Publish provenance.
This determines if the provenance is for a single file or multiple, and then calls the appropriate morespecific serialize method.
serializeList(provenance)
serializePipeline(pipeline)
serializeSingle(provenance)
serializeStatistics(provenance)
6.1.24 niprov.formatdict module
class niprov.formatdict.DictFormat(dependencies=<niprov.dependencies.Dependencies object>)Bases: niprov.format.Format
serializeList(fileList)
serializeSingle(item)
6.1. Submodules 27
niprov Documentation, Release 0.5
6.1.25 niprov.formatfactory module
class niprov.formatfactory.FormatFactory(dependencies=<niprov.dependencies.Dependenciesobject>)
Bases: object
create(formatName)
6.1.26 niprov.formatjson module
class niprov.formatjson.DateTimeAwareJSONDecoder(**kwargs)Bases: json.decoder.JSONDecoder
Converts a json string, where datetime and timedelta objects were converted into objects using the Date-TimeAwareJSONEncoder, back into a python object.
dict_to_object(d)
class niprov.formatjson.DateTimeAwareJSONEncoder(skipkeys=False, ensure_ascii=True,check_circular=True, allow_nan=True,sort_keys=False, indent=None, sep-arators=None, encoding=’utf-8’,default=None)
Bases: json.encoder.JSONEncoder
Converts a python object, where datetime and timedelta objects are converted into objects that can be decodedusing the DateTimeAwareJSONDecoder.
default(obj)
class niprov.formatjson.JsonFormat(dependencies)Bases: niprov.format.Format
Helper to convert provenance data to and from json encoded strings.
deserialize(jsonRecord)Convert one provenance item from its json string version to the native python dictionary format.
Parameters jsonRecord (str) – The provenance item to convert as json-encoded string.
Returns Python dictionary of the provenance.
Return type dict
deserializeList(jsonListOfRecords)Convert a list of provenance items from its json string version to the a list of the native python dictionaryformat.
Parameters jsonListOfRecords (str) – The provenance items to convert as json-encodedstring.
Returns Python list of dictionaries of the provenance.
Return type list
serializeList(listOfRecords)Convert a list of provenance items from its native list of python dict type to a json string.
Parameters listOfRecords (list) – The provenance items to convert.
Returns Json version of the provenance items.
Return type str
28 Chapter 6. niprov package
niprov Documentation, Release 0.5
serializeSingle(record)Convert one provenance item from its native python dict type to a json string.
Parameters record (dict) – The provenance item to convert.
Returns Json version of the provenance.
Return type str
6.1.27 niprov.formatnarrated module
class niprov.formatnarrated.NarratedFormat(dependencies=<niprov.dependencies.Dependenciesobject>)
Bases: niprov.format.Format
Generates human-readable text that describes provenance objects.
serializeSingle(img)
6.1.28 niprov.formatobject module
class niprov.formatobject.ObjectFormat(dependencies=<niprov.dependencies.Dependencies ob-ject>)
Bases: niprov.format.Format
serializeList(provenance)
serializePipeline(provenance)
serializeSingle(provenance)
serializeStatistics(provenance)
6.1.29 niprov.formatsimple module
class niprov.formatsimple.SimpleFormat(dependencies=<niprov.dependencies.Dependencies ob-ject>)
Bases: niprov.format.Format
serializeList(images)Publish the provenance for several images on the commandline.
Parameters provenance (list) – List of provenance dictionaries.
serializePipeline(pipeline)Pretty-print a pipeline to the terminal
Parameters pipeline (Pipeline) – Pipeline object to publish.
serializeSingle(img)Publish the provenance for one image on the commandline.
Parameters provenance (dict) – Provenance for one image file
serializeStatistics(stats)Publish statistics for collected provenance in the terminal.
Parameters stats (dict) – Dictionary with summary values.
serializeSummary(image)Publish a summary of the provenance for one image as one line in the terminal.
6.1. Submodules 29
niprov Documentation, Release 0.5
Parameters provenance (dict) – Provenance for one image file
6.1.30 niprov.formatxml module
class niprov.formatxml.XmlFormat(dependencies)Bases: niprov.format.Format
serializeList(itemOrList)
serializeSingle(item)
6.1.31 niprov.hashing module
class niprov.hashing.HasherBases: object
blocksize = 524288
digest(filename)Determine the unique hash digest for a file.
Note: takes 26s for a 14GB file.
Parameters filename (str) – Path to the file for which a hash digest should be made.
6.1.32 niprov.importing module
niprov.importing.importp(filepath, dependencies=<niprov.dependencies.Dependencies object>)Add provenance in bulk from a file, such as saved by export().
Named importp as opposed to import because the latter is a reserved word in Python.
This can serve as a backup, migration tool, or for exchange.
6.1.33 niprov.inheriting module
niprov.inheriting.inheritFrom(provenance, parentProvenance)
6.1.34 niprov.inspection module
niprov.inspection.inspect(location, dependencies=<niprov.dependencies.Dependencies object>)
6.1.35 niprov.jsonfile module
class niprov.jsonfile.JsonFile(dependencies=<niprov.dependencies.Dependencies object>)Bases: object
Stores provenance in a local text file encoded as json.
add(image)Add the provenance for one file to storage.
Parameters image (BaseFile) – Image file to store.
30 Chapter 6. niprov package
niprov Documentation, Release 0.5
all()Retrieve all known provenance from storage.
Returns List of provenance for known files.
Return type list
byId(uid)
byLocation(locationString)Get the provenance for a file at the given location.
In the case of a dicom series, this returns the provenance for the series.
Parameters locationString (str) – Location of the image file.
Returns Provenance for one image file.
Return type dict
byLocations(listOfLocations)
byParents(listOfParentLocations)
getSeries(image)Get the object that carries provenance for the series that the image passed is in.
Parameters image (DicomFile) – File that is part of a series.
Returns Image object that caries provenance for the series.
Return type DicomFile
inquire(query)
latest(n=20)
search(text)
serializeAndWrite(images)
statistics()
update(image)Save changed provenance for this file..
Parameters image (BaseFile) – Image file that has changed.
updateApproval(fpath, approvalStatus)
6.1.36 niprov.libraries module
class niprov.libraries.LibrariesBases: object
hasDependency(libname)
6.1.37 niprov.location module
class niprov.location.Location(locationString)Bases: object
Represents the location of a file.
toDictionary()
6.1. Submodules 31
niprov Documentation, Release 0.5
toString()
toUrl()
6.1.38 niprov.locationfactory module
class niprov.locationfactory.LocationFactory(dependencies=<niprov.dependencies.Dependenciesobject>)
Bases: object
Creates Location objects.
completeString(locationString)Validates a location string.
If locationString only contains a filesystem path, computer info will be added.
fromString(locationString)Creates a Location object.
Parameters locationString (str) – String with path and optionally computer id.
6.1.39 niprov.mediumdirect module
class niprov.mediumdirect.DirectMediumBases: object
Dummy Medium class which simply returns the serialized provenance passed.
export(formattedProvenance, form)
6.1.40 niprov.mediumfactory module
class niprov.mediumfactory.MediumFactory(dependencies=<niprov.dependencies.Dependenciesobject>)
Bases: object
create(mediumName)
6.1.41 niprov.mediumfile module
class niprov.mediumfile.FileMedium(dependencies)Bases: object
export(formattedProvenance, form)
6.1.42 niprov.mediumstdout module
class niprov.mediumstdout.StandardOutputMediumBases: object
export(formattedProvenance, form)
32 Chapter 6. niprov package
niprov Documentation, Release 0.5
6.1.43 niprov.mediumviewer module
class niprov.mediumviewer.ViewerMedium(dependencies)Bases: object
Uses the system picture viewer to display an picture file.
export(formattedProvenance, form=None)
6.1.44 niprov.mnefun module
mnefun support module
This module provides handlers to attach to mnefun events.
import mnefunimport niprov.mnefunsupportparams = mnefun.Params()params.on_process = niprov.mnefunsupport.handler
niprov.mnefunsupport.handler(text, func, out, params, dependen-cies=<niprov.dependencies.Dependencies object>)
mnefun on_process handler
Responds to the following mnefun steps: fetch_raw_files: Runs discover on the subjects’ raw data dirs.fetch_sss_files, apply_preprocessing_combined, save_epochs: For these steps, runs log for the new files.
6.1.45 niprov.mongo module
class niprov.mongo.MongoRepository(dependencies=<niprov.dependencies.Dependencies object>)Bases: object
add(image)Add the provenance for one file to storage.
Parameters image (BaseFile) – Image file to store.
all()Retrieve all known provenance from storage.
Returns List of provenance for known files.
Return type list
byId(uid)
byLocation(locationString)Get the provenance for a file at the given location.
In the case of a dicom series, this returns the provenance for the series.
Parameters locationString (str) – Location of the image file.
Returns Provenance for one image file.
Return type dict
byLocations(listOfLocations)
byParents(listOfParentLocations)
deflate(img)
6.1. Submodules 33
niprov Documentation, Release 0.5
getSeries(image)Get the object that carries provenance for the series that the image passed is in.
Parameters image (DicomFile) – File that is part of a series.
Returns Image object that caries provenance for the series.
Return type DicomFile
inflate(record)
inquire(query)
latest()
search(text)
statistics()
update(image)Save changed provenance for this file..
Parameters image (BaseFile) – Image file that has changed.
updateApproval(locationString, approvalStatus)
6.1.46 niprov.nifti module
class niprov.nifti.NiftiFile(location, **kwargs)Bases: niprov.basefile.BaseFile
attach(form=’json’)Attach the current provenance to the file by injecting it as a json-encoded extension to the nifti header.
Parameters form (str) – Data format in which to serialize provenance. Defaults to ‘json’.
inspect()
6.1.47 niprov.parrec module
6.1.48 niprov.pictures module
class niprov.pictures.PictureCache(dependencies)Bases: niprov.format.Format
getBytes(for_)
getFilepath(for_)
keep(picture, for_)
new()
saveToDisk(for_)
serializeSingle(image)Provides file path to picture of image.
This is part of the Format interface.
34 Chapter 6. niprov package
niprov Documentation, Release 0.5
6.1.49 niprov.pipeline module
class niprov.pipeline.Pipeline(files)Bases: object
asFilenameTree()
6.1.50 niprov.pipelinefactory module
class niprov.pipelinefactory.PipelineFactory(dependencies=<niprov.dependencies.Dependenciesobject>)
Bases: object
forFile(image)Create a Pipeline object based on known files ‘parents’ field.
6.1.51 niprov.plogging module
niprov.plogging.log(new, transformation, parents, code=None, logtext=None, transient=False,script=None, user=None, provenance=None, opts=None, dependen-cies=<niprov.dependencies.Dependencies object>)
Register a transformation that creates a new image (or several).
This will retrieve the primary parent’s provenance. if no provenance is availale for the primary parent, callslistener.unknownFile. Otherwise, some fields are copied from the primary parent, subject to availability. Forinstance, if the parent has no ‘subject’ field, the new file’s provenance won’t either.
Parameters
• new (str or list) – Path(s) to the newly created file(s).
• transformation (str) – Name of the operation that has been used.
• parents (str or list) – Path(s) to the file(s) that were used as the basis of the trans-formation. Assumes that the first file in the list is the primary parent for which basic prove-nance is known.
• code (str, optional) – Code that was used to generate the new file
• logtext (str, optional) – Any information about the transformation that waslogged.
• script (str, optional) – Path to the code file that contains the transformation code.
• transient (bool, optional) – Set this to True to indicate that the file is only tem-porary and future checks should not expect it to be physically present. Defaults to False,assuming that the file remains.
• user (string, optional) – Name of the user logging provenance.
• provenance (dict, optional) – Add the key-value pairs in this dictionary to theprovenance record for the new files.
• opts (Configuration) – General settings for niprov. See niprov.config
Raises IOError – ‘[Errno 2] File not found’ is raised if the new file does not exist on the filesystemand is not marked as transient.
Returns New provenance, if multiple files were created, this is a list of images, otherwise, it is asingle object.
6.1. Submodules 35
niprov Documentation, Release 0.5
Return type BaseFile
6.1.52 niprov.querying module
class niprov.querying.Query(dependencies)Bases: object
all()
allModalities()
allProjects()
allUsers()
byApproval(val)
byLocation(val)
byModality(val)
byProject(val)
bySubject(val)
byUser(val)
copiesOf(target)
getFields()
latest()
statistics()
class niprov.querying.QueryField(name, value, all)Bases: tuple
__getnewargs__()Return self as a plain tuple. Used by copy and pickle.
__getstate__()Exclude the OrderedDict from pickling
__repr__()Return a nicely formatted representation string
allAlias for field number 2
nameAlias for field number 0
valueAlias for field number 1
6.1.53 niprov.recording module
niprov.recording.record(command, new=None, parents=None, transient=False,args=None, kwargs=None, user=None, opts=None, dependen-cies=<niprov.dependencies.Dependencies object>)
Execute a command and log it as provenance for the newly created file.
36 Chapter 6. niprov package
niprov Documentation, Release 0.5
Parameters
• command (list or str or callable) – Command to be executed. Either a stringof executable system code, a list of components thereof, or a python function object.
• new (list or str, optional) – Override path to the new file(s), i.e. if they cannotbe parsed from the command.
• parents (list or str, optional) – Override paths to parent file(s), i.e. if theycannot be parsed from the command.
• transient (bool, optional) – Set this to True to indicate that the file is only tem-porary and future checks should not expect it to be physically present. Defaults to False,assuming that the file remains.
• args (list, optional) – Positional arguments to be passed to command.
• kwargs (dict, optional) – Keyword arguments to be passed to command.
• opts (Configuration) – General settings for niprov. See niprov.config
Returns New provenance
Return type dict
6.1.54 niprov.renaming module
niprov.renaming.renameDicoms(dicomdir, dependencies=<niprov.dependencies.Dependencies ob-ject>)
Add the .dcm extension to any non-hidden files without extension.
Parameters dicomdir (str) – Directory in which to rename files.
6.1.55 niprov.reporting module
6.1.56 niprov.repository module
class niprov.repository.RepositoryBases: object
add(image)Add the provenance for one file to storage.
Parameters image (BaseFile) – Image file to store.
all()Retrieve all known provenance from storage.
Returns List of provenance for known files.
Return type list
byApproval(approvalStatus)
byId(uid)Get the provenance for a file with the given id.
Parameters uid (str) – Unique id for the file.
Returns File with the given id.
Return type BaseFile
6.1. Submodules 37
niprov Documentation, Release 0.5
byLocation(locationString)Get the provenance for a file at the given location.
In the case of a dicom series, this returns the provenance for the series.
Parameters locationString (str) – Location of the image file.
Returns Provenance for one image file.
Return type dict
byLocations(listOfLocations)Get any files that match one of these locations
In the case of a dicom series, this returns the provenance for the series.
Parameters listOfLocations (list) – List of image locations.
Returns List with BaseFile objects
Return type list
bySubject(subject)Get the provenance for all files of a given participant.
Parameters subject (str) – The name or other ID string.
Returns List of provenance for known files imaging this subject.
Return type list
getSeries(image)Get the object that carries provenance for the series that the image passed is in.
Parameters image (DicomFile) – File that is part of a series.
Returns Image object that caries provenance for the series.
Return type DicomFile
knows(image)Whether this file has provenance associated with it.
Returns True if provenance is available for this image.
Return type bool
knowsByLocation(locationString)Whether the file at this location has provenance associated with it.
Returns True if provenance is available for that path.
Return type bool
knowsSeries(image)Whether this file is part of a series for which provenance is available.
Parameters image (BaseFile) – File for which the series is sought.
Returns True if provenance is available for this series.
Return type bool
latest(n=20)Get the images that have been registered last.
Parameters n (int) – The number of files to retrieve. Defaults to 20.
Returns List of BaseFile objects.
38 Chapter 6. niprov package
niprov Documentation, Release 0.5
Return type list
update(image)Save changed provenance for this file..
Parameters image (BaseFile) – Image file that has changed.
updateApproval(locationString, approvalStatus)
6.1.57 niprov.searching module
niprov.searching.search(text, dependencies=<niprov.dependencies.Dependencies object>)Search for files with the given text in their provenance fields.
Parameters text (str) – Words to look for.
Returns List of BaseFile objects
Return type list
6.1.58 niprov.users module
class niprov.users.Users(dependencies)Bases: object
determineUser(passedUserValue)Determine the name of the user creating provenance.
The following methods will be tried to determine the username; 1) The passedUserValue argument if notNone 2) The configuration file or setting has a value for the key user 3) The OS username.
Parameters passedUserValue (str) – This value will override any other method to deter-mine the username.
Returns The username or handle of the user.
Return type string
6.1.59 niprov.views module
niprov.views.home(request)
niprov.views.latest(request)
niprov.views.location(request)
niprov.views.modalities(request)
niprov.views.modality(request)
niprov.views.pipeline(request)
niprov.views.project(request)
niprov.views.projects(request)
niprov.views.search(request)
niprov.views.short(request)
niprov.views.stats(request)
6.1. Submodules 39
niprov Documentation, Release 0.5
niprov.views.subject(request)
niprov.views.user(request)
niprov.views.users(request)
6.1.60 niprov.webapp module
niprov.webapp.main(global_config, **settings)This function returns a Pyramid WSGI application.
niprov.webapp.serve()
6.2 Module contents
40 Chapter 6. niprov package
CHAPTER 7
Indices and tables
• genindex
• modindex
• search
41
niprov Documentation, Release 0.5
42 Chapter 7. Indices and tables
Python Module Index
nniprov, 40niprov.adding, 17niprov.approval, 17niprov.basefile, 18niprov.camera, 18niprov.capturing, 19niprov.clock, 19niprov.cnt, 19niprov.commandline, 19niprov.comparing, 20niprov.config, 20niprov.context, 21niprov.dcm, 22niprov.dependencies, 22niprov.diff, 23niprov.discovery, 25niprov.exceptions, 25niprov.exporting, 25niprov.externals, 26niprov.fif, 26niprov.filefilter, 26niprov.filesystem, 26niprov.format, 27niprov.formatdict, 27niprov.formatfactory, 28niprov.formatjson, 28niprov.formatnarrated, 29niprov.formatobject, 29niprov.formatsimple, 29niprov.formatxml, 30niprov.hashing, 30niprov.importing, 30niprov.inheriting, 30niprov.inspection, 30niprov.jsonfile, 30niprov.libraries, 31niprov.location, 31niprov.locationfactory, 32niprov.mediumdirect, 32
niprov.mediumfactory, 32niprov.mediumfile, 32niprov.mediumstdout, 32niprov.mediumviewer, 33niprov.mnefunsupport, 33niprov.mongo, 33niprov.nifti, 34niprov.pictures, 34niprov.pipeline, 35niprov.pipelinefactory, 35niprov.plogging, 35niprov.querying, 36niprov.recording, 36niprov.renaming, 37niprov.repository, 37niprov.searching, 39niprov.users, 39niprov.views, 39niprov.webapp, 40
43
niprov Documentation, Release 0.5
44 Python Module Index
Index
Symbols__getnewargs__() (niprov.querying.QueryField method),
36__getstate__() (niprov.querying.QueryField method), 36__repr__() (niprov.querying.QueryField method), 36
Aadd() (in module niprov.adding), 17add() (niprov.context.ProvenanceContext method), 21add() (niprov.jsonfile.JsonFile method), 30add() (niprov.mongo.MongoRepository method), 33add() (niprov.repository.Repository method), 37addUnknownParent() (niprov.commandline.Commandline
method), 19all (niprov.querying.QueryField attribute), 36all() (niprov.jsonfile.JsonFile method), 30all() (niprov.mongo.MongoRepository method), 33all() (niprov.querying.Query method), 36all() (niprov.repository.Repository method), 37allModalities() (niprov.querying.Query method), 36allProjects() (niprov.querying.Query method), 36allUsers() (niprov.querying.Query method), 36approve() (in module niprov.approval), 17approve() (niprov.context.ProvenanceContext method),
21areEqual() (niprov.diff.Diff method), 23areEqualProtocol() (niprov.diff.Diff method), 23asFilenameTree() (niprov.pipeline.Pipeline method), 35assertEqual() (niprov.diff.Diff method), 24assertEqualProtocol() (niprov.diff.Diff method), 24attach (niprov.config.Configuration attribute), 20attach() (niprov.basefile.BaseFile method), 18attach() (niprov.fif.FifFile method), 26attach() (niprov.nifti.NiftiFile method), 34attach_format (niprov.config.Configuration attribute), 20
Bbackup() (in module niprov.exporting), 25backup() (niprov.context.ProvenanceContext method), 21BaseFile (class in niprov.basefile), 18
blocksize (niprov.hashing.Hasher attribute), 30byApproval() (niprov.querying.Query method), 36byApproval() (niprov.repository.Repository method), 37byId() (niprov.jsonfile.JsonFile method), 31byId() (niprov.mongo.MongoRepository method), 33byId() (niprov.repository.Repository method), 37byLocation() (niprov.jsonfile.JsonFile method), 31byLocation() (niprov.mongo.MongoRepository method),
33byLocation() (niprov.querying.Query method), 36byLocation() (niprov.repository.Repository method), 37byLocations() (niprov.jsonfile.JsonFile method), 31byLocations() (niprov.mongo.MongoRepository
method), 33byLocations() (niprov.repository.Repository method), 38byModality() (niprov.querying.Query method), 36byParents() (niprov.jsonfile.JsonFile method), 31byParents() (niprov.mongo.MongoRepository method),
33byProject() (niprov.querying.Query method), 36bySubject() (niprov.querying.Query method), 36bySubject() (niprov.repository.Repository method), 38byUser() (niprov.querying.Query method), 36
CCamera (class in niprov.camera), 18Clock (class in niprov.clock), 19Commandline (class in niprov.commandline), 19compare() (in module niprov.comparing), 20compare() (niprov.basefile.BaseFile method), 18compare() (niprov.context.ProvenanceContext method),
21completeString() (niprov.locationfactory.LocationFactory
method), 32Configuration (class in niprov.config), 20copiesOf() (niprov.querying.Query method), 36create() (niprov.formatfactory.FormatFactory method), 28create() (niprov.mediumfactory.MediumFactory method),
32
45
niprov Documentation, Release 0.5
Ddatabase_type (niprov.config.Configuration attribute), 20database_url (niprov.config.Configuration attribute), 21DateTimeAwareJSONDecoder (class in
niprov.formatjson), 28DateTimeAwareJSONEncoder (class in
niprov.formatjson), 28default() (niprov.formatjson.DateTimeAwareJSONEncoder
method), 28defaultIgnore (niprov.diff.Diff attribute), 24deflate() (niprov.mongo.MongoRepository method), 33Dependencies (class in niprov.dependencies), 22deserialize() (niprov.formatjson.JsonFormat method), 28deserializeList() (niprov.formatjson.JsonFormat method),
28determineUser() (niprov.users.Users method), 39DicomFile (class in niprov.dcm), 22dict_to_object() (niprov.formatjson.DateTimeAwareJSONDecoder
method), 28DictFormat (class in niprov.formatdict), 27Diff (class in niprov.diff), 23digest() (niprov.hashing.Hasher method), 30DirectMedium (class in niprov.mediumdirect), 32discover() (in module niprov.discovery), 25discover() (niprov.context.ProvenanceContext method),
21discover_file_extensions (niprov.config.Configuration at-
tribute), 21discoveryFinished() (niprov.commandline.Commandline
method), 19dryrun (niprov.config.Configuration attribute), 21
Eexport() (in module niprov.exporting), 25export() (niprov.context.ProvenanceContext method), 21export() (niprov.mediumdirect.DirectMedium method),
32export() (niprov.mediumfile.FileMedium method), 32export() (niprov.mediumstdout.StandardOutputMedium
method), 32export() (niprov.mediumviewer.ViewerMedium method),
33exportedToFile() (niprov.commandline.Commandline
method), 19Externals (class in niprov.externals), 26
FFifFile (class in niprov.fif), 26fileAdded() (niprov.commandline.Commandline
method), 19fileError() (niprov.commandline.Commandline method),
19fileExists() (niprov.filesystem.Filesystem method), 26
FileFilter (class in niprov.filefilter), 26FileMedium (class in niprov.mediumfile), 32filesMarkedForApproval()
(niprov.commandline.Commandline method),19
Filesystem (class in niprov.filesystem), 26forFile() (niprov.pipelinefactory.PipelineFactory
method), 35Format (class in niprov.format), 27FormatFactory (class in niprov.formatfactory), 28fromString() (niprov.locationfactory.LocationFactory
method), 32
Gget() (niprov.context.ProvenanceContext method), 21getBytes() (niprov.pictures.PictureCache method), 34getCamera() (niprov.dependencies.Dependencies
method), 22getClock() (niprov.dependencies.Dependencies method),
22getConfiguration() (niprov.dependencies.Dependencies
method), 22getctime() (niprov.filesystem.Filesystem method), 26getDifferences() (niprov.diff.Diff method), 24getDifferenceString() (niprov.diff.Diff method), 24getExternals() (niprov.dependencies.Dependencies
method), 22getFields() (niprov.querying.Query method), 36getFileFactory() (niprov.dependencies.Dependencies
method), 22getFileFilter() (niprov.dependencies.Dependencies
method), 22getFilepath() (niprov.pictures.PictureCache method), 34getFilesystem() (niprov.dependencies.Dependencies
method), 23getFormatFactory() (niprov.dependencies.Dependencies
method), 23getHasher() (niprov.dependencies.Dependencies
method), 23getLibraries() (niprov.dependencies.Dependencies
method), 23getListener() (niprov.dependencies.Dependencies
method), 23getLocationFactory() (niprov.dependencies.Dependencies
method), 23getMediumFactory() (niprov.dependencies.Dependencies
method), 23getNowString() (niprov.clock.Clock method), 19getPictureCache() (niprov.dependencies.Dependencies
method), 23getPipelineFactory() (niprov.dependencies.Dependencies
method), 23getProtocolFields() (niprov.basefile.BaseFile method), 18getProvenance() (niprov.basefile.BaseFile method), 18
46 Index
niprov Documentation, Release 0.5
getQuery() (niprov.dependencies.Dependencies method),23
getRepository() (niprov.dependencies.Dependenciesmethod), 23
getSame() (niprov.diff.Diff method), 24getSameString() (niprov.diff.Diff method), 24getSerializer() (niprov.dependencies.Dependencies
method), 23getSeries() (niprov.jsonfile.JsonFile method), 31getSeries() (niprov.mongo.MongoRepository method), 33getSeries() (niprov.repository.Repository method), 38getSeriesId() (niprov.basefile.BaseFile method), 18getSeriesId() (niprov.dcm.DicomFile method), 22getsize() (niprov.filesystem.Filesystem method), 27getSnapshotFilepath() (niprov.basefile.BaseFile method),
18getUsers() (niprov.dependencies.Dependencies method),
23
Hhandler() (in module niprov.mnefunsupport), 33hasDependency() (niprov.libraries.Libraries method), 31hasFile() (niprov.dcm.DicomFile method), 22Hasher (class in niprov.hashing), 30home() (in module niprov.views), 39
Iimportp() (in module niprov.importing), 30importp() (niprov.context.ProvenanceContext method),
21include() (niprov.filefilter.FileFilter method), 26inflate() (niprov.mongo.MongoRepository method), 34inheritFrom() (in module niprov.inheriting), 30inquire() (niprov.jsonfile.JsonFile method), 31inquire() (niprov.mongo.MongoRepository method), 34inspect() (in module niprov.inspection), 30inspect() (niprov.basefile.BaseFile method), 18inspect() (niprov.cnt.NeuroscanFile method), 19inspect() (niprov.context.ProvenanceContext method), 21inspect() (niprov.dcm.DicomFile method), 22inspect() (niprov.fif.FifFile method), 26inspect() (niprov.nifti.NiftiFile method), 34interpretedRecording() (niprov.commandline.Commandline
method), 19
JJsonFile (class in niprov.jsonfile), 30JsonFormat (class in niprov.formatjson), 28
Kkeep() (niprov.pictures.PictureCache method), 34keepVersionsFromPrevious() (niprov.basefile.BaseFile
method), 18knows() (niprov.repository.Repository method), 38
knowsByLocation() (niprov.repository.Repositorymethod), 38
knowsSeries() (niprov.repository.Repository method), 38
Llatest() (in module niprov.views), 39latest() (niprov.jsonfile.JsonFile method), 31latest() (niprov.mongo.MongoRepository method), 34latest() (niprov.querying.Query method), 36latest() (niprov.repository.Repository method), 38Libraries (class in niprov.libraries), 31Location (class in niprov.location), 31location() (in module niprov.views), 39LocationFactory (class in niprov.locationfactory), 32log() (in module niprov.plogging), 35log() (niprov.commandline.Commandline method), 19log() (niprov.context.ProvenanceContext method), 21
Mmain() (in module niprov.webapp), 40markedForApproval() (in module niprov.approval), 18markedForApproval() (niprov.context.ProvenanceContext
method), 21markForApproval() (in module niprov.approval), 17markForApproval() (niprov.context.ProvenanceContext
method), 21MediumFactory (class in niprov.mediumfactory), 32mergeWith() (niprov.dcm.DicomFile method), 22missingDependencyForImage()
(niprov.commandline.Commandline method),19
mnefunEventReceived() (niprov.commandline.Commandlinemethod), 19
modalities() (in module niprov.views), 39modality() (in module niprov.views), 39MongoRepository (class in niprov.mongo), 33
Nname (niprov.querying.QueryField attribute), 36NarratedFormat (class in niprov.formatnarrated), 29NCHARSCOL (niprov.diff.Diff attribute), 23NeuroscanFile (class in niprov.cnt), 19new() (niprov.pictures.PictureCache method), 34NiftiFile (class in niprov.nifti), 34niprov (module), 40niprov.adding (module), 17niprov.approval (module), 17niprov.basefile (module), 18niprov.camera (module), 18niprov.capturing (module), 19niprov.clock (module), 19niprov.cnt (module), 19niprov.commandline (module), 19niprov.comparing (module), 20
Index 47
niprov Documentation, Release 0.5
niprov.config (module), 20niprov.context (module), 21niprov.dcm (module), 22niprov.dependencies (module), 22niprov.diff (module), 23niprov.discovery (module), 25niprov.exceptions (module), 25niprov.exporting (module), 25niprov.externals (module), 26niprov.fif (module), 26niprov.filefilter (module), 26niprov.filesystem (module), 26niprov.format (module), 27niprov.formatdict (module), 27niprov.formatfactory (module), 28niprov.formatjson (module), 28niprov.formatnarrated (module), 29niprov.formatobject (module), 29niprov.formatsimple (module), 29niprov.formatxml (module), 30niprov.hashing (module), 30niprov.importing (module), 30niprov.inheriting (module), 30niprov.inspection (module), 30niprov.jsonfile (module), 30niprov.libraries (module), 31niprov.location (module), 31niprov.locationfactory (module), 32niprov.mediumdirect (module), 32niprov.mediumfactory (module), 32niprov.mediumfile (module), 32niprov.mediumstdout (module), 32niprov.mediumviewer (module), 33niprov.mnefunsupport (module), 33niprov.mongo (module), 33niprov.nifti (module), 34niprov.pictures (module), 34niprov.pipeline (module), 35niprov.pipelinefactory (module), 35niprov.plogging (module), 35niprov.querying (module), 36niprov.recording (module), 36niprov.renaming (module), 37niprov.repository (module), 37niprov.searching (module), 39niprov.users (module), 39niprov.views (module), 39niprov.webapp (module), 40NiprovError, 25
OObjectFormat (class in niprov.formatobject), 29ordinal() (in module niprov.commandline), 20OutputCapture (class in niprov.capturing), 19
Pparents (niprov.basefile.BaseFile attribute), 18PictureCache (class in niprov.pictures), 34Pipeline (class in niprov.pipeline), 35pipeline() (in module niprov.views), 39PipelineFactory (class in niprov.pipelinefactory), 35print_() (in module niprov.exporting), 25print_() (niprov.context.ProvenanceContext method), 21project() (in module niprov.views), 39projects() (in module niprov.views), 39ProvenanceContext (class in niprov.context), 21
QQuery (class in niprov.querying), 36QueryField (class in niprov.querying), 36
Rread() (niprov.filesystem.Filesystem method), 27readlines() (niprov.filesystem.Filesystem method), 27receivedBashCommand()
(niprov.commandline.Commandline method),19
reconfigureOrGetConfiguration()(niprov.dependencies.Dependencies method),23
record() (in module niprov.recording), 36record() (niprov.context.ProvenanceContext method), 22renamedDicom() (niprov.commandline.Commandline
method), 20renameDicoms() (in module niprov.renaming), 37renameDicoms() (niprov.context.ProvenanceContext
method), 22Repository (class in niprov.repository), 37Result (class in niprov.externals), 26run() (niprov.externals.Externals method), 26
SsaveSnapshot() (niprov.camera.Camera method), 18saveToDisk() (niprov.pictures.PictureCache method), 34search() (in module niprov.searching), 39search() (in module niprov.views), 39search() (niprov.context.ProvenanceContext method), 22search() (niprov.jsonfile.JsonFile method), 31search() (niprov.mongo.MongoRepository method), 34selectApproved() (in module niprov.approval), 18selectApproved() (niprov.context.ProvenanceContext
method), 22serialize() (niprov.format.Format method), 27serializeAndWrite() (niprov.jsonfile.JsonFile method), 31serializeList() (niprov.format.Format method), 27serializeList() (niprov.formatdict.DictFormat method), 27serializeList() (niprov.formatjson.JsonFormat method),
28
48 Index
niprov Documentation, Release 0.5
serializeList() (niprov.formatobject.ObjectFormatmethod), 29
serializeList() (niprov.formatsimple.SimpleFormatmethod), 29
serializeList() (niprov.formatxml.XmlFormat method), 30serializePipeline() (niprov.format.Format method), 27serializePipeline() (niprov.formatobject.ObjectFormat
method), 29serializePipeline() (niprov.formatsimple.SimpleFormat
method), 29serializeSingle() (niprov.format.Format method), 27serializeSingle() (niprov.formatdict.DictFormat method),
27serializeSingle() (niprov.formatjson.JsonFormat method),
28serializeSingle() (niprov.formatnarrated.NarratedFormat
method), 29serializeSingle() (niprov.formatobject.ObjectFormat
method), 29serializeSingle() (niprov.formatsimple.SimpleFormat
method), 29serializeSingle() (niprov.formatxml.XmlFormat method),
30serializeSingle() (niprov.pictures.PictureCache method),
34serializeStatistics() (niprov.format.Format method), 27serializeStatistics() (niprov.formatobject.ObjectFormat
method), 29serializeStatistics() (niprov.formatsimple.SimpleFormat
method), 29serializeSummary() (niprov.formatsimple.SimpleFormat
method), 29serve() (in module niprov.webapp), 40short() (in module niprov.views), 39SimpleFormat (class in niprov.formatsimple), 29StandardOutputMedium (class in niprov.mediumstdout),
32statistics() (niprov.jsonfile.JsonFile method), 31statistics() (niprov.mongo.MongoRepository method), 34statistics() (niprov.querying.Query method), 36stats() (in module niprov.views), 39subject() (in module niprov.views), 39
TtakeSnapshot() (niprov.camera.Camera method), 18toDictionary() (niprov.location.Location method), 31toString() (niprov.location.Location method), 31toUrl() (niprov.location.Location method), 32
UUnknownFileError, 25update() (niprov.jsonfile.JsonFile method), 31update() (niprov.mongo.MongoRepository method), 34update() (niprov.repository.Repository method), 39
updateApproval() (niprov.jsonfile.JsonFile method), 31updateApproval() (niprov.mongo.MongoRepository
method), 34updateApproval() (niprov.repository.Repository method),
39user (niprov.config.Configuration attribute), 21user() (in module niprov.views), 40Users (class in niprov.users), 39users() (in module niprov.views), 40usingCopyAsParent() (niprov.commandline.Commandline
method), 20
Vvalue (niprov.querying.QueryField attribute), 36verbosity (niprov.config.Configuration attribute), 21versions (niprov.basefile.BaseFile attribute), 18view() (in module niprov.exporting), 25view() (niprov.context.ProvenanceContext method), 22ViewerMedium (class in niprov.mediumviewer), 33viewSnapshot() (niprov.basefile.BaseFile method), 18vlevels (niprov.commandline.Commandline attribute), 20
Wwalk() (niprov.filesystem.Filesystem method), 27write() (niprov.filesystem.Filesystem method), 27
XXmlFormat (class in niprov.formatxml), 30
Index 49
Top Related