Python in Civil/Environmental Engineering
-
Upload
pmhobson -
Category
Technology
-
view
2.509 -
download
12
Transcript of Python in Civil/Environmental Engineering
![Page 1: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/1.jpg)
PYTHON IN CIVIL AND ENVIRONMENTAL ENGINEERING
PAUL HOBSON GEOSYNTEC CONSULTANTS PDX PYTHON 2016-03-24
![Page 2: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/2.jpg)
PYTHON IN CIVIL ENGINEERING
HEALTH AND SAFETY
▸ Nearest exit in case of emergency
▸ Ergonomics
▸ Right angles
▸ Head level
▸ Take breaks, look and move around
![Page 3: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/3.jpg)
PYTHON IN CIVIL ENGINEERING
I’M AN ENGINEER (I STUDIED MUD)
▸ What is Civil Engineering?
▸ Structural
▸ Transportation
▸ Construction Management
▸ Geotechnical
▸ Environmental
▸ Water and Natural Resources
![Page 4: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/4.jpg)
PYTHON IN CIVIL ENGINEERING
I’M AN ENGINEER (I STUDIED MUD)
▸ What is Civil Engineering?
▸ Structural
▸ Transportation
▸ Construction Management
▸ Geotechnical
▸ Environmental
▸ Water and Natural Resources
Thesis Title:
Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers
![Page 5: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/5.jpg)
PYTHON IN CIVIL ENGINEERING
I’M AN ENGINEER (I STUDIED MUD)
▸ What is Civil Engineering?
▸ Structural
▸ Transportation
▸ Construction Management
▸ Geotechnical
▸ Environmental
▸ Water and Natural Resources
Thesis Title:
Rheologic and Flume Erosion Characteristics of Georgia Sediments at Bridge Piers
![Page 6: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/6.jpg)
PYTHON IN CIVIL ENGINEERING
I WORK AS AN ENVIRONMENTAL CONSULTANT
▸ Help industry with compliance and remediation
▸ Litigation support for allocation of environmental damages
▸ Grant-funded research with NCHRP, FHWA, WERF
![Page 7: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/7.jpg)
PYTHON IN CIVIL ENGINEERING
I WORK AS AN ENVIRONMENTAL CONSULTANT
![Page 8: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/8.jpg)
PYTHON IN CIVIL ENGINEERING
I WORK AS AN ENVIRONMENTAL CONSULTANT
▸ Build spatial databases of contaminant concentrations
![Page 9: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/9.jpg)
PYTHON IN CIVIL ENGINEERING
I WORK AS AN ENVIRONMENTAL CONSULTANT
▸ Build spatial databases of contaminant concentrations
▸ Scrape the web (brute force) to meteorologic and hydrologic data
![Page 10: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/10.jpg)
PYTHON IN CIVIL ENGINEERING
I WORK AS AN ENVIRONMENTAL CONSULTANT
▸ Build spatial databases of contaminant concentrations
▸ Scrape the web (brute force) to meteorologic and hydrologic data
▸ Statistical analysis of pollution mitigation and remediation systems
![Page 11: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/11.jpg)
PYTHON IN CIVIL ENGINEERING
I WORK AS AN ENVIRONMENTAL CONSULTANT
▸ Build spatial databases of contaminant concentrations
▸ Scrape the web (brute force) to meteorologic and hydrologic data
▸ Statistical analysis of pollution mitigation and remediation systems
▸ Build inputs for, run, and analyze output from vetted numerical models
![Page 12: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/12.jpg)
PYTHON IN CIVIL ENGINEERING
I WORK AS AN ENVIRONMENTAL CONSULTANT
▸ Build spatial databases of contaminant concentrations
▸ Scrape the web (brute force) to meteorologic and hydrologic data
▸ Statistical analysis of pollution mitigation and remediation systems
▸ Build inputs for, run, and analyze output from vetted numerical models
▸ Build tools to help my colleagues do the same
![Page 13: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/13.jpg)
PYTHON IN CIVIL ENGINEERING
SUMMARY FOR THE REST OF THE TALK
▸ My take on the general state of the practice in engineering consulting
▸ Characteristic challenges of environmental consulting
▸ Tools I like to use
▸ Tools I have built over the course of my career
![Page 14: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/14.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
![Page 15: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/15.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
![Page 16: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/16.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
![Page 17: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/17.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
▸ Last Client’s DB
![Page 18: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/18.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
▸ Last Client’s DB
▸ 110 tables
![Page 19: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/19.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
▸ Last Client’s DB
▸ 110 tables
▸ 200 saved insert/create/update
![Page 20: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/20.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
▸ Last Client’s DB
▸ 110 tables
▸ 200 saved insert/create/update
▸ 100 save select and pivot queries
![Page 21: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/21.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
▸ Last Client’s DB
▸ 110 tables
▸ 200 saved insert/create/update
▸ 100 save select and pivot queries
▸ 25 queries that deleted stuff
![Page 22: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/22.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DATABASES
▸ Overwhelming everything is in MS Access
▸ PROTIP: Your MS Office, MS Access Drivers, and python all must have the same architecture to work together
▸ Last Client’s DB
▸ 110 tables
▸ 200 saved insert/create/update
▸ 100 save select and pivot queries
▸ 25 queries that deleted stuff
▸ Everything connected with 40 VBA forms, 15 utility modules (9000 LOC)
![Page 23: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/23.jpg)
PYTHON IN CIVIL ENGINEERING
![Page 24: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/24.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - ANALYSIS
▸ Excel is the behemoth that cannot be stopped
▸ Most people manage their data with it
▸ No one follows best practices (www.datacarpentry.org/spreadsheet-ecology-lesson)
▸ Will be pried from cold dead hands, despite will published statistical errors (resolved, IIRC)
![Page 25: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/25.jpg)
PYTHON IN CIVIL ENGINEERING
MANNING’S EQUATION
![Page 26: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/26.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
![Page 27: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/27.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
![Page 28: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/28.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows
![Page 29: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/29.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows
▸ HEC-RAS
▸ 2-D River hydraulics
▸ Completely closed source
▸ Guarded over by US Army Corps of Engineers
![Page 30: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/30.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows
▸ HEC-RAS
▸ 2-D River hydraulics
▸ Completely closed source
▸ Guarded over by US Army Corps of Engineers
▸ EFDC
![Page 31: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/31.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows
▸ HEC-RAS
▸ 2-D River hydraulics
▸ Completely closed source
▸ Guarded over by US Army Corps of Engineers
▸ EFDC
▸ Spaghetti FORTRAN
![Page 32: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/32.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows
▸ HEC-RAS
▸ 2-D River hydraulics
▸ Completely closed source
▸ Guarded over by US Army Corps of Engineers
▸ EFDC
▸ Spaghetti FORTRAN
▸ Compiles on Linux but results are garbage
![Page 33: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/33.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows
▸ HEC-RAS
▸ 2-D River hydraulics
▸ Completely closed source
▸ Guarded over by US Army Corps of Engineers
▸ EFDC
▸ Spaghetti FORTRAN
▸ Compiles on Linux but results are garbage
▸ 3-D river hydraulics, sediment and pollutant fate and transport
![Page 34: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/34.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
▸ GUI or cmd-line based numerical models maintained by federal agencies
▸ SWMM
▸ 1-D urban hydrology
▸ Written in very legible C
▸ Open source
▸ Maintained by EPA, transitioning to UT Austin
▸ Compiles on Linux and Windows
▸ HEC-RAS
▸ 2-D River hydraulics
▸ Completely closed source
▸ Guarded over by US Army Corps of Engineers
▸ EFDC
▸ Spaghetti FORTRAN
▸ Compiles on Linux but results are garbage
▸ 3-D river hydraulics, sediment and pollutant fate and transport
▸ Very optimized: 1 yr simulation takes ~ 1 day
![Page 35: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/35.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN-SPECIFIC TOOLS
![Page 36: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/36.jpg)
PYTHON IN CIVIL ENGINEERING
![Page 37: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/37.jpg)
PYTHON IN CIVIL ENGINEERING
STATE OF THE PRACTICE - DOMAIN SPECIFIC TOOLS
▸ All (numerically) solve the St. Venant Equations
▸ Differential equations in time and space
▸ Very difficult to even work with symbolically
▸ Large code bases
▸ Represent a significant intellectual investment from civil/environmental community
![Page 38: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/38.jpg)
PYTHON IN CIVIL ENGINEERING
WHERE DOES PYTHON FIT IN ALL OF THIS?
▸ Scrape model inputs from web
▸ Hack input files for batch processing
▸ Move data around between formats
▸ General data and model results analysis
▸ (Rarely) Wrap C/Fortran libraries with ctypes/cython/numpy
![Page 39: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/39.jpg)
PYTHON IN CIVIL ENGINEERING
CIVIL/ENVIRONMENTAL CHALLENGES
▸ Arcane input file formats
▸ Left-censored (non-detect) data
▸ Non-parametric statistics
▸ Regression-on-order statistics
▸ Kaplan-Meier
▸ Project management inertia
▸ Disconnect between libraries and project needs
▸ ~5 coworkers (out of ~600) are comfortable with python
![Page 40: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/40.jpg)
PYTHON IN CIVIL ENGINEERING
TOOLS I USE
▸ numpy/scipy
▸ Fast numeric arrays implemented in C
▸ Specialized scientific functions
▸ Matplotlib/seaborn: generic and statistical 2-D visualizations
▸ pandas/statsmodels: table-like data structures and statistical models
▸ Jupyter: interactive computing via notebooks
▸ conda: sane installation of python packages on Windows and other operating systems
![Page 41: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/41.jpg)
PYTHON IN CIVIL ENGINEERING
![Page 42: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/42.jpg)
PYTHON IN CIVIL ENGINEERING
SCRAPING WEATHER DATA
▸ conda install —channel=phobson cloudside
▸ Python 3 fork of python-metar by Tom Pollard (github.com/tomp/python-metar.git)
▸ ~1000 lines of regex
▸ Cloudside uses python-metar to pull 5-min and hourly data from the FAA
▸ ~1000 lines of regex
▸ 24229KPDX PDX20120101000009801/01/12 00:00:31 5-MIN KPDX 010800Z 31005KT 10SM OVC085 04/02 A3026 -280 85 -1500 290/05 RMK A02 P0005
![Page 43: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/43.jpg)
PYTHON IN CIVIL ENGINEERING
WEATHER DATA
▸ Simple commands for bulk data download and parsing
▸ Creates a directory structure for each station, source, and stage of processing
▸ searches for existing files before attempting to download or parse
▸ High-level plotting functions
![Page 44: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/44.jpg)
PYTHON IN CIVIL ENGINEERING
![Page 45: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/45.jpg)
PYTHON IN CIVIL ENGINEERING
![Page 46: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/46.jpg)
PYTHON IN CIVIL ENGINEERING
WQIO - WATER QUALITY, INFLOW/OUTFLOW
![Page 47: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/47.jpg)
PYTHON IN CIVIL ENGINEERING
WQIO - WATER QUALITY, INFLOW/OUTFLOW
▸ conda install --channel=phobson wqio
![Page 48: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/48.jpg)
PYTHON IN CIVIL ENGINEERING
WQIO - WATER QUALITY, INFLOW/OUTFLOW
▸ conda install --channel=phobson wqio
▸ Centered around examining the efficacy of so-called BMPs
![Page 49: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/49.jpg)
PYTHON IN CIVIL ENGINEERING
WQIO - WATER QUALITY, INFLOW/OUTFLOW
▸ conda install --channel=phobson wqio
▸ Centered around examining the efficacy of so-called BMPs
▸ Handles left-censored data with ROS
▸ No such things as having “zero” pollution
▸ Can only say that things are “less than” the instrument’s precision
![Page 50: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/50.jpg)
PYTHON IN CIVIL ENGINEERING
WQIO - WATER QUALITY, INFLOW/OUTFLOW
▸ conda install --channel=phobson wqio
▸ Centered around examining the efficacy of so-called BMPs
▸ Handles left-censored data with ROS
▸ No such things as having “zero” pollution
▸ Can only say that things are “less than” the instrument’s precision
▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics
![Page 51: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/51.jpg)
PYTHON IN CIVIL ENGINEERING
WQIO - WATER QUALITY, INFLOW/OUTFLOW
▸ conda install --channel=phobson wqio
▸ Centered around examining the efficacy of so-called BMPs
▸ Handles left-censored data with ROS
▸ No such things as having “zero” pollution
▸ Can only say that things are “less than” the instrument’s precision
▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics
▸ High-level plotting interface built on seaborn
![Page 52: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/52.jpg)
PYTHON IN CIVIL ENGINEERING
WQIO - WATER QUALITY, INFLOW/OUTFLOW
▸ conda install --channel=phobson wqio
▸ Centered around examining the efficacy of so-called BMPs
▸ Handles left-censored data with ROS
▸ No such things as having “zero” pollution
▸ Can only say that things are “less than” the instrument’s precision
▸ Bias-corrected, accelerated non-parametric bootstrapping to estimate confidence intervals around statistics
▸ High-level plotting interface built on seaborn
▸ Basis for client-specific libraries that I distribute to them through conda
![Page 53: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/53.jpg)
PYTHON IN CIVIL ENGINEERING
PARAMNORMAL - CONVENTIONALLY PARAMETERIZED STATISTICAL DISTRIBUTIONS
▸ http://phobson.github.io/paramnormal/index.html
▸ conda install --channel=phobson paramnormal
▸ Problem: scipy distributions are incredibly flexible, but also perhaps a bit over generalized
▸ Wanted to create an API that let statisticians of all caliber specify and fit distributions using the parameters they read about in text books.
▸ The activity module provides a simple interface for creating, fitting, and plotting statistical distributions
![Page 54: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/54.jpg)
PYTHON IN CIVIL ENGINEERING
PARANORMAL: BRIEF EXAMPLES
▸ (super fast demo)
![Page 55: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/55.jpg)
PYTHON IN CIVIL ENGINEERING
PARAMNORMAL: ENDORSEMENTS
![Page 56: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/56.jpg)
PYTHON IN CIVIL ENGINEERING
PARAMNORMAL: ENDORSEMENTS
![Page 57: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/57.jpg)
PYTHON IN CIVIL ENGINEERING
PARAMNORMAL: ENDORSEMENTS
![Page 58: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/58.jpg)
PYTHON IN CIVIL ENGINEERING
MPL-PROBSCALE▸ Legit probability
scales for matplotlib
▸ Similar to a quantile plot, but expressed as a probability instead of a z-score
▸ Simply import probscale and you’re set
import probscale ax1.set_xscale('prob') ax1.set_xlim(left=2, right=98) ax1.set_xlabel('Normal probability scale')
ax2.set_xscale('prob', dist=paramnormal.beta(α=3, β=2)) ax2.set_xlim(left=2, right=98) ax2.set_xlabel('Beta probability scale (α=3, β=2)')
![Page 59: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/59.jpg)
PYTHON IN CIVIL ENGINEERING
MPL-PROBSCALE▸ Can fit linear
regression in probability space
▸ Top-level functions for easy plotting
![Page 60: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/60.jpg)
PYTHON IN CIVIL ENGINEERING
GENERATING MODEL GRIDS
▸ Multi-corner, curvilinear-orthogonal grids are fairly difficult to compute — lots of very expensive proprietary stuff out there.
▸ Dr. Pavel Sakov (Australian Bureau of Meteorology wrote gridgen-c
▸ Built a types interface with help from:
▸ Dr. Robert Hetland (Texas A&M)
▸ Dr. Richard Signell (USGS)
▸ Mac and Linux binaries available through conda
▸ Docs at: phobson.github.io/pygridgen
![Page 61: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/61.jpg)
PYTHON IN CIVIL ENGINEERING
GRID EXAMPLE
![Page 62: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/62.jpg)
PYTHON IN CIVIL ENGINEERING
GRID EXAMPLE
![Page 63: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/63.jpg)
PYTHON IN CIVIL ENGINEERING
HIGHER LEVEL GRID OPERATIONS
▸ conda install --channel=conda-forge pygridtools
▸ geosyntec.github.io/pygridtools
▸ Interactivity with grid generation parameters
▸ Simple, general manipulations methods (merge, split, refine, transform)
▸ Took inspiration from pandas/xarray/seaborn
▸ File IO with general and GIS data formats via fiona
![Page 64: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/64.jpg)
PYTHON IN CIVIL ENGINEERING
GRID EXAMPLE - MANIPULATION
grid1, grid2, grid3 = make_test_grids()
fig = ( grid2.merge(grid3, how='vert', where='-', shift=2) .merge(grid1, how='horiz', where='-', shift=11) .refine(10, axis=1, n_points=4) .refine(13, axis=0, n_points=2) .transform(lambda x: x*5 + 2) .update_cell_mask() .plotCells(ax=ax, cell_kws=dict(facecolor=palette[2], alpha=0.75)) )
![Page 65: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/65.jpg)
PYTHON IN CIVIL ENGINEERING
GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON
![Page 66: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/66.jpg)
PYTHON IN CIVIL ENGINEERING
GRIDS IN THE REAL WORLD: SULLIVAN CREEK, NE WASHINGTON
![Page 67: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/67.jpg)
PYTHON IN CIVIL ENGINEERING
PROJECTING MODEL RESULTS ON THE GRID
![Page 68: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/68.jpg)
PYTHON IN CIVIL ENGINEERING
PROJECTING MODEL RESULTS ON THE GRID
![Page 69: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/69.jpg)
PYTHON IN CIVIL ENGINEERING
RESOURCES FOR GETTING STARTED AS A MEAT-SPACE ENGINEER
▸ I was lucky enough to get started with numpy and matplotlib while:
▸ I was still very familiar with MATLAB
▸ Both projects were directly trying to take market share away from MATLAB
▸ APIs and efforts have changed since then.
▸ More recently, a new employee started with MATLAB experience from college. We gave them a copy of Python For Data Analysis by Wes McKinney (author of pandas), and were pretty productive with python in a couple of week.
![Page 70: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/70.jpg)
PYTHON IN CIVIL ENGINEERING
LINK DUMP
▸ Me:
▸ https://twitter.com/pmhobson
▸ https://github.com/phobson
▸ My (documented) Projects
▸ http://geosyntec.github.io/pygridtools/
▸ http://phobson.github.io/paramnormal/
▸ http://phobson.github.io/mpl-probscale
▸ https://github.com/phobson/cloudside (docs forthcoming)
![Page 71: Python in Civil/Environmental Engineering](https://reader033.fdocuments.us/reader033/viewer/2022051101/587920a01a28abf13a8b6f3b/html5/thumbnails/71.jpg)
DEMOS &
QUESTIONS.
Thank you