Using python to automate the integration of data sets into...
Transcript of Using python to automate the integration of data sets into...
![Page 1: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/1.jpg)
Using python to automate the integration of data sets into
ArcGIS
Chelsea Welker
GEOG 6180
December 6, 2015
![Page 2: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/2.jpg)
BACKGROUND: seismic exploration
1. Seismic exploration is the search of subsurface hydrocarbons (ex. oil and gas). Artificial seismic energy is generated. The seismic waves bounce of subsurface rock features and travel back to receivers and turned into seismic profiles.
2. The seismic profiles are imported into seismic software and rock type layers are interpreted (left, layer cake view) and also displayed as map view (right).
3. A 3D view of seismic profiles and a rock layer interpretation horizon.
4. Ideally, the seismic horizons are interpolated into a surface to better understand regional geology.
![Page 3: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/3.jpg)
PROBLEM: Merging data from another software program into ArcGIS
• Every seismic horizon is converted to points XYZ and then exported as text file
• The text file is not ready to be imported into ArcGIS, needs reformatting
• Additionally, these XYZ files are very large, over 1,400,000 points (60M file) for one horizon alone!
• Once reformatted, it is imported in ArcGIS as a XYZ layer event file, then converted to a feature class
• Interpolation process
• Very tedious and time consuming!!!
![Page 4: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/4.jpg)
OBJECTIVES: Can python help automate this tedious, time-consuming process?
• Clean up the export file to be ArcGIS ready
• Sample down the number of points to decrease process time, but still maintain integrity of original data set
• Create a feature class from the formatted csv
• Interpolate points into a surface that closely approximates manual surface contouring
![Page 5: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/5.jpg)
METHODSData: propriety text file results from seismic project export
Research: various python modules for reformatting and FC to FC
Research: Explore the Empirical Bayesian Kriging (EBK) interpolation
method in the ArcGIS Geostatistical Analysis extension
“Empirical Bayesian kriging (EBK) is a geostatisticalinterpolation method that automates the mostdifficult aspects of building a valid krigingmodel…via…multiple simulations.” - ESRI
![Page 6: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/6.jpg)
Python
Create a function to:
• Convert space-delimited DAT file into comma-delimited CSV file with headers XYZ• Clean up lock files
• Read in space-delimited text file
• Output new comma-delimited csv file
• Write new headers in csv file
• Reduce file size to 10%• Keep every nth (10th) row
• Do we lose shape of data by down-sampling?
Regional project covers 2 million square km
By decreasing data set to 10% = increasing point distance by only
0.14 km
![Page 7: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/7.jpg)
Python
Create a function to
• Convert csv to feature class• Make temporary XY event table view
• Convert table into a geodatabase table
• Convert GDB table into feature class
• Iterate for every text file in folder• Batch process for more than one
horizon file
![Page 8: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/8.jpg)
Distribution of data points(dark black lines)
Manual interpretation by a geologist (thin, black lines)
Computer interpolation (color gradient background)
First approximation isn’t too bad!
ArcGIS: comparing EBK to manual contouring
![Page 9: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/9.jpg)
Using python to automate the process of converting text files to a feature class was a huge time saver, so it gets a gold star!
Manual interpretation is always better, but if time does not permit or as an exploratory analysis, EBK isn’t too bad. Furthermore, more refinements in EBK parameters could improve it even more.
In summary…
![Page 10: Using python to automate the integration of data sets into ...cwmapping.com/assets/files/CWelker_Presentation_GeopPython.pdfUsing python to automate the integration of data sets into](https://reader034.fdocuments.us/reader034/viewer/2022042805/5f5fe8c01643fa12db5743bd/html5/thumbnails/10.jpg)
REFERENCES
https://desktop.arcgis.com/en/desktop/latest/guide-books/extensions/geostatistical-analyst/what-is-empirical-bayesian-kriging-.htmhttp://gis.stackexchange.com/http://resources.arcgis.com/http://www.esri.com/esri-news/arcuser/summer-2014/creating-faulted-geologic-surfaces-with-arcgis
The End