NCL and PyNGLCommunity data analysis and

visualization software

2005 User Forum, May 17-19 CISL/IMAGe/SCD Mary Haley, haley@ucar.edu

2005 CISL User Forum May 17-19 haley@ucar.edu

Topics• NCAR Command Language

– Quick overview– Workshop update– What’s new– Near future plans

• Python interface to NCL’s graphics library– Introduction– Near future plans

• Long range plans for NCL and PyNGL

2005 CISL User Forum May 17-19 haley@ucar.edu

NCL - Quick overview

• An interpreted language designed for the analysis and visualization of data

• Powerful in three areas: file I/O, data analysis, visualization

• Runs on most UNIX systems, including MacOSX, LINUX, Windows/Cygwin

• FREE

2005 CISL User Forum May 17-19 haley@ucar.edu

NCL overview: file I/O• Data model based on netCDF model• One function to read multiple data formats• Handles netCDF, GRIB-1, HDF4, HDF-

EOS4, CCM History tape• GRIB reader one of the best• ASCII, binary

Answer to question from yesterday: NCL supports NARR data and includes the NARR parameter table (NCEP parameter table 131).

2005 CISL User Forum May 17-19 haley@ucar.edu

NCL overview: data analysis• Over 800 functions

- Spherical harmonics- Scalar and vector regridding- Vertical interpolation- EOFs

• Many tailored to geosciences• Most handle missing data• Can call C and Fortran routines

2005 CISL User Forum May 17-19 haley@ucar.edu

NCL overview: visualization

• High-quality visualizations

• Contours, XY, vectors, streamlines, maps

• Specialized scripts for skew-T, wind roses, histograms, panels, bar charts

• GSUN interface: NCL scripts for simplifying visualization calls

• Hundreds of plot options available

2005 CISL User Forum May 17-19 haley@ucar.edu

NCL strengths• Huge web-based example suite• NCL workshops• Consulting is knowledgeable and quick • Trivial to install, precompiled binaries• Enhancements based on user demand*

* Get ready for the data analysis and visualization survey!

2005 CISL User Forum May 17-19 haley@ucar.edu

NCL workshops• 3 full-day classes with lectures and labs• Taught by CGD staff• 22 in the last 4 years: 15 in Boulder, rest off-site:

– University of California at Santa Cruz– University of Alaska at Fairbanks– UCLA (twice)– Purdue– Naval Research Lab - Stennis Space Center– National Ocean Service - NOAA

• Cancelled until further notice

2005 CISL User Forum May 17-19 haley@ucar.edu

What’s new

• New NCL website

• Command line options

• Contouring on triangular meshes

• NCL binaries on Earth System Grid

• Enhancements to GRIB reader

• Vis5D writer functions

Online survey later this month

2005 CISL User Forum May 17-19 haley@ucar.edu

New NCL websitehttp://www.ncl.ucar.edu/

• Consolidates ngwww.ucar.edu and the www.cgd.ucar.edu graphics website

• Function documentation in one location

• “Applications” includes over 500 examples

• Lots of documentation; printable manuals

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

2005 CISL User Forum May 17-19 haley@ucar.edu

Command line option preview

• Predefined options (-V, -x, -P)

• Can assign variables and execute commands:ncl i=1 x=3.14 ‘vars=(/“T”,“U”,“V”/)’ file.ncl

• More options to come: byte-swapping control, printing control; would like to hear from you!

http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/

2005 CISL User Forum May 17-19 haley@ucar.edu

Contouring on triangular meshes

Dave Kennison and Dave Brown

2005 CISL User Forum May 17-19 haley@ucar.edu

New types of grids you can contour

• Grids already on a triangular mesh

• Grids represented by one-dimensional arrays of x, y, and z values

• Grids with “cell boundary” information

Grid from Tom Gross, NOAA/NOS/CSDL/MMAP

Grid from Tom Gross, NOAA/NOS/CSDL/MMAP

Grid from Brett D. Estrada, Naval Research Laboratory

Grid from Brett D. Estrada, Naval Research Laboratory

HOMME grid courtesy of Steve Thomas, SCD

HOMME grid courtesy of Steve Thomas, SCD

HOMME grid courtesy of Steve Thomas, SCD

From Dave Randall, CSU

From Dave Randall, CSU

http://isccp.giss.nasa.gov/

http://isccp.giss.nasa.gov/

Christophe Cassou (CNRS/CERFACS)

Christiane Jablonowski (ECMWF)

Christiane Jablonowski (ECMWF)

2005 CISL User Forum May 17-19 haley@ucar.edu

To see some NCL examples

go to:

http://www.ncl.ucar.edu/Applications/

and click on “Non-uniform grids”

2005 CISL User Forum May 17-19 haley@ucar.edu

NCL near future plans• CLOs - initial release• Revisiting our I/O module: HDF5, GRIB2,

netCDF 4 coming up• Streamlines and vectors on a triangular

mesh• Large file support, currently has same

limitations as older versions of netCDF

Did I mention the survey?

2005 CISL User Forum May 17-19 haley@ucar.edu

Sites that have downloaded and/or used NCL

in the last few years

Department of Meteorology, Yugoslavia

Nanjing University, China

University of Michigan

UCLA

METRI/KMA, Korea

Frontier Research System for Global Change, Japan

Iowa State

Hong Kong University of Science and Technology

UCSD

Czech Hydrometeorological Institute

Osservatorio Meteorologico Regionale, Italy

Institute for Atmospheric and Climate Science, Switzerland

University of Reading, UK

International Centre for Theoretical Physics, Italy

NOAA-CIRES Climate Diagnostics Center

Cooperative Institute for Mesoscale Meteorological Studies

Woods Hole Oceanographic Institution Joint Program

Department of Geology and Geophysics, Yale

NOAA/NOS/Coast Survey Development Laboratory

Center for Ocean-Land-Atmosphere StudiesDepartment of Physics (Atmospheric Physics), University of TorontoUniversidade Federal da Bahia New Mexico Institute of Mining and TechnologyPacific Northwest National LaboratoryAtmospheric Science Division at LLNLArgonne National LabUniversity of ColoradoLawrence Berkeley National LabUniversity of IllinoisNRL, NavyClimate Change & Impacts Laboratory, UCSCUniversity of WashingtonUniversity of ArizonaArctic Region Supercomputing Center, University of AlaskaCenter for Sustainability and the Global Environment, Univ of Wisconsin-

MadisonFleet Numerical Meteorology and Oceanography CenterAir pollution Section/Research Department, Norwegian Meteorological InstituteFrontier Research System for Global Change, University of Alaska FairbanksUniversity of Virginia

University of NevadaAstronomy, Physics and Geophysics, University of CopenhagenInstitute of Urban Meteorology, Beijing, ChinaScripps Institution of Oceanography, University of CaliforniaCERFACS, FranceEMS Environment Institute, Penn StateCentre Meteo UQAM Montreal, MontrealNational Research Institute for Earth Science and Disaster Prevention, JapanUniversity of L'Aquila, Department of Physics, ItalyInternational Research Institute for Climate Prediction, Columbia UniversityAlaska Experimental Forecast FacilityCentral Research Institute of Electric Power Industry, JapanNational Severe Storms LaboratoryLamont-Doherty Earth Observatory, Columbia University, NYUBC Earth and Ocean Sciences, CanadaEarth Tech, Concord, MassachusettsSchool of the Environment, Leeds University, UKInstitute For Terrestrial and Planetary Atmospheres, State Univ of NYCollege of Oceanic and Atmospheric Sciences, Oregon State UniversityInstitute of Geophysics - Climatology, University of Bergen, NorwayInstitute for Geophysics, UT-Austin

University New South WalesIndonesia National Institute of Aeronautics and SpaceUniversity of NebraskaQueensland Department of Natural ResourcesNC State UniversityUniversity of Costa RicaLouisiana State UniversityMITUniversity College LondonNagasaki UniversityDeutscher WetterdienstInstitute of Atmospheric Physics, Chinese Academy of SciencesUnited Nations University YSA Corporation Chesapeake Research Consortium Naval Postgraduate School University of North Carolina at Chapel HillResearch Center for Environmental Changes, TaiwanIMAQS, U. of Houston Pusan national University, KoreaUniversity of Victoria, British Columdia, Canada

Institute of Oceanology, RussiaRudjer Boskovic Institute, CroatiaMississippi State UniversityAdvanced Environment Monitoring Research Center, KoreaKyushu University, KoreaUniversity of Northern British ColumbiaKorea Meteorological AdministrationInternational Pacific Research Center/School of Ocean and Earth Science, HawaiiGerman Weather ServiceUniversity of WaterlooComalco Aluminium Ltd, New ZealandAltai State University, RussiaUniversity of MaineIQAFF, QuebecKinetica Analysis Corporation/University of Central FloridaCentre for Mathematical Modelling and Computer Simulation, IndiaUniversity of British Columbia Howard UniversityUniversidad del Pais VascoIndian Institute of ScienceUniversity of Delaware

PrincetonCentral Weather Bureau of TaiwanSeoul National University, KoreaCNR-Venice-ItalyWeathernews Americas Inc.Mephise Technology (Taiwan)Purdue UniversityOhio State UniversityUSDA Forest ServiceInternational Arctic Research CenterCalifornia Air Resources BoardU.A.E. Air Force Air DefenseUniversity of NebraskaCWB (Center for Wooden Boats? Central Weather Bureau? Canadian Wheat

Board?)GFDL/NOAAChung-Cheng Institute of TechnologyNational Renewable Energy LabZhejiang Climate Center, BeijingUniversity of Texas at AustinWageningen University, Netherlands

University of Athens Macao Meteorological Bureau, Hong KongDuke University Texas A&M University SD School of Mines Department of Marine, Earth and Atmospheric Sciences, NCSU Universidad Politecnica de Catalunya, SpainNational Taiwan UniversityUniversity of Pecs, HungaryKobe University, JapanUniversity of Tartu, EstoniaUC, Irvine Institute of Meteorology, University of EdinburghSan Francisco State University University of Kentucky College of Resources Science and Technology, Beijing Normal University

Meteo-France AgrometLS Cable Ltd., chiller developmentPotsdam Institute for Climate Impact Research, GermanyARPA Piemonte, Italy

USDA Animal Improvement Programs LaboratoryMarine Meteorological Observation and InvestigationDrexel University Indian Institute of Technology, DelhiCaltechAbu Dhabi International Airport Meteorological OfficeMichigan Tech UniversitySongXiaoya UniversityYonsei University, Seoul, KoreaJackson State UniversityClimate Research, Tokyo University

2005 CISL User Forum May 17-19 haley@ucar.edu

Python interface toNCL’s Graphics Library

(PyNGL)

Fred Clare and Mary Haley

http://www.pyngl.ucar.edu/

2005 CISL User Forum May 17-19 haley@ucar.edu

What is Python?

An interpreted, interactive, object-oriented programming language often compared

with Perl, Tcl, and Java.

2005 CISL User Forum May 17-19 haley@ucar.edu

Python strengths

• Open source and portable• Full object oriented capabilities• Well documented and easy to learn• Modular• Toolkits for building GUIs - Tkinter• 3D visualization - vtk, OpenGL• Extendable with C/C++ and Fortran• Large and rapidly growing user base

2005 CISL User Forum May 17-19 haley@ucar.edu

Python strengths: lots of existing code

• PyTables - HDF5 reader• PIL - Python Imaging Library • Libraries of scientific tools

- netCDF module - FFTPACK, SPHEREPACK, LAPACK

• PyGlobus - toolkit for GRID computing

2005 CISL User Forum May 17-19 haley@ucar.edu

Why a Python interface?

• Demand for high-quality 2D visualizations

• Mainstream language - wider audience

• Open source

• Collaboration opportunities

• Capitalize on the work of others

• Contribution to Python community

2005 CISL User Forum May 17-19 haley@ucar.edu

Introduction to PyNGL

• Modeled after popular GSUN interface• Same 2D publication-quality graphics as NCL• Multiple output formats - same as NCL• Extensive documentation• Line-by-line explanation of examples

import Nglfrom Scientific.IO.NetCDF import *

# Open the NetCDF file.nf = NetCDFFile("meccatemp.cdf","r")

# Get lat/lon/temperature variables.lat = nf.variables["lat"][:]lon = nf.variables["lon"][:]T = nf.variables["t"][0,:,:]

# Open a PS workstation.wks = Ngl.open_wks("ps","mecca")

# Contour & scalar field resources.res = Ngl.Resources()res.sfXArray = lonres.sfYArray = latres.cnFillOn = True

# Draw contour plot.contour = Ngl.contour(wks,T,res)

Ngl.end()

PyNGL

load "gsn_code.ncl"

begin; Open the NetCDF file. nf = addfile("meccatemp.cdf","r")

; Get lat/lon/temperature variables. lat = nf->lat lon = nf->lon T = nf->t(0,:,:)

; Open a PS workstation. wks = gsn_open_wks("ps","mecca")

; Contour & scalar field resources. res = True res@sfXArray = lon res@sfYArray = lat res@cnFillOn = True res@lbPerimOn = False

; Draw contour plot. contour = gsn_contour(wks,T,res)

end

NCL/GSUN

Data from the MECCA Project

2005 CISL User Forum May 17-19 haley@ucar.edu

Sample PyNGL visualizations

Dummy data

Courtesy Dennis Shea

Data from the MECCA Project

Data from the MECCA Project

Data from the MECCA Project

Courtesy Sylvia Murphy, NCAR ACD

From John Ertl, FNMOC

Based on a visualization of Joel Norris (Scripps) using dummy data

Climate division data courtesy Mark Stevens, NCAR CGD

Courtesy Mark Stevens, NCAR CGD

2005 CISL User Forum May 17-19 haley@ucar.edu

Current PyNGL status• Documentation complete (200+ pages) • Released PyNGL alpha in January 2004• Announced PyNGL beta in September at SciPy• Implementing processing and I/O modules • Researching compatibility with other packages

- especially 3D and file I/O• Responding to user requests and questions

Early reactions have been positive

2005 CISL User Forum May 17-19 haley@ucar.edu

Near future PyNGL plans

• Finish beta test

• Release processing and I/O modules for alpha testing

• Add more functionality and examples

• Announce to Python world

• Potential collaboration with other Python developers

2005 CISL User Forum May 17-19 haley@ucar.edu

Long range plans forNCL and PyNGL

• Re-architecture the data model

• Performance enhancements - mainly file I/O (aggregation)

• Image processing

• Review of display model

• Open source

2005 CISL User Forum May 17-19 haley@ucar.edu

Documentation

http://www.ncl.ucar.edu/

http://www.pyngl.ucar.edu/

My email address: haley@ucar.edu

2005 CISL User Forum May 17-19 haley@ucar.edu

Questions?