What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software...
-
Upload
diana-rodgers -
Category
Documents
-
view
222 -
download
1
Transcript of What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software...
![Page 1: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/1.jpg)
What's New with Programmability and Scripting
Jon Peck
Technical Advisor and Principal Software Engineer
Las Vegas, November 2008 Copyright (c) SPSS Inc, 2008Copyright (c) SPSS Inc, 2008
![Page 2: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/2.jpg)
Do you have...
• Repetitive SPSS tasks?• A need to reduce manual work?• Problems you can't solve with traditional
syntax?• A need for statistical procedures not in SPSS
Statistics?• A need for higher productivity?• A need to reduce code maintenance time?
See if programmability can help...
Copyright (c) SPSS Inc, 2008
![Page 3: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/3.jpg)
Agenda
• Programmability introduction• Four examples
– Automating repetitive work: applySyntaxToFiles
– Integrating programs and scripting: SPSSINC MODIFY TABLES
– Adding R statistical procedures: SPSSINC QUANTILE REGRESSION
– Making your own user interface with .NETThe Statistical Explorer
Copyright (c) SPSS Inc, 2008
![Page 4: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/4.jpg)
SPSS Statistics embeds three programming languages
• Traditional syntax and Basic scripting still present• Plug-ins let you extend capabilities using
– Python
– R
– .NET languages (Windows only)
• Free plug-in downloads or get from cd• SPSS Developer Central web site provides
articles, SPSS-written modules, plug-ins and user contributions
Copyright (c) SPSS Inc, 2008
![Page 5: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/5.jpg)
BEGIN PROGRAM PYTHON.import spssprint "Hello, Las Vegas"END PROGRAM
• Python or R program code goes in the normal SPSS Statistics syntax window
My first program
Copyright (c) SPSS Inc, 2008
![Page 6: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/6.jpg)
• A program in the input stream can communicate with SPSS Statistics and control it and use Python or R facilities and modules (internal mode)
spss.Submit("GET FILE='c:/data/cars.sav'.")
• A Python or .NET application can embed SPSS Statistics inside itself (external mode)– User interface does not appear
• There is a lower level C API available in an SDK
Programmability combines SPSS Statistics with Python or R
Copyright (c) SPSS Inc, 2008
![Page 7: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/7.jpg)
Python and R are open source software
• Programmability plug-ins are an optional install– They are free– They make possible tapping the work of the
Python and R communities– Python and R have license agreements– SPSS has a Freeware license agreement
• SPSS is not the owner or licensor of the Python or R software. Any user of Python or R must agree to the terms of the license agreement located on the Python or R web site. SPSS is not making any statement about the quality of the Python or R programs. SPSS fully disclaims all liability associated with your use of the Python or R programs.
Copyright (c) SPSS Inc, 2008
![Page 8: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/8.jpg)
Programmability increases your power, flexibility, and productivity
• Generalization
– React flexibly to metadata, results, and the environment
• Automation
– Embed program logic in jobs
• Extension
– Add or extend procedures and transformations
– Tap existing R or Python statistical modules
• Integration
– Connect SPSS inputs and outputs to other agents
• And programmability is more funCopyright (c) SPSS Inc, 2008
![Page 9: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/9.jpg)
Version 17 has new programmability and scripting features
V 15
• Processor– SPSS syntax– Python
programs– .NET
programs
• Front End– SaxBasic
scripts– COM support
V 16 Additions
• Processor– R programs– Extension
commands– Dataset class
• Front End– Basic script
upgrade– Python
scripts
V 17 Additions
• Processor– Extension
improvements– Program and
script integration– R graphics in
Viewer
• Front End– Custom Dialog
Builder– New syntax
editorCopyright (c) SPSS Inc, 2008
![Page 10: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/10.jpg)
• Programs run in the regular syntax stream• Users can define SPSS Statistics syntax for
program and scripts via Extension mechanism.
• Users can create dialog boxes and menus using the Custom Dialog Builder.– Not just for extensions or programs
• Python and R output appears in the Viewer– plain text– pivot tables– charts
Programmability functionality is fully integrated into SPSS Statistics
Copyright (c) SPSS Inc, 2008
![Page 11: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/11.jpg)
• New for version 17– SETSMACRO: syntax for using Variable Sets– SPSSINC APRIORI: association rules (R)– SPSSINC HETCOR: polychoric and serial correlation (R)– SPSSINC MERGE TABLES: merge pivot tables– SPSSINC MODIFY OUTPUT: outline titling and styling– SPSSINC MODIFY TABLES: pivot table styling– SPSSINC QUANTREG: quantile regression (R)– SPSSINC RAKE: adjust weights to control totals– SPSSINC RANFOR/RANPRED: random forests (R)– SPSSINC RASCH: Rasch models (R)– SPSSINC ROBUST REGR: robust regression (R)– SPSSINC TOBIT REGR: Tobit regression (R)
• For version 16 (and 17)– Seven others
There are new extension commands available from Developer Central
Copyright (c) SPSS Inc, 2008
![Page 12: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/12.jpg)
– Write Python or R functions to implement the functionality• Use input API's to get data to Python or R• Use output API's to create pivot tables
– For extensions,• Define the syntax in an xml file• Use tools in extension.py to receive parsed output and pass to
implementing function– R extensions can be wrapped in Python code
• Use organization or author name as first word of command
– Use the Custom Dialog Builder to create the interface• The CDB is not just for extensions
– Test!– Package and distribute– Contributions to Developer Central are welcome
• Documentation available at SPSS Developer Central
You can create and share your own additions to SPSS Statistics
Copyright (c) SPSS Inc, 2008
![Page 13: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/13.jpg)
Expand the audience by creating SPSS Statistics syntax and dialog boxes
Copyright (c) SPSS Inc, 2008
![Page 14: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/14.jpg)
Generalize and automate work
Example I
Copyright (c) SPSS Inc, 2008
![Page 15: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/15.jpg)
• You have syntax files and need to process datasets not known in advance every day
• applySyntaxToFiles function applies a syntax file to each file in input specification
• Optionally saves new sav and output files• Optionally creates a log• Can be run in internal or external mode• Can be integrated into a larger process
applySyntaxToFiles
Copyright (c) SPSS Inc, 2008
![Page 16: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/16.jpg)
• Apply standard processing to an unknown set of files• Produce processed data and reports
Automating a routine process
Copyright (c) SPSS Inc, 2008
![Page 17: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/17.jpg)
begin program.import spss, spssaux3spssaux3.applySyntaxToFiles(inputspec="c:/temp/parts/*.sav", syntax = "c:/userconf2008lasvegas/dailychecks.sps", outputdatadir = "c:/temp/processed", outputfiledir = "c:/temp/processed", logfile ="c:/temp/processed/report.txt")end program.
• dailychecks.sps could apply data cleaning rules, modify data, and create reports
• This program could be run daily through Production Mode or PES job scheduler or used interactively
Use a program to drive processing
Copyright (c) SPSS Inc, 2008
![Page 18: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/18.jpg)
Use integrated scripting for better table presentation
Example II
Copyright (c) SPSS Inc, 2008
![Page 19: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/19.jpg)
• TableLooks provide static formatting for entire areas of a table– data cells– row and column layers
• You want tables with formatting beyond tableLooks• Many users copy tables to Excel and manually format
them • Basic and Python Scripting provide programmatic way
to do formatting• SPSSINC MODIFY TABLES provides syntax for
extensive formatting– Eliminates need to know scripting– Uses Extension mechanism for programs and Python
scripting
SPSSINC MODIFY TABLES extension command
Copyright (c) SPSS Inc, 2008
![Page 20: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/20.jpg)
SPSSINC MODIFY TABLES SUBTYPE='Crosstabulation' DIMENSION=ROWS SELECT='Std. Residual'/STYLES TEXTSTYLE=BOLD BACKGROUNDCOLOR=255 0 0APPLYTO='abs(x) >2'.
Use dynamic highlighting to make crosstab table easier to read
Copyright (c) SPSS Inc, 2008
![Page 21: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/21.jpg)
• Dialog created withCustom Dialog Builder
• Generates extension command syntax
• Easy to distribute
Custom dialog boxes are easy to create
Copyright (c) SPSS Inc, 2008
![Page 22: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/22.jpg)
SPSSINC MODIFY TABLES subtype='variables in the equation' SELECT="B" "Sig."/STYLES TEXTCOLOR = 0 0 255 BACKGROUNDCOLOR=0 255 0.
Copyright (c) SPSS Inc, 2008
Use static formatting to call out parts of a table
![Page 23: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/23.jpg)
SPSSINC MODIFY TABLES SUBTYPE="Custom Table" SELECT = "Total" DIMENSION=ROWS/STYLES BACKGROUNDCOLOR=255 255 88 TEXTSTYLE = BOLD
Format CTABLES totals to call them out
Copyright (c) SPSS Inc, 2008
![Page 24: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/24.jpg)
SPSSINC MODIFY TABLES SUBTYPE='Report' SELECT="<<ALL>>"/STYLES APPLYTO=DATACELLS TEXTCOLOR=255 255 255 TEXTSTYLE=BOLDCUSTOMFUNCTION="customstylefunctions.washColumnsBlue".
def washColumnsBlue(obj, i, j, numrows, numcols, section, more): mincolor=150. maxcolor=255. increment = (maxcolor - mincolor)/(numcols-1) colorvalue = round(mincolor + increment * j) obj.SetBackgroundColorAt(i,j, RGB((mincolor, mincolor, colorvalue)))
Use custom functions for special effects
Copyright (c) SPSS Inc, 2008
![Page 25: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/25.jpg)
Copyright (c) SPSS Inc, 2008
It is possible to get carried away with this
![Page 26: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/26.jpg)
Add R procedures seamlessly to SPSS Statistics
Example III
Copyright (c) SPSS Inc, 2008
![Page 27: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/27.jpg)
• R – has many advanced statistical procedures– is a complex language for programming statistics– is difficult to learn
• The R plug-in makes it easy to use R packages– SPSS Statistics datasets and Viewer output can be
processed by R using plug-in– Graphical, text, and table output appear in the Viewer– New SPSS datasets can be created from R– R can communicate with SPSS via 30 API's– Integration requires writing some R wrapper code
• Plug-in is downloadable from Developer Central
R procedures can be accessed from SPSS Statistics using the R plug-in
Copyright (c) SPSS Inc, 2008
![Page 28: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/28.jpg)
• Ordinary regression models conditional mean• Median regression is 50th quantile• Estimating quantiles is useful with varying
spread, asymmetries, outliers• Areas of application include
– empirical finance• value at risk• mutual fund investment styles• credit scoring
– school quality– demand analysis– others
Quantile regression models conditional quantiles
Copyright (c) SPSS Inc, 2008
![Page 29: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/29.jpg)
SPSSINC QUANTILE REGRESSION extension embeds R quantreg
Copyright (c) SPSS Inc, 2008
![Page 30: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/30.jpg)
Plots and pivot tables appear in the Viewer
Copyright (c) SPSS Inc, 2008
![Page 31: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/31.jpg)
New datasets appear in Data Editor windows
Copyright (c) SPSS Inc, 2008
![Page 32: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/32.jpg)
The .NET plug-in allows an easy .NET way to do your own user interface for SPSS Statistics
Example IV
Copyright (c) SPSS Inc, 2008
![Page 33: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/33.jpg)
The Statistical Explorer uses the .NET plug-in
Copyright (c) SPSS Inc, 2008
![Page 34: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/34.jpg)
The Statistical Explorer
Copyright (c) SPSS Inc, 2008
![Page 35: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/35.jpg)
Statistics and chart shown depends on variable measurement level
Copyright (c) SPSS Inc, 2008
![Page 36: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/36.jpg)
• SPSS Statistics 17 completes programmability and scripting building blocks
• Unification of programs and scripts
• Custom Dialog Builder
• Extension improvements and new extensions
• R integration and graphics in Viewer
• SPSS Developer Central is your friend
Where we have been today
Copyright (c) SPSS Inc, 2008
![Page 37: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/37.jpg)
? ?
Questions
Copyright (c) SPSS Inc, 2008
![Page 38: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/38.jpg)
• Generalization and automation
– applySyntaxToFiles
– SPSS MODIFY TABLES
• Extension
– SPSSINC MODIFY TABLES using integrated scripting
– SPSS QUANTREG using R
– Many new extension commands available
• Integration
– applySyntaxToFiles as part of a process
• And it's still more fun
Programmability increases your power, flexibility, and productivity
Copyright (c) SPSS Inc, 2008
![Page 39: What's New with Programmability and Scripting Jon Peck Technical Advisor and Principal Software Engineer Las Vegas, November 2008 Copyright (c) SPSS Inc,](https://reader035.fdocuments.us/reader035/viewer/2022062320/56649b47550346318e8bd06e/html5/thumbnails/39.jpg)
Contact
Copyright (c) SPSS Inc, 2008