Carl Wang

Indigo Books & Music

Get Data - ETL Source

Data Warehouse

MS SQL

Teradata

Oracle

SAP ……

Connect via interface/adapter (require SAS/ACCESS module license)

Simple and straight forward

PC Files (txt, csv, excel etc)

Sometimes can be diverse and difficult to manipulate

Usually by feed and delivered periodically

Scenario External files: Files without such pattern that easy to manipulate, eg: CustomExtract-Data-ICO_D_1-1624521-100713.xlsx

CustomExtract-Data-ICO_D_1-1609742-100813.xlsx

No valid DBMS type for importing (SAS/ACCESS to PC File Formats is not licensed on the SAS Server *) EXCEL

XLS

* proc setinit; run;

Possible Approaches Automation via:

SAS Server

SAS Macro

X Command

Stored Process

EG

EG Custom Task

EG automation interface (OLE automation)

Real World SAS Server SAS Macro only has limited control outside of SAS

session (with FILENAME statement)

X command turned off (due to security concern?? del /f /q *.*)

Stored process is SAS program hosted in central server and has same limitation as above

EG Automation Require knowledge of computer programming language

But more powerful and flexible than other approaches

Bridge the two very powerful

worlds: the world of your

SAS session and the world

of your Windows desktop

Note: COM - Component

Object Model

What EG Automation can do Anything that you can do with a SAS program, you can do

with a (custom task). And, anything that you can do with a Windows application you can do with a (custom task). If you need access to data or resources that you cannot reach with a SAS program, a (custom task) can act as the bridge to bring this information into SAS.

from Custom Tasks for SAS® Enterprise Guide® Using Microsoft .NET by Chris Hemedinger

Replace the custom task with scripting and the statement still true

Well Known Practice Schedule EG Project (see below sample)Option ExplicitDim appCall dowork'shut down the appIf not (app Is Nothing) Thenapp.Quit

Set app = NothingEnd IfSub dowork()

On Error Resume Next'----

' Start up Enterprise Guide using the project name'----Dim prjNameDim prjObject

prjName = "C:\Users\cwang\Documents\EG Projects\Testschedule.egp" 'Project NameSet app = CreateObject("SASEGObjectModel.Application.4.2")If Checkerror("CreateObject") = True Then

Exit SubEnd If'-----

' open the project'-----Set prjObject = app.Open(prjName,"")

Interface Approach by VBScriptSet sasApp = CreateObject("SASEGObjectModel.Application.5.1") *

' Set to your metadata profile namesasApp.SetActiveProfile("sasserver") *

' Create a new ProjectSet sasProject = sasApp.New

' add a new code object to the ProjectSet sasProgram = sasProject.CodeCollection.Add

' set the results types, overriding Application defaultssasProgram.UseApplicationOptions = FalsesasProgram.GenListing = FalsesasProgram.GenSasReport = False

' Set the server and text for the codesasProgram.Server = "SASApp " *

* See next slide for the location where to grab those parameters

Files ManipulationSet objFSO = CreateObject("Scripting.FilesystemObject")

‘automatic retrive directory

workingDir = objFSO.GetParentFolderName(WScript.ScriptFullName)

‘obtain collections of the folder

Set myFolder = objFSO.GetFolder(workingDir)

‘obtain files under directory

Set fileCollection = myFolder.Files

Transform Excel files (Optional*)Set objExcel = CreateObject("Excel.Application")

objExcel.Visible = False

objExcel.DisplayAlerts= False

For Each allFile In fileCollection

ext = Right(allFile.Name,5)

If UCase(ext) = UCase(extension)

‘and allFile.DateLastModified>=CDate(Date)-1

‘if we want define days of files to be processed

then

revPath = Mid(allFile, 1, InStrRev(allFile, "\")) & "CSV\"

‘ revFileName = Mid(allFile,InStrRev(allFile,"-")+1,InstrRev(allFile,".")-InStrRev(allFile,"-")-1)

FileName = Left(allFile.Name,InStr(allFile.Name,".")-1)

'open excel

Set objWorkbook =

objExcel.Workbooks.Open(allFile)

SaveName = revPath & FileName & ".csv"

objWorkbook.SaveAs SaveName,6

objWorkbook.Close False

objExcel.Quit

txtSAS = txtSAS & "proc import file='" & SaveName & "' out=test.out" & revFileName & " replace; run;“

End If

Next

* If has SAS/ACCESS to PC File Formats installed

If Multi-Sheet (Optional)‘in case the excel file has multi-sheet:

i = objWorkbook.Worksheets.Count

For k = 1 to i

SaveName = "\\file path\" & objWorkbook.Worksheets(k).Name & ".csv"

owb.Sheets(k).Activate

owb.SaveAs SaveName,6

Next

Assemble the ScriptOption Explicit

Dim sasApp ' EG ApplicationDim objExcel ' ExcelDim objFSO ' Filesystem Navigation

Const Extension = ".xlsx"

Call Automate

'release object reference

Set sasApp= NothingSet objExcel = NothingSet objFso = Nothing

Sub Automate()

Dim myFolder,fileCollection,allFile,FileName,revFileName,revPath,SaveName,ext,workingDir

Dim txtSAS,readSAS,sasProgDim objWorkbookDim runDate

Dim sasProject ' Project objectDim sasProgram ' Code object

Set objFso = CreateObject("Scripting.FilesystemObject")workingDir = objFso.GetParentFolderName(WScript.ScriptFullName)

Set myFolder = objFso.GetFolder(workingDir)

Set fileColl ection= myFolder.Files

Set objExcel = CreateObject("Excel.Application")

objExcel.Visible = FalseobjExcel.DisplayAlerts= False

'SAS Portion

Set sasApp = CreateObject("SASEGObjectModel.Application.5.1")

sasApp.SetActiveProfile("sasserver")

Set sasProject = sasApp.New

Set sasProgram = sasProject.CodeCollection.Add

sasProgram.UseApplicationOptions = FalsesasProgram.GenListing = FalsesasProgram.GenSasReport = False

sasProgram.Server = "SASApp"

' Create the SAS program to runsasProgram.Text = "libname test '" & workingDir & "\DS';"

Cont.For Each allFile In fileCollection

ext = Right(allFile.Name,5)If UCase(ext) = UCase(extension and allFile.DateLastModified>=CDate(Date)-1

'open excelrevPath = Mid(allFile, 1, InStrRev(allFile, "\")) & "CSV\"

revFileName = Mid(allFile,InStrRev(allFile,"-")+1,InstrRev(allFile,".")-InStrRev(allFile,"-")-1)

FileName = Left(allFile.Name,InStr(allFile.Name,".")-1)

Set objWorkbook = objExcel.Workbooks.Open(allFile)SaveName = revPath & FileName & ".csv"objWorkbook.SaveAs SaveName,6objWorkbook.Close False

objExcel.Quit

txtSAS = txtSAS & "proc import file='" & SaveName & "' out=test.out" & revFileName & " replace; run;"

End IfNext

'execute stored sas code

Set readSAS = objFSO.OpenTextFile(workingdir&"\test.sas", 1)sasProg = readSAS.ReadAll

sasProgram.Text = sasProgram.Text & txtSAS & sasProg

'WScript.Echo sasProgram.Text

' Run the codesasProgram.Run

' Save the log file to LOCAL disk

runDate="ETL_" & Year(Date) & Month(Date) & Day(Date)

sasProgram.Log.SaveAs workingDir & "\DS\LOG\" & runDate & ".log"

Application.Quit

End Sub

Alternative Other Script Language

Power shell; Python; Perl etc.

Write own custom task C#

(ETLTask.dll) built by me and will be covered next slide

If has SAS/ACCESS to PC File Formats installed Using existing EG Add In: SysCommand.dll

http://support.sas.com/documentation/onlinedoc/guide/customtasks/samples/SystemCommand42.zip

Base SAS: FILENAME statement and SAS Macro

Custom Task – ETL Task

Choose the files’ extension - available from dropdown list: xlsx; xls; txt; csv or *.* for any flat files (default: txt)

More Options Choose the start line of the file, similar to the firstobs option of the data step

(default: 1)Whether or not transforms the excel files

(default: false) Specify the getnames option for the proc import

(default: yes) Select the folder where files reside or manually enter it if you are using UNC

path (default: C:\)

Whether or not provide the prefix for the output dataset name(default: false)

Specify the prefix if option 6 checked.(default: Out)

Specify how many days back files to be processed based on the modified date (default: 7 Days)

Powerful but not Easy Build your own

If you have reasonable computer programming knowledge, especially object-oriented language

If the job can be modularized and need to be performed regularly

Or Get the resource or ask help from SAS community, use the

pre-built

You can contact me at carlwang2006@gmail.com to acquire a copy of the ETLTask.dll or the source code if you want compile by your own.

Use Pre-Built Custom Task Simpler but with a bit more steps

Still need to code in SAS

SAS Codedata dir;infile '\\file path\files.txt' truncover;input fn $100.;format dt date9.;dt=input(reverse(substr(left(reverse(fn)),6,6)),mmddyy8.);if intck('day', dt, today())>0; /*define days of files to be processed*/

run;

data _null_;set dir;call symput('filename',left(fn));run;%put &filename;

FILENAME statement with Macro%let path = \\file path;

data files_1; path = "&path";rc = filename("fref", path);if rc = 0 then did = dopen("fref"); else putlog "Error: cant open &path" ;

if did<=0 then putlog "Error: cant open &path" ;else do i = 1 to dnum(did);

filename = dread(did, i); fid = mopen(did, filename); if fid > 0 then do;

kb=finfo(fid,'File Size (bytes)')/1024; crdate=finfo(fid,'Create Time');moddate=finfo(fid,'Last Modified');if kb > 0 then output;

end;end;keep filename kb crdate moddate path;

run;

data files_2;set files_1; format created_on modified_on date9.;created_on=input(crdate,anydtdte22.);modified_on=input(moddate,anydtdte22.); keep filename path created_on modified_on;

run;

data _null_;set files_2;if intck('day', modified_on, today()) = 1 thencall symput('filename',filename);

run;

%put &filename;

Issues: 64 bit vs 32 bit EG version before 5.1 is 32bit only application

If running EG under 64bit Windows, CreateObject("SASEGObjectModel.Application.4.2") will prompt error

Solution: use C:\Windows\SysWOW64\cscript.exe instead

SysCommand.dll may not compatible with EG installed under 64bit Windows

Compiled working version, contact me if you need

Solution: Use modified version or run batch (*.bat) from the task

Issues: Dos pipe output

Reference Creating Custom Add-In Tasks for SAS Enterprise

Guide (http://support.sas.com/documentation/onlinedoc/guide/customtasks/index.htm)

The SAS Dummy - A SAS® blog for the rest of us

by Chris Hemedinger

(http://blogs.sas.com/content/sasdummy/)

SAS Product Documentation

(http://support.sas.com/documentation)