ODI Waiting for a File and Processing It

8/13/2019 ODI Waiting for a File and Processing It

1/13

Waiting for a file and processing it

One of the perennial issues developers have is that they want to wait for incoming files ina directory, and when those files arrive, to process each one in a scenario. ODI providestools to enable you to wait for files (OdiWaitForFile tool) and to take an action after that.The complexity comes when you want to wait for files (many of which may come in at atime) and process each individually. In order to address this issue, we need to add a littlecunning to our approach.

The starting point is to create a new ODI procedure, with a set of options:

This procedure will be used in the package which we create. The package will have threesteps:

An OdiWaitForFile step, which polls the appropriate directory, waiting for files to be ready to process. (Note that it is best practice when moving files, especiallywhen receiving files transmitted using FTP to actually move two files, the file youwant and an associated marker file, which is moved AFTER the transmit of the

actual file. This technique gets round the problem caused by files created with aslow write being processed before they are ready. The marker file may containthe name of the file you want, or be named similarly, or only be transmitted oncompletion of all files in the transmission)

Process Waiting Files is the procedure we create to be able to deal with thefile(s) which need to be processed.

A Start Self step , which initiates the execution of the whole Wait and process package. (the reason we dont just loop is so that we can see that task has run,and potentially purge the log of completed executions)


2/13

In the Package this image shows the OdiFileWait parameters. In this case I am waitingon the d:/Temp directory for all files with a .zip extension. I have told ODI not to doanything with the file ( Action: NONE ) although it may be an idea to move the file to aprocessing directory. In this case, I have not used any of the other options, where I canfor instance specify the number of files to wait for (I might want to process batches).


3/13

When ODI has detected files which match the criteria specified, it will move on to thenext step in the process, which is my procedure to process the incoming files. This

process has the set of parameters shown below, which I need to fill in:

Of the options shown on this one, the PATTERN is for the files that we actually want to process not those we are

waiting on (which may be different). SCEN_NAME is the name of the scenario to be executed for each of the files SYNC_MODE is 1 for Synchronous and 2 for Asynchronous. Be careful with

this one, as if you execute asynchronously, you will get as many concurrentexecutions of the scenario as there are files. If the tasks within the scenario have

not been specifically modified to allow for concurrent execution, you may have problems. (by default for instance all the temporary tables for a particularinterfaces will use the same name)

INCOMING_DIRECTORY is where the files for processing are located. Thiswill be the same as the first step, unless you chose the MOVE option, to move thefiles to a separate directory


4/13

The last step I have put in the package is the execution of itself, asynchronously. This isusing the SESS_NAME as the name of the scenario to start. This may not be correct ifthe original scenario was started using a NAME= parameter.


5/13

Procedure DetailThis is the detail of the procedure I created. As you can see it has only four steps. Itcould be done in less, but this makes the process more readable.

Figure 1 The steps of the file processing procedure


6/13

The first of the steps is the one to create the temporary table I will use to store the namesof the file to process. I have put this table in the Sunopsis Memory Engine, an in-memory database functionality which is part of the ODI product.In this first step, as I only have one command, I put it in the Command on Target tab.The first part of this gets us the JDBC connection we will use, and to get the parameters

of that, on the Command on Source tab, I have set the parameters of the database Iwish to use (see the following image) As the table will be created in memory in theexecution agent, and it should be disposed of on completion of the session, there should

be little chance of this taking up too much memory. I have also created it with thesessionid as well as the filename, in case.

Figure 2 Command on target for the create table step of the procedure


7/13

Figure 3Command on Source for the create table step of the procedure


8/13

Next is the step to retrieve the list of files, and insert them into the newly created table (inmemory). Here we are using some native functionality of the Jython scriptingenvironment. The glob.glob(filepattern) will return a collection of the names, which wecan then use to insert into the database with the INSERT statements.

Figure 4Command on Source for the Retrieve file list step of the procedure


9/13

Command on Source for the step which retrieves the file names from the table. Notethat we set the Technology and Schema here to match the memory engine, pre-defined inTopology.

Figure 5 Execute Scenario for each file step "Command on Source"


10/13

For the Command on Target we execute the OdiTool command OdiStartScen oncefor each of the files retrieved on the Command on Source. Note that to get it tosubstitute the value of FILENAME in the resultset from the source command, we use the# prefix. Note also that the name of the variable I am passining in to each of thescenarios is here set as ORACLE.FileToBeProcessed. This implies that I have a

project with a code ORACLE, and the variable is called FileToBeProcessed. Itmight be better to use a Global Variable, to eliminate the need to edit this in the procedure, and a variable FileToBeProcessed needs to be created. This does supposethat the scenario you are starting DECLAREs this variable, which may then be used inthe scenario including in resource names (file names) etc

Figure 6 Execute Scenario for each file step "Command on Target"


11/13

The last Step in the procedure is just a tidy-up step, to drop the table I created earlier.As there is only one command, this goes on the Command on Target tab.

Figure 7 Last step, Drop Table, Clean up after yourself


12/13

Appendix 1: Getting the value of parameters passed to ascenarioOne last extra little tidbit which may be useful: if you are passing variables into ascenario and want to know that those variables have been set, by default these are notshown in the log. A couple of workarounds exist to get that information:

1) Put a tool step in your package, something like an OdiSleep, and modify the codeto be executed as illustrated:

The result of which will show in the log as follows:


13/13

2) The second option is to do a similar thing, but to put the code into a procedure asfollows:

As you can see, here I simply made a Java BeanShell step and put the commented (/**/) code into there, so it is ignored by the interpreter.

ODI Waiting for a File and Processing It

Documents

Transcript of ODI Waiting for a File and Processing It