Jon Paris

Jon.Paris @ Partner400.comwww.Partner400.comwww.SystemiDeveloper.com

Simplify RPG ApplicationsUsing SPECIAL Files

1

NotesJon Paris is co-founders of Partner400, a firm specializing in customized education and mentoring services for IBM i (AS/400, System i, iSeries, etc.) developers.

Jon's career in IT spans 40+ years including a 12 year period with IBM's Toronto Laboratory.

Together with his partner Susan Gantner, Jon devotes his time to educating developers on techniques and technologies to extend and modernize their applications and development environments. Together Jon and Susan author regular technical articles for the IBM publication, IBM Systems Magazine, IBM i edition, and the companion electronic newsletter, IBM i EXTRA. You may view articles in current and past issues and/or subscribe to the free newsletter at: www.IBMSystemsMag.com.

Jon and Susan also write a weekly blog on Things "i" - and indeed anything else that takes their fancy. You can find the blog here: ibmsystemsmag.blogs.com/idevelop/

Feel free to contact the author at: Jon.Paris @ Partner400.com

2

AgendaWhat is a SPECIAL file• And why should you care about them

Examples of SPECIAL file usage• Reading IFS directory and file names• "Printing" to the Web

! A simple conversion of a program described print file to output to the web

• Running QShell and PASE commandsLimitations of SPECIAL files• And some additional thoughts on working round them

The future• How IBM's new RPG Open Access for RPG may change the game

3

NotesImagine if ...

You could use simple READ. WRITE, etc. operations to process files in the IFS - Or read through IFS directory structures - Or produce web pages - Or create Excel spreadsheets - Or ...

IBM hasn’t actually added these capabilities to RPG, but you can. The programs I am presenting here, when used as RPG SPECIAL files can provide exactly this kind of capability. This is a powerful approach because it removes the necessity of having all the RPG programmers in the shop from having to understand the mechanics of reading (say) IFS directory entries before they can perform such processing. ONly the person who writes the SPECIAL file program requires that knowledge. The others simply have to know how to specify they want to use this "file" and then use regular READ/WRITE operations to read and write the "records".The way a SPECIAL file works is really straightforward. Whenever the program executes an I/O operation against the file, the request is passed to the SPECIAL file program for action. It will first receive a request to open the file. Once the file is open, it will receive a series of I/O requests. It will respond to each request by reading or writing an entry just as if it was handling a database.This technique can be used in a variety of ways - during the course of the session I will discuss just a few and try to get you thinking about other ways in which you might make your own programs SPECIAL.

Of course SPECIAL files have limits, and IBM's V7 Open Access for RPG offers ways to circumvent them. We'll be taking a very brief look at how this works later in the session.

4

What is a SPECIAL File?A program that acts as an I/O device• Allows conventional RPG I/O operations to be used against devices or

file types not natively supported by RPG• e.g. IFS files, web browsers, web services, e-mail, etc.

Every I/O operation results in a call to the program• It is passed a parameter identifying the operation required

! e.g. Open, Read, Write, Close ...

• Two status reporting fields! To report errors, end-of-file, etc.

• And a record buffer ! On a write operation it contains the data to be written! On a read it is where the program will place the data required

• At your discretion, additional parameters can be passed! To provide perhaps the URL for the web service to use. ! Or the full path name of an IFS file - or ...

5

NotesThe standard parameters passed to the SPECIAL file program are:Option code - This identifies the type of operation being requested:

C - Close the fileD - Delete the record F - Force the end of fileO - Open the file R - Read a record U - Update the last record read W - Write the record (Note that W(rite) is the code passed for EXCEPT as well as WRITE)

Status - The program sets a value in this field to indicate the success / failure of the operation. The values are:

0 - Normal return. The requested action was processed1 - End of file - no record is returned. 2 - Error condition exists.

Error - If the program indicates an error by setting Status to "2" it can place a 5 digit code in this field to indicate the exact cause of the errorRecord Area - On an output operation this will contain the record data. On an Input operation it is the area into which the program should place the requested information.

In addition to the standard set of parameters there can be any number of optional user-defined parameters passed. More on this in a moment.

6

FQPRINT O F 132 Special PgmName('SPECIALC')

F PList(ExtraParms): : : // Dummy subroutine to contain PList(s)C PLists BegSr

C ExtraParms PList

C Parm ExceptType

C EndSr

F-Spec Keywords for SPECIAL FilesFile type of SPECIAL requires the PGMNAME keyword• Normal program name rules apply

Additional parameters can be supplied in the form of a PLIST• Identified by the F-spec PLIST keyword

You never actually define the basic four parms in the "using" program• The compiler does that for you

! But you must code them in the SPECIAL program as you'll see in a minute

7

// Procedure Interface used instead of *Entry PList D SPECIALC PI // First four are the standard parameter definitions // passed for all SPECIAL files from user program D action 1a D status 1a D error 5s 0 D data 132a // This is the extra parameter specified by the user program D exceptType 1a

The Resulting ParametersHere's the SPECIAL program’s Procedure Interface (PI)• In this case it receives five parameters

! The first four are standard and all SPECIAL file programs receive them

The first parameter (action) tells us what action is being requested• Note your program will always receive a request to open the file

! Even in cases where there’s no explicit OPEN coded in your RPG program

The second and third are for error and end-of-file reporting• We won't go into detail here - see the earlier notes page

The fourth is used for the record dataThe fifth is the additional parameter supplied via our user program’s PLIST

8

What Operations Are Supported ?Most operations related to sequential files are supported• However SPECIAL files cannot be specified as Input with Add

! You must use C(ombined) instead! Nobody seems to know why this limitation was introduced

! Note that while EXCEPT is not listed in the RPG manual it is supported as it is translated by RPG into one or more WRITE operations

File Type Operations Supported

Combined (C) READ, WRITE, OPEN, CLOSE, FEOD

Input (I) READ, OPEN, CLOSE, FEOD

Output (O) WRITE, OPEN, CLOSE, FEOD (and EXCEPT)

Update (U) READ, UPDATE, DELETE, OPEN, CLOSE, FEOD

9

The Basic MechanicsWe will start with a basic skeleton program example• It will allows us to see the basic process in action• And demonstrate the effect of RPG's automatic file open/close

The program that uses our SPECIAL file will simply read until EOF• It will display the content of all the records it reads

! And output a message when it detects EOF

• Once EOF is reached it will set on LR and return

The SPECIAL file we are using will do the following• When asked to "open" the file it will display an appropriate message• It will then fake out the reading of three records

! Placing text including the count in the record buffer• When asked to read a fourth record it signals that it has reached EOF• When asked to "close" the file it outputs a message

10

FDemoSpFileIF F 40 SPECIAL PgmName('DEMOSPFILE')

D record DS D data 40a

/Free

Dou %EOF(DemoSpFile); Read DemoSpFile record;

If not %EOF(DemoSpFile); dsply ('Read data = ' + data); // Display the record Else; dsply ('End of file reached'); EndIf;

EndDo;

*InLR = *On; /End-free

Our Simple "User" Program100% Standard RPG• As you can see it does little except read the file

! And displays the details of the record readIn this example the file is program described• And uses a DS to receive the record to avoid coding I-specs

Names the SPECIAL file program

Nothing different here - just an ordinary read loop

11

// Prototype keeps the compiler happy - not needed in V7 - hooray! D DemoSpFile Pr ExtPGM('DEMOSPFILE') D requestedAction... D 1a D statusFlag 1a D errorCode 5s 0 D recordData 40a

// Procedure Interface used instead of *Entry PList D DemoSpFile PI // Four standard parameters passed from the RPG program D action 1a D status 1a D error 5s 0 D recordData 40a

// Constants for SPECIAL file operation requests D OPEN C 'O' D READ C 'R' D CLOSE C 'C' // And for the status values to be returned D EOF C '1' D ERR C '2' D OK C '0'

D count s 5i 0 Inz

SPECIAL Program DEMOSPFILE - Part 1The D-specs for the SPECIAL file program• The constants would normally be in a /COPY source

12

/FREE Select;

When action = READ; count = count + 1;

If count

FDirEntriesIF F 732 SPECIAL PgmName('DIRREADER') F PList(dirReaderParm) F UsrOpn

C PListDummy BegSr C dirReaderParm PList C Parm directoryName C EndSr

Example 1: A File to Read IFS DirectoriesRPG has no direct op-code access to IFS files• A SPECIAL file can fulfill that function

! This particular program is described in "Get More from RPG"- www.ibmsystemsmag.com/ibmi/enewsletterexclusive/32135p1.aspx

This version uses a program described file• Defined as 732 characters long

! We will look at the record layout momentarily

Note the additional parameter used to pass the directory name• What other information might you pass?

! How about a character string for a wildcard search?! A flag to indicate whether the program should recurse into lower level directories?

USROPN is not compulsory. See notes page

15

NotesThe file type is specified as SPECIAL. That tells the compiler that a program will be used to handle the file processing. The PgmName keyword identifies the actual program used; ours is called DIRREADER. Because there’s no equivalent to an OVRDBF command to allow us to specify the actual directory that we want to read, we need another way of passing that information to the SPECIAL file program. That’s the purpose of the PList keyword. It names the "extra" parameter list used to supply the directory name.Because you’re forced to use a PLIST, you’re also forced to use fixed-form coding. We normally do all of our coding in /Free, so we’ve buried the PLIST at the bottom of the program in a dummy subprocedure to keep it out of the way and preserve the aesthetics of our beautiful code!For this particular program, only one additional parameter (the name of the directory to be opened) is needed. It’ll actually become the fifth parameter to the SPECIAL file program, since it will be added to the standard group of four parameters that are always passed. If you need additional parameters, they’d simply be added to this PLIST.

Note that in this particular example, I chose to code USROPN to simplify the detection and reporting of any open failures. It also has the advantage that I can close and reopen the file to read through a different directory. It also makes it easier to see what is going on when debugging.

16

Example 1: Main Program Flow

+d directoryData DS 732+d type 12a+d name 640a Varying+d created z+d accessed z+d modified z++d DIRECTORY c '*DIR'+d FILE c '*FILE'+d SYMLINK c '*SYM'

/Free Open(E) DirEntries; If %Error; message = 'Directory: ' + directoryName; // Report error and quit Dsply message; *InLR = *On; Return; EndIf;

Dou %EOF(DirEntries); Read(E) DirEntries directoryData; If ( not %Error ) AND ( not %EOF(DirEntries) ); shortName = name; except main; ElseIf %Error; message = directoryName; // Report (truncated) file name and quit Dsply message; Leave; // Exit read loop EndIf; EndDo;

Record layout /COPY'd into the program

Main processing loop

17

NotesAt the time I wrote this program I was convinced that SPECIAL files could not be externally described. I have since been disavowed of that foolishness, but because of that belief, I chose to use a /COPY to provide the DS directoryData which is used as the record layout. This layout is included in the member DIRDATA. I used a /COPY because that's an easy way to ensure that the data definitions used in the main program and in the SPECIAL file program match. By specifying this DS as the result field on the READ operation, I avoided having to code any input specifications.

The full source files for this example can be found here:www.partner400.com/Examples/SPECIALIFS.html

A better version which uses external descriptions and passes the INFDS to the SPECIAL file is described in the May edition of IBM Systems Magazine's Extra and you can find the sources here:

www.partner400.com/Examples/SPECIALIFS2.html

18

When action = READ; p_dirEnt = ReadDir( pdir ); // Read next directory entry If p_dirEnt *Null; // If pointer is valid we have a record recordData.name = %Subst( dE_name: 1: dE_nameLen ); // Use the stat() function to retrieve information for the record If Stat( recordData.name: p_statDS ) < 0; // Check for failure dirName = 'Name: ' + recordData.name; // Identify failing name ExSr ReportError; Else; // Create required entries in the record buffer ... status = OK; EndIf; Else; // otherwise we're at the end of the "File" status = EOF; Endif; When action = OPEN; pdir = OpenDir( dirName ); If pdir = *Null; // If pointer is null then open failed ExSr ReportError; // Obtain error details and reports failure EndIf; When action = CLOSE; CloseDir( pdir ); EndSl;

Example 1: SPECIAL File Main LogicThis example makes use of the IFS Directory processing APIs• See the notes page for a link to an explanation of how they work

19

NotesIf you want to understand all the gory details of how directory processing works, take a look at "Walking Through an IFS Directory" which you can find at http://www.ibmsystemsmag.com/ibmi/8681p1.aspx. It is a relatively straightforward process, but after studying it we think you'll agree that using a SPECIAL file make sit a lot simpler.

20

Example 2: Simple Print to the Web Conversion

Advantages:• Minimal changes required to the source code • A generic program could be used for multiple reports• Can be used for both Externally and Program described files

Disadvantages:• Only really suitable for low volume output

! Page-at-a-time processing is complicated

• Does not offer many formatting opportunitiesI'm using CGIDEV2 in this example

The browser output shown here was produced using this simple technique.

Coding details in a moment

21

NotesMy example uses CGIDEV2 - if you are unfamiliar with this free web tool you can download the software and find out more at www.easy400.net. My partner Susan Gantner and myself have written about it many times, as have a number of others - just Google CGIDEV2 and you'll find everything you need.Despite the limited amount of formatting that this simple approach provides, a number of people have worked with the technique to produce spectacular results. Notably Trevor Seeney has achieved wonders - check for his articles and presentations on the subject. He has even gone as far as to use SPECIAL files with iPhones and other mobile devices, although admittedly with techniques a little more complex than these.

22

FQPRINT O F 132 PRINTER OFLIND(*INOF)

// Becomes

FQPRINT O F 132 SPECIAL PgmName('SPECIALA')

: :

OQPRINT E Headings 2 06: : O E Headings 2: :O E Detail 1

Example 2: Modifying the Report ProgramRelatively few modifications are needed• The source lines below are the only ones to be changed

! Of course changing a comment or two is a good idea too!

• In this example I also deleted an EXCEPT that handled page overflowOnce the changes were made there was nothing much left to do• Just Compile and Test

! But since the source has hardly changed there's not a lot to test!

Remove Spacing controls

23

NotesNote how few changes are involved here. If we are going to implement paging logic, it will be done within the SPECIAL file program itself so the overflow logic is removed. In fact we have to remove it since the keywords involved are only valid on PRINTER files - and although logically that is what we still have, we have to change the name to SPECIAL to make the whole thing work.

As we will discuss later, it is possible to either add an additional parameter to specify the type of line being output - if that is important to you. Alternatively you could add an extra field to the beginning of each print line and use that or ... In fact if you are using an externally described printer file you can use the record format name to control the output.

24

Content-type: text/html

Payroll Report

/%LINEDATA%/

Example 2: HTML SkeletonCGIDEV2 uses a "mail merge" approach to building web pages• Markers such as /%LINEDATA%/ are replaced at run time• In our case the replacement data will be the "print" line

The skeleton into records - known as sections in CGIDEV2ese• Each section begins with

Example 2: Processing the Requests

// Process requested operation for file (Open, Write or Close) Select;

// Open - Load external HTML using supplied file name, & write headers When Action = Open_File GetHtml('HTMLSRC':'PARTNER400': 'PAYRPTSHC': '

D UNIXCMD PR ExtPgm('UNIXCMD') D option 1a const D status 1a D error 5s 0 D record 65535a options(*varsize) D cmd 5000a const D type 1a const options(*nopass) D recordLen 5p 0 const options(*nopass)

Example 3: Scott Klement's UNIXCMD Introduced in late 2009• Latest version can be downloaded from www.ScottKlement.com/unixcmd

! LInks to relevant articles are on the same page

Allows an RPG program to issue commands and process results• Works with both QShell and PASE

Challenge is knowing the available commands and their usage• And how "pipes" can connect the output of one to the input of another

The first four parameters are the standard ones• The fifth is the command to be executed and is compulsory• Sixth is used to indicate that command must run in PASE (Default is QShell)• Seventh is used to supply the record length

29

NotesBelow is an extract from Scott's web site describing the basic purpose of the tool. The great thing about it is that you don't have to understand all of the the ins and outs of how to interface with QSHELL or PASE in order to be able to use the feature since it is all implemented as a SPECIAL file. All of the mechanics - other than knowing the required unix-style command - are hidden from you. There are many interesting and useful tools available in these environments, my personal favorite being PHP. PHP runs in PASE and UNIXCMD provides a great method for invoking PHP functions from within an RPG program.

From www.ScottKlement.com/unixcmd/"This is a tool to help simplify the process of running QShell or PASE commands from RPG or CL. For RPG, the tool implements a SPECIAL file that you open and connect to a QShell or PASE command that you want to run. When you read the file, you're reading the output of the command. When you write to the file, you're writing data to the command. For CL, it provides OPNPIPE, RCVPIPE, SNDPIPE and CLOPIPE commands that open the Unix and send/receive data from it, very similar to the way you use files in CL."

Scott has written a number of articles describing this utility they are:

Run QShell/PASE Commands with RPG SPECIAL Files (systeminetwork.com/article/run-qshellpase-commands-rpg-special-files)

It's Easy to Run PHP, QShell, and PASE Tools from RPG(systeminetwork.com/article/its-easy-run-php-qshell-and-pase-tools-rpg)

It's Easy to Run PHP, QShell, and PASE Tools from CL, Too!(systeminetwork.com/article/its-easy-run-php-qshell-and-pase-tools-cl-too)

30

FUNIX IF F 1000 SPECIAL PGMNAME('UNIXCMD') F PLIST(UNIXPARM) USROPN FQSYSPRT O F 132 PRINTER

D cmd s 5000a

D record ds 1000 D outrec ds 132

C UNIXPARM PLIST C PARM CMD /free cmd = 'cd /Partner400; ls'; open UNIX;

read UNIX record; dow not %eof(UNIX); outrec = record; write QSYSPRT outrec; read UNIX record; enddo;

close UNIX; *inlr = *on; /end-free

Example 3: Simple Command UsageThis is based on Scott's Example4 • Very similar in effect to our first example but only reports files

! There are options to the ls command that could be used to list directories

Two commands are issued - separated by a semi-colon

31

Notes

This is a list of the examples that are currently part of the UNIXCMD download package.

! Example1 - Lists contents of an IFS directory! Example2 - Writes text to a named IFS file! Example3 - Write ASCII and reads converted EBCIDIC! Example4 - Almost identical to 1 but uses Result field I/O into a Data Structure to avoid using I specs.! Example5 - Same as 4 but runs in PASE and not QShell! Example6 - Uses an SSH connection to sign on and retrieve data from a remote system. It is Scott's system

so it will just tell you to go away - but you can see how it hangs together! Example7 - Invokes PHP and retrieves the program's response

They are explained in the referenced articles.

32

/free cmd = 'PATH=$PATH:/usr/local/Zend/Core/bin && + iconv -f 37 -t 819 | + php /www/zendcore/htdocs/testGeocoder.php'; open UNIX;

record = '1600 W Pennsylvania Av, Washington DC'; write UNIX record;

read UNIX record; lat = %dec(record: 11: 7);

read UNIX record; lon = %dec(record: 11: 7);

close UNIX;

dsply ('lat=' + %char(lat) + ' and lon=' + %char(lon)); *inlr = *on; /end-free

Example 3: Calling a PHP ScriptPHP is very popular on IBM i lately - running it from RPG can be very useful• UNIXCMD allows you to integrate PHP functions into your RPG

! The actual PHP script used accesses a web service and is shown on the notes page

This might not be a good approach if a large number of calls were needed• The PHP startup time could cause performance problems• Although other than on the first call performance seems to be pretty good

This line (iconv etc.) may not be needed on your system

33

NotesThis is the php script used in the example - very simple

The "\n" characters in the script add a newline to the end of the text. It is the newline rather than the two "echo" statements that cause the two records to be seen by the UNIXCMD file. In other words:

! echo $result [0]->lat . "\n" . $result [0]->long . "\n";would produce identical results.

As I found out (the hard way) UNIX cmd will trim blanks and add a line feed character to the end of your input - but your input is considered to have a default length of 1,000 bytes - even if you have specified a shorter length on the F-spec. This is because the record length is not part of the data made available to the SPECIAL file program - so UNIXCMD does not know the actual length - unless you tell it. So you may need to do one or both of:

! Specify the fifth parameter so that UNIXCMD knows how long the record actually is! Use a trim() within your PHP code to avoid having a whole bunch of spaces and the trailing LF be considered

part of the data.

34

Limitations of SPECIAL Files?There is no information passed to the "file"• Just the operation request and the data buffer• You can get a little more data by passing the INFDS as a parameter

! But it is limited due to RPG400 legacy

The SPECIAL file program has very limited ability to report errors• It cannot set information into the feedback sections of the INFDS

It has no architected mechanism for recalling state information• We'll look in a moment at why this is important

Only programs can be used - not subprocedures• Not a huge issue but can impact performance in high-volume usage

SPECIAL files only allow operations that relate to sequential devices• So CHAIN and other keyed operations cannot be used

SPECIAL files cannot handle null capable fields• Something that increases in importance as more SQL is used

35

Why Is Maintaining State Important ?What if a SPECIAL file program is used twice in the same program ?• For two different files that are going to be processed at the same time

If you don't maintain the state of the files you will have a mess• You need to maintain the cursor position for each file

! And to differentiate the read requests as to which was for which file

These are the kind of thing we mean by "state"• We take them for granted with normal disk and printer I/O• But have to code them for ourselves when writing SPECIAL files

For an externally described file the record format _might_ work• But it is more likely that you will need to add additional parameters

36

How Does Open Access Resolve The Issues?No information passed to the "file"• OA can pass you the names and datatypes of the fields in the record

! And a whole bunch more information besides

Very limited ability to report errors• An OA handler can set the status code and I/O feedback

! And can send escape messages to ensure that diagnostic messages are in the log

No architected mechanism for recalling state information• OA has an architected state handling mechanism

Only programs can be used - not subprocedures• Programs or Service Program routines can be used by OA

Restricted to sequential device operations• All device types and operation codes can be used

! For keyed operations key data is supplied

SPECIAL files cannot handle null capable fields• Null flags are passed for any null capable fields and/or keys

37

That's All Folks - Any Questions ?

?

38