Release 0.8.22 Christopher Singley - Read the Docs

ofxtools DocumentationRelease 0.8.22

Christopher Singley

Nov 20, 2020

Contents:

1 Installing ofxtools 31.1 Installation dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Standard installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Bleeding edge installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Developer’s installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Extra goodies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Downloading OFX Data With ofxget 52.1 Locating ofxget . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Using ofxget - TL;DR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Storing ofxget passwords in the system keyring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 Using ofxget - in depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.5 Scanning for OFX connection formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Using OFXClient in Another Program 15

4 Parsing OFX Data 174.1 Deviations from the OFX specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5 Generating OFX 21

6 Using ofxtools with SQL 23

7 Contributing to ofxtools 27

8 Adding New OFX Messages 298.1 Request and Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298.2 Recurring Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368.3 Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408.4 Extending the Message Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

9 Additional Resources 459.1 More open-source OFX code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

10 What is it? 47

11 Where is it? 49

i

12 Installation Dependencies 51

ii

ofxtools Documentation, Release 0.8.22

ofxtools is a Python library for working with Open Financial Exchange (OFX) data - the standard format fordownloading financial information from banks and stockbrokers. OFX data is widely provided by financial institutionsso that their customers can import transactions into financial management software such as Quicken, Microsoft Money,or GnuCash.

If you want to download your transaction data outside of one of these programs - if you wish to develop a Pythonapplication to use this data - if you need to generate your own OFX-formatted data. . . ofxtools is for you!

Contents: 1


2 Contents:

CHAPTER 1

Installing ofxtools

You have a few options to install ofxtools. If you like, you can install it in a virtual environment, but sinceofxtools has no external dependencies, that doesn’t really gain you much.

A simpler option for keeping clutter out of your system Python site is the user install option, which is recommendedif only one system user needs the package (the normal situation).

1.1 Installation dependencies

You need to install Python 3 (at least version 3.6) in order to use ofxtools. It won’t work at all under Python 2.

In order to use the OFX client to download OFX files, your Python 3 installation needs to be able to validate SSLcertificates. Users of Mac OS X should heed the following note from the ReadMe.rtf included with the Pythoninstaller as of version 3.6:

This variant of Python 3.6 now includes its own private copy of OpenSSL 1.0.2. Unlike previous releases,the deprecated Apple-supplied OpenSSL libraries are no longer used. This also means that the trustcertificates in system and user keychains managed by the Keychain Access application and the securitycommand line utility are no longer used as defaults by the Python ssl module. For 3.6.0, a sample com-mand script is included in /Applications/Python 3.6 to install a curated bundle of default root certificatesfrom the third-party certifi package.

To facilitate keeping this important security package up to date, it’s advisable for Mac users to instead employ pip:

$ pip install certifi

1.2 Standard installation

If you just want to use the ofxtools library, and you don’t have any special needs, you should probably install themost recent release on PyPI:

3

https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments

https://pip.pypa.io/en/stable/user_guide/#user-installs

https://pypi.org/project/certifi/

https://pypi.python.org/pypi/ofxtools


$ pip install --user ofxtools

Or if you want to install it systemwide, as root just run:

$ pip install ofxtools

1.3 Bleeding edge installation

To install the most recent prerelease (which is where the magic happens, and also the bugs), you can download thecurrent master, unzip it, and install via the included setup file:

$ pip install --user .

1.4 Developer’s installation

If you want to hack on ofxtools, you should clone the source and install is in development mode:

$ git clone https://github.com/csingley/ofxtools.git$ cd ofxtools$ pip install -e .$ pip install -r ofxtools/requirements-development.txt

1.5 Extra goodies

In addition to the Python package, these methods will also install the ofxget script - a basic command line interfacefor downloading files from OFX servers. pip uninstall ofxtools will remove this script along with thepackage. Some financial institutions make you use their web application to generate OFX (or QFX) files that youcan download via your browser. If they give you a choice, prefer “OFX” or “Microsoft Money” format over “QFX”or “Quicken”.

Other financial institutions are good enough to offer you a server socket, to which ofxtools can connect anddownload OFX data for you.

4 Chapter 1. Installing ofxtools

https://github.com/csingley/ofxtools/archive/master.zip

https://setuptools.readthedocs.io/en/latest/setuptools.html?highlight=development%20mode#develop-deploy-the-project-source-in-development-mode

CHAPTER 2

Downloading OFX Data With ofxget

2.1 Locating ofxget

The ofxget shell script should have been installed by pip along with the ofxtools library. If the install locationisn’t already in your $PATH, you’ll likely want to add it.

User installation

• Mac: ~/Library/PythonX.Y/bin/ofxget

• Windows: AppData\Roaming\Python\PythonXY\Scripts\ofxget

• Linux/BSD/etc.: ~/.local/bin/ofxget

Site installation

• Mac: /Library/Frameworks/Python.framework/Versions/X.Y/bin/ofxget

• Windows: Good question; anybody know?

• Linux/BSD/etc.: /usr/local/bin/ofxget

Virtual environment installation

• </path/to/venv/root>/bin/ofxget

If all else fails, you can execute python -m ofxtools.scripts.ofxget, or directly run python </path/to/ofxtools>/scripts/ofxget.py. You can check where exactly that is by opening a Python interpreter andsaying:

>>> from ofxtools.scripts import ofxget>>> print(ofxget.__file__)

2.2 Using ofxget - TL;DR

Find your financial institution’s nickname:

5


$ ofxget list

If your financial institution is listed, then the quickest way to get your hands on some OFX data is to say:

$ ofxget stmt <server_nickname> -u <your_username> --all

Enter your password when prompted.

However, you really won’t want to set the --all option every time you download a statement; it’s very inefficient.Slightly more verbosely, you might say :

$ ofxget acctinfo <server_nickname> -u <your_username> --write$ ofxget stmt <server_nickname>

The first command requests a list of accounts and saves it to your config file along with your user name. This is in thenature of a first-time setup chore.

The second command is the kind of thing you’d run on a regular basis. It requests statements for each account listedin your config file for a given server nickname.

2.3 Storing ofxget passwords in the system keyring

Note: this feature is experimental. Expect bugs; kindly report them.

Rather than typing them in each time, you can securely store your passwords in the system keyring (if one is available)and have ofxget retrieve them for you. Examples of such keyring software include:

• Windows Credential Locker

• Mac Keychain

• Freedesktop Secret Service (used by GNOME et al.)

• KWallet (used by KDE)

To use these services, you will need to clutter up your nice clean ofxtools by installing the python-keyring package.

$ pip install --user keyring

Additionally, KDE users will need to install dbus-python. Note the recommendation in the python-keyring docsto install it systemwide via your package manager.

Once these dependencies have been satisfied, you can pass the --savepass option to ofxget anywhere it wants apassword, e.g.

$ ofxget acctinfo <server_nickname> -u <your_username> --write --savepass

That should set you up to download statements easily.

To overwrite an existing password, simply add the --savepass option again and you will be prompted for a newpassword.

To delete a password entirely, you’ll need to use your OS facilities for managing these passwords (they are storedunder “ofxtools”, with an entry for each server nickname).

6 Chapter 2. Downloading OFX Data With ofxget

https://pypi.org/project/keyring/

https://pypi.org/project/dbus-python/


2.4 Using ofxget - in depth

ofxget takes two positional arguments - request type (mandatory) and server nickname (optional) - along with abunch of optional keyword arguments.

See the --help for explanation of the script options.

Available request types (as indicated in the --help) are list, scan, prof, acctinfo, stmt, stmtend andtax1099. We’ll work through most of these in an example of bootstrapping a full configuration for AmericanExpress.

2.4.1 Basic connectivity: requesting an OFX profile

We must know the OFX server URL in order to connect at all. ofxtools contains a database of all US financialinstitutions listed on the OFX Home website that I could get to speak OFX with me. If you can’t find your bankin ofxget (or if you’re having a hard time configuring a connection), OFX Home should be your first stop. If youprefer, the OFX Blog also makes the same data available in a different format. Be sure to review user-posted commentson either site. You can also try the fine folks at GnuCash, who share the struggle.

OFX Home has a listing for AmEx, giving a URL plus the ORG/FID pair (i.e. <FI><ORG> and <FI><FID> inthe signon request.) This aggregate is optional per the OFX spec, and if your FI is running its own OFX server it isoptional - many major providers don’t need it to connect. However, Quicken always sends <FI>, so your bank mayrequire it anyway. AmEx appears to be one of these; its OFX server throws HTTP error 503 if you omit ORG/FID.

Using the connection information from OFX Home, first we will try to establish basic connectivity by requesting anOFX profile, which does not require authenticating a login.

$ ofxget prof --org AMEX --fid 3101 --url https://online.americanexpress.com/myca/→˓ofxdl/desktop/desktopDownload.do\?request_type\=nl_ofxdownload

This hairy beast of a command can be used for any arbitrary OFX server. If the server is already known to ofxget,then you can just use its nickname instead:

$ ofxget prof amex

Or, if the server is known to OFX Home, then you can just use its database ID (the end part of its institution page onOFX Home):

$ ofxget prof --ofxhome 424

Any of these work just fine, dumping a load of markup on the screen telling us what OFX services are available andsome parameters for accessing them.

If it doesn’t work, see below for a discussion of scanning version and format parameters.

2.4.2 Creating a configuration file

We probably don’t want to keep typing out multiline commands every time, so we’ll create a configuration file to storethese parameters for reuse.

The simplest way to accomplish this is just to tell ofxget to save the arguments you’ve passed on the command lineto the config file. To do that, append the “–write” option to your CLI invocation. You’ll also need to provide a servernickname.

$ ofxget prof myfi --write --org AMEX --fid 3101 --url https://online.americanexpress.→˓com/myca/ofxdl/desktop/desktopDownload.do\?request_type\=nl_ofxdownload

2.4. Using ofxget - in depth 7

http://www.ofxhome.com/


https://wiki.gnucash.org/wiki/OFX_Direct_Connect_Bank_Settings

http://www.ofxhome.com/index.php/institution/view/424

http://www.ofxhome.com/index.php/institution/view/424


If your server is up on OFX Home, this works as well:

ofxget prof myfi --ofxhome 424 --write

It’s also easy to write a configuration file manually in a text editor - it’s just the command line options in simple INIformat, with a server nicknames as section headers. You can find a sample at </path/to/ofxtools>/config/ofxget_example.cfg, including some hints in the comments.

The location of the the config file depends on the platform.

• Windows: <userhome>\AppData\Roaming\ofxtools\ofxget.cfg

• Mac: <userhome>/Library/Preferences/ofxtools/ofxget.cfg

• Linux/BSD/etc.: <userhome>/.config/ofxtools/ofxget.cfg

(Of course, these locations may differ if you have exported nondefault environment variables for APPDATA orXDG_CONFIG_HOME)

You can verify where precisely ofxget is looking for its configuration file by opening a Python interpreter andsaying:

>>> from ofxtools.scripts import ofxget>>> print(ofxget.USERCONFIGPATH)

Our configuration file will look like this:

# American Express[amex]url: https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_→˓type=nl_ofxdownloadorg: AMEXfid: 3101

Alternatively, since AmEx has working parameters listed on OFX Home, you could just use the OFX Home API tolook them up for each request. Using the OFX Home database id (at the end of the webpage URL), the config lookslike this:

# American Express[amex]ofxhome: 424

With either configuration, we can now use the provider nickname to make our connection more conveniently:

$ ofxget prof amex

2.4.3 Logging in and requesting account information

The next step is to log into the OFX server with our username & password, and get a list of accounts for which we candownload statements.

$ ofxget acctinfo amex --user <username>

After passing authentication, a successful result looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><?OFX OFXHEADER="200" VERSION="203" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID=→˓"e1259eaf-b54e-46de-be22-fe07a9172b79"?>

(continues on next page)



(continued from previous page)

<OFX><SIGNONMSGSRSV1>

<SONRS><STATUS>

<CODE>0</CODE><SEVERITY>INFO</SEVERITY><MESSAGE>Login successful</MESSAGE>

</STATUS><DTSERVER>20190430093324.000[-7:MST]</DTSERVER><LANGUAGE>ENG</LANGUAGE><FI>

<ORG>AMEX</ORG><FID>3101</FID>

</FI></SONRS>

</SIGNONMSGSRSV1><SIGNUPMSGSRSV1>

<ACCTINFOTRNRS><TRNUID>2a3cbf11-23da-4e77-9a55-2359caf82afe</TRNUID><STATUS>

<CODE>0</CODE><SEVERITY>INFO</SEVERITY>

</STATUS><ACCTINFORS>

<DTACCTUP>20190430093324.150[-7:MST]</DTACCTUP><ACCTINFO>

<CCACCTINFO><CCACCTFROM>

<ACCTID>888888888888888</ACCTID></CCACCTFROM><SUPTXDL>Y</SUPTXDL><XFERSRC>N</XFERSRC><XFERDEST>N</XFERDEST><SVCSTATUS>ACTIVE</SVCSTATUS>

</CCACCTINFO></ACCTINFO><ACCTINFO>

<CCACCTINFO><CCACCTFROM>

<ACCTID>999999999999999</ACCTID></CCACCTFROM><SUPTXDL>Y</SUPTXDL><XFERSRC>N</XFERSRC><XFERDEST>N</XFERDEST><SVCSTATUS>ACTIVE</SVCSTATUS>

</CCACCTINFO></ACCTINFO>

</ACCTINFORS></ACCTINFOTRNRS>

</SIGNUPMSGSRSV1></OFX>

(Indentation applied and Intuit proprietary extension tags removed to improve readability)

Within all that markup, the part we’re looking for is this:

2.4. Using ofxget - in depth 9


<CCACCTFROM><ACCTID>888888888888888</ACCTID></CCACCTFROM><CCACCTFROM><ACCTID>999999999999999</ACCTID></CCACCTFROM>

We have two credit card accounts, 888888888888888 and 999999999999999. We can request activity statements forthem like so:

$ ofxget stmt amex --user <username> --creditcard 888888888888888 --creditcard→˓999999999999999

Note that multiple accounts are specified by repeating the creditcard argument.

Of course, nobody wants to memorize and type out their account numbers, so we’ll go ahead and include this infor-mation in our ofxget.cfg:

# American Express[amex]url: https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_→˓type=nl_ofxdownloadorg: AMEXfid: 3101user: <username>creditcard: 888888888888888,999999999999999

Note that multiple accounts are specified as a comma-separated sequence.

To spare your eyes from looking through all that tag soup, you can just tell ofxget to download the ACCTINFOresponse and update your config file automatically:

$ ofxget acctinfo amex --user <username> --write

Alternatively, as touched on in the TL;DR - if you’re in a hurry, you can skip configuring which accounts you want,and instead just pass the --all argument:

$ ofxget stmt amex --user <username> --all

This tells ofxget to generate an ACCTINFO request as above, parse the response, and generate a STMT request foreach account listed therein. You might as well tack on a --write to save these parameters to your config file, so youdon’t have to do all that again next time.

2.4.4 Requesting statements

To rehash, a full statement request constructed entirely through the CLI looks like this:

$ export URL="https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.→˓do\?request_type\=nl_ofxdownload"$ ofxget stmt --url $URL --org AMEX --fid 3101 -u <username> -c 888888888888888 -c→˓999999999999999$ unset URL

This is for a credit card statement; for a bank statement you will also need to pass in --bankid (usually the bank’sABA routing number), and for a brokerage statement you will need to pass in --brokerid (usually the broker’sDNS domain).

Presumably you will have migrated most/all of these parameters to your config file as described above, so you caninstead just say this:


http://routingnumber.aba.com/default1.aspx


$ ofxget stmt amex

By default, a statement request asks for all transaction activity available from the server. To restrict the statement to acertain time period, we use the --start and --end arguments:

$ ofxget stmt amex --start 20140101 --end 20140630 > 2014-04_amex.ofx

Please note that the CLI accepts OFX-formatted dates (YYYYmmdd) rather than ISO-8601 (YYYY-mm-dd).

You can also pass‘‘–asof‘‘ to set the reporting date for balances and/or investment positions, although it tends to beignored for the latter.

There are additional statement options for omitting transactions, balances, and/or investment positions if you so desire,or including open securities orders as of the statement end date. See the --help for more details.

2.5 Scanning for OFX connection formats

What if you can’t make an OFX connection? Your bank isn’t in ofxtools; it isn’t at OFX Home; it is in OFX Homebut you can’t request a profile; or you’re trying to connect to a non-US institution and all you have is the URL.

Quicken hasn’t yet updated to OFX version 2, so your bank may require a lower protocol version in order to connect.The --version argument is used for this purpose.

As well, some financial institutions are picky about formatting. They may fail to parse OFXv1 that includes closingtags - the --unclosedelements argument comes in handy here. They may require that OFX requests either musthave or can’t have tags separated by newlines - try setting or unsetting the --prettyprint argument.

ofxget includes a scan command to help you discover these requirements. Here’s how to use it.

$ # E*Trade$ ofxget scan https://ofx.etrade.com/cgi-ofx/etradeofx[{"versions": [102], "formats": [{"pretty": false, "unclosedelements": true}, {"pretty→˓": false, "unclosedelements": false}]}, {"versions": [], "formats": []}, {→˓"chgpinfirst": false, "clientuidreq": false, "authtokenfirst": false,→˓"mfachallengefirst": false}]$ ofxget scan usaa[{"versions": [102, 151], "formats": [{"pretty": false, "unclosedelements": true}, {→˓"pretty": true, "unclosedelements": true}]}, {"versions": [200, 202], "formats": [{→˓"pretty": false}, {"pretty": true}]}, {"chgpinfirst": false, "clientuidreq": false,→˓"authtokenfirst": false, "mfachallengefirst": false}]$ ofxget scan vanguard[{"versions": [102, 103, 151, 160], "formats": [{"pretty": false, "unclosed_elements→˓": true}, {"pretty": true, "unclosed_elements": true}, {"pretty": true, "unclosed_→˓elements": false}]}, {"versions": [200, 201, 202, 203, 210, 211, 220], "formats": [{→˓"pretty": true}]}, {}]

(Try to exercise restraint with this command. Each invocation sends several dozen HTTP requests to the server; youcan get your IP throttled or blocked.)

The output shows configurations that worked.

E*Trade will only accept OFX version 1.0.2; they don’t care about newlines or closing tags.

USAA only accepts OFX versions 1.0.2, 1.5.1, 2.0.0, and 2.0.2. Version 1 needs to be old-school SGML - no closingtags. Newlines are optional. [Nota bene: in actual fact, while USAA accepts profile requests in OFX versions 2.0.0and 2.0.2, it only accepts statement requests in OFX versions 1.0.2 and 1.5.1. . . without closing tags, as indicatedabove].

2.5. Scanning for OFX connection formats 11



Vanguard is a little funkier. They accept all versions of OFX, but version 2 must have newlines. For version 1, youmust either insert newlines or leave element tags unclosed (or both). Closing tags will fail without newlines.

Copyng these configs into your ofxget.cfg manually, they would look like this:

[etrade]version = 102

[usaa]version = 151unclosedelements = true

[vanguard]version = 203pretty = true

The config for USAA is just an example to show the syntax; in reality you’d be better off just setting version =202.

As before, instead of manually editing the config file, you can also just ask ofxget to do it for you:

$ ofxget scan myfi --write --url https://ofx.mybank.com/download

2.5.1 Setting CLIENTUID

Returning to the JSON screen dump from the scan output - the last set of configs, after OFXv1 and OFXv2, containsinformation extracted from the SIGNONINFO in the profile. For the above institutions, this has contained nothinginteresting - all fields are false, except in the case of Vanguard, which is blank because they deviate from the OFXspec and require an authenticated login in order to return a profile. However, in some cases there’s some importantinformation in the SIGNONINFO.

$ ofxget scan bofa[{"versions": [102], "formats": [{"pretty": false, "unclosedelements": true}, {"pretty→˓": false, "unclosedelements": false}, {"pretty": true, "unclosedelements": true}, {→˓"pretty": true, "unclosedelements": false}]}, {"versions": [], "formats": []}, {→˓"chgpinfirst": false, "clientuidreq": true, "authtokenfirst": false,→˓"mfachallengefirst": false}]$ ofxget scan chase[{"versions": [], "formats": []}, {"versions": [200, 201, 202, 203, 210, 211, 220],→˓"formats": [{"pretty": false}, {"pretty": true}]}, {"chgpinfirst": false,→˓"clientuidreq": true, "authtokenfirst": false, "mfachallengefirst": false}]

Of the 3 JSON objects included in the output, here we are focused on the last (reformatted for readability):

{"chgpinfirst": false,"clientuidreq": true,"authtokenfirst": false,"mfachallengefirst": false

}

Both Chase and BofA have the CLIENTUIDREQ flag set, which means you’ll need to set clientuid (a valid UUIDv4 value) either from the command line or in your ofxget.cfg.

Not to worry! ofxget will automatically set a global default CLIENTUID for you if you ask it to --write aconfiguration. You can override this global default by setting a clientuid value under a server section in yourconfig file (in UUID4 format). More conveniently, you can just pass ofxget the --clientuid option, e.g.:


https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)

https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)


# The following generates a global default CLIENTUID$ ofxget scan chase --write# So does this$ ofxget prof chase --write# The following additionally generates a Chase-specific CLIENTUID$ ofxget acctinfo chase -u <username> --savepass --clientuid --write

Note: if you choose to use an FI-specific CLIENTUID, as in that last command, then you really want to be sure to passthe --write option in order to save it to your config file. It is important that the CLIENTUID be consistent acrosssessions.

After setting CLIENTUID, heed the <SONRS><STATUS> in the ACCTINFO response returned by Chase. It has anonzero <CODE> (indicating a problem), and the <MESSAGE> instructs you to verify your identity within 7 days. Todo this, you need to log into the bank’s website and perform some sort of verification process.

In Chase’s case, they want you to click a link in their secure messaging facility and enter a code sent via SMS/email.Other banks make you jump through slightly different hoops, but they usually involve logging into the bank’s websiteand performing some sort of high-hassle/low-security MFA routine for first-time access.

The master configs for OFX connection parameters are located in ofxtools/config/fi.cfg. If you get a newserver working, edit it there and submit a pull request to share it with others.

Many banks configure their servers to reject any connections that aren’t from Quicken. It’s usually safest to tell themyou’re a recent version of Quicken for Windows. ofxget does this by default, so you probably don’t need to worryabout it. If you do need to fiddle with it, use the appid and appver arguments, either from the command line or inyour ofxget.cfg.

We’ve also had some problems with FIs checking the User-Agent header in HTTP requests, so it’s been blankedout. If we can figure out what Quicken sends for User_Agent, it might be a good idea to spoof that as well.

What I’d really like to do is set up a packet sniffer on a PC running Quicken and pull down a current list of workingURLs. If that sounds like your idea of a fun time, drop me a line.

2.5. Scanning for OFX connection formats 13

CHAPTER 3

Using OFXClient in Another Program

To use within another program, first initialize an ofxtools.Client.OFXClient instance with the relevant con-nection parameters.

Using the configured OFXClient instance, make a request by calling the relevant method, e.g. OFXClient.request_statements(). Provide the password as the first positional argument; any remaining positional argu-ments are parsed as requests. Simple data containers for each statement type (StmtRq, CcStmtRq, InvStmtRq,StmtEndRq, CcStmtEndRq are provided for this purpose. Options follow as keyword arguments.

The method call therefore looks like this:

>>> import datetime; import ofxtools>>> from ofxtools.Client import OFXClient, StmtRq, CcStmtEndRq>>> client = OFXClient("https://ofx.chase.com", userid="MoMoney",... org="B1", fid="10898",... version=220, prettyprint=True,... bankid="111000614")>>> dtstart = datetime.datetime(2015, 1, 1, tzinfo=ofxtools.utils.UTC)>>> dtend = datetime.datetime(2015, 1, 31, tzinfo=ofxtools.utils.UTC)>>> s0 = StmtRq(acctid="1", accttype="CHECKING", dtstart=dtstart, dtend=dtend)>>> s1 = StmtRq(acctid="2", accttype="SAVINGS", dtstart=dtstart, dtend=dtend)>>> c0 = CcStmtEndRq(acctid="3", dtstart=dtstart, dtend=dtend)>>> response = client.request_statements("t0ps3kr1t", s0, s1, c0)

Other methods available:

• OFXClient.request_profile() - PROFRQ

• OFXClient.request_accounts()- ACCTINFORQ

• OFXClient.request_tax1099()- TAX1099RQ (still a WIP)

15


16 Chapter 3. Using OFXClient in Another Program

CHAPTER 4

Parsing OFX Data

ofxtools parses OFX messages in two steps.

The first step parses serialized OFX data into a Python data structure. The ofxtools.Parser.OFXTree parserparser subclasses xml.etree.ElementTree.ElementTree, and follows the ElementTree API:

In [1]: from ofxtools.Parser import OFXTreeIn [2]: parser = OFXTree()In [3]: with open('2015-09_amtd.ofx', 'rb') as f: # N.B. need to open file in binary→˓mode

...: parser.parse(f)

...:In [4]: parser.parse('2015-09_amtd.ofx') # Can also use filename directlyIn [5]: type(parser._root)Out[5]: xml.etree.ElementTree.ElementIn [6]: parser.find('.//STATUS')[:] # The full ElementTree API can be used,→˓including XPathOut[6]:[<Element 'CODE' at 0x7f4cc0aa4868>,<Element 'SEVERITY' at 0x7f4cc0aa49f8>,<Element 'MESSAGE' at 0x7f4cc0aa4d68>]

At this stage, you can modify the entire Element structure arbitrarily - move branches around the tree, add or deleteelements, rewrite tags and text, etc.

The second step of parsing converts the Element structure into a hierarchy of custom class instances, namely subclassesof ofxtools.models.base.Aggregate and ofxtools.models.Types.Element, following the OFXspecification’s classification of nodes into either containers (“Aggregates”) or data-bearing leaf nodes (“Elements”,not to be confused with xml.etree.ElementTree.Element). This parsing step validates the deserializedOFX data against the OFX spec, and performs type conversion (so that, for example, an OFX element specified as amonetary quantity will be converted to an instance of decimal.Decimal, while another element specified as date& time will be converted to an instance of datetime.datetime) The original structure of the OFX data hierarchyis preserved through this conversion.

17


In [7]: ofx = parser.convert()In [8]: type(ofx)Out[8]: ofxtools.models.ofx.OFX

Following the OFX spec , you can navigate the OFX hierarchy using normal Python dotted-attribute access, andstandard slice notation for lists.

In [9]: tx = ofx.invstmtmsgsrsv1[0].invstmtrs.invtranlist[-1]In [10]: tx.dtpostedOut[10]: datetime.datetime(2015, 9, 16, 17, 9, 48, tzinfo=<UTC>)In [11]: tx.trnamtOut[11]: Decimal('4.7')

While it’s obvious that INTRANLIST is a list, it’s perhaps less obvious that INVSTMTMSGSRSV1 is also a list, sinceOFX specifies that a single statement response wrapper can contain multiple statements.

It can get to be a real drag crawling all the way to the bottom of deeply-nested SGML hierarchies to extract thedata that you really want, so subclasses of ofxtools.models.base.Aggregate provide some navigationalconveniences.

First, each Aggregate provides proxy access to the attributes of its SubAggregates (and its sub-subaggregates,and so on). If the data you’re looking for is located in a.b.c.d.e.f, you can access it more simply as a.f. Thiswon’t work across lists, of course; you have to select an item from the list. So in this example, if c is a list type, youcould get your data from a.c[10].f.

Second, the upper-level Aggregates define some human-friendly aliases for the data structures you’re really lookingfor. Here’s an example.

In [12]: stmts = ofx.statements # All {``STMTRS``, ``CCSTMTRS``, `ÌNVSTMTRS``} in→˓the responseIn [13]: txs = stmts[0].transactions # The relevant ``*TRANLIST`Ìn [14]: acct = stmts[0].account # The relevant ``*ACCTFROM`Ìn [15]: balances = stmts[0].balances # `ÌNVBAL`` - use ``balance`` for bank→˓statement ``LEDGERBAL`Ìn [16]: securities = ofx.securities # ``SECLIST`Ìn [17]: len(securities)Out[17]: 5In [18]: len(txs)Out[18]: 6In [19]: tx = txs[-1]In [20]: tx.trnamtOut[20]: Decimal('4.7')In [21]: tx = txs[1]In [22]: type(tx)Out[22]: ofxtools.models.invest.transactions.TRANSFERIn [23]: tx.invtran.dttrade # Who wants to remember where to find the trade date?Out[23]: datetime.datetime(2015, 9, 8, 17, 14, 8, tzinfo=<UTC>)In [24]: tx.dttrade # That's more like itOut[24]: datetime.datetime(2015, 9, 8, 17, 14, 8, tzinfo=<UTC>)In [25]: tx.secid.uniqueid # Yet more layersOut[25]: '403829104'In [26]: tx.uniqueid # Flat access is less cognitively taxingOut[26]: '403829104'In [27]: tx.uniqueidtypeOut[27]: 'CUSIP'

The designers of the OFX spec did a good job avoiding name collisions. However you will need to remember that<UNIQUEID> always refers to securities; if you’re looking for a transaction unique identifier, you want tx.fitid

18 Chapter 4. Parsing OFX Data

http://www.ofx.net/downloads.html


(which is a shortcut to tx.invtran.fitid).

4.1 Deviations from the OFX specification

For handling multicurrency transactions per OFX section 5.2, Aggregates that can contain ORIGCURRENCY havean additional curtype attribute, which is not part of the OFX spec. curtype yields 'CURRENCY' if the moneyamounts have not been converted to the home currency, or yields 'ORIGCURRENCY' if they have been converted.

YIELD elements are renamed yld, and FROM elements are renamed frm, in order to avoid name collision withPython reserved keywords.

Proprietary OFX tags (e.g. <INTU.BANKID>) are stripped and dropped.

4.1. Deviations from the OFX specification 19


20 Chapter 4. Parsing OFX Data

CHAPTER 5

Generating OFX

Creating your own OFX requests or responses - as would be neeeded for, say, a Python-powered OFX server - is fairlystraightforward. However, you will need to be pretty familiar with the OFX spec. ofxtools validates individualnodes in the hierarchy, but doesn’t really do anything to verify compliant sequence order, for example. It doesn’tvalidate against a DTD. That is on you, friend.

Don’t forget to make datetimes timezone-aware.

As an example, we’ll create a trivial bank statement response. You can follow along in section 11.4.2.2 of the OFXspec.

In [1]: from ofxtools.models import *In [2]: from ofxtools.utils import UTCIn [3]: from decimal import DecimalIn [4]: from datetime import datetimeIn [5]: ledgerbal = LEDGERBAL(balamt=Decimal('150.65'),

...: dtasof=datetime(2015, 1, 1, tzinfo=UTC))In [6]: acctfrom = BANKACCTFROM(bankid='123456789',

...: acctid='23456', accttype='CHECKING') # OFX Section→˓11.3.1In [7]: stmtrs = STMTRS(curdef='USD', bankacctfrom=acctfrom,

...: ledgerbal=ledgerbal)

So far so good. Now to slather it in wrapper cruft and garnish with metadata.

In [8]: status = STATUS(code=0, severity='INFO')In [9]: stmttrnrs = STMTTRNRS(trnuid='5678', status=status, stmtrs=stmtrs)In [10]: bankmsgsrs = BANKMSGSRSV1(stmttrnrs)In [11]: fi = FI(org='Illuminati', fid='666') # Required for Quicken compatibilityIn [12]: sonrs = SONRS(status=status,

...: dtserver=datetime(2015, 1, 2, 17, tzinfo=UTC),

...: language='ENG', fi=fi)In [13]: signonmsgs = SIGNONMSGSRSV1(sonrs=sonrs)In [14]: ofx = OFX(signonmsgsrsv1=signonmsgs, bankmsgsrsv1=bankmsgsrs)

21


OK, that’s the complete OFX message body. To serialize it, we transform the ofxtools.models structure backinto an instance of xml.etree.ElementTree.ElementTree.

In [15]: import xml.etree.ElementTree as ETIn [16]: root = ofx.to_etree()In [17]: message = ET.tostring(root).decode()In [18]: messageOut[18]: '<OFX><SIGNONMSGSRSV1><SONRS><STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY>→˓</STATUS><DTSERVER>20150102170000</DTSERVER><LANGUAGE>ENG</LANGUAGE><FI><ORG>→˓Illuminati</ORG><FID>666</FID></FI></SONRS></SIGNONMSGSRSV1><BANKMSGSRSV1>→˓<STMTTRNRS><TRNUID>5678</TRNUID><STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY></→˓STATUS><STMTRS><CURDEF>USD</CURDEF><BANKACCTFROM><BANKID>123456789</BANKID><ACCTID>→˓23456</ACCTID><ACCTTYPE>CHECKING</ACCTTYPE></BANKACCTFROM><LEDGERBAL><BALAMT>150.65→˓</BALAMT><DTASOF>20150101000000</DTASOF></LEDGERBAL></STMTRS></STMTTRNRS></→˓BANKMSGSRSV1></OFX>'

One last step - we need to prepend an OFX header.

In [19]: from ofxtools.header import make_headerIn [20]: header = str(make_header(version=220))In [21]: headerOut[21]: '<?xml version="1.0" encoding="UTF-8" standalone="no"?>\r\n<?OFX OFXHEADER=→˓"200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="NONE"?>\r\n'In [22]: response = header + messageIn [23]: responseOut[23]: '<?xml version="1.0" encoding="UTF-8" standalone="no"?>\r\n<?OFX OFXHEADER=→˓"200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="NONE"?>\r\n<OFX>→˓<SIGNONMSGSRSV1><SONRS><STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY></STATUS>→˓<DTSERVER>20150102170000</DTSERVER><LANGUAGE>ENG</LANGUAGE><FI><ORG>Illuminati</ORG>→˓<FID>666</FID></FI></SONRS></SIGNONMSGSRSV1><BANKMSGSRSV1><STMTTRNRS><TRNUID>5678</→˓TRNUID><STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY></STATUS><STMTRS><CURDEF>USD</→˓CURDEF><BANKACCTFROM><BANKID>123456789</BANKID><ACCTID>23456</ACCTID><ACCTTYPE>→˓CHECKING</ACCTTYPE></BANKACCTFROM><LEDGERBAL><BALAMT>150.65</BALAMT><DTASOF>→˓20150101000000</DTASOF></LEDGERBAL></STMTRS></STMTTRNRS></BANKMSGSRSV1></OFX>'

Hand that to your HTTP server, and off you go.

22 Chapter 5. Generating OFX

CHAPTER 6

Using ofxtools with SQL

As of version 0.7, ofxtools no longer includes the ofxalchemy subpackage. The nature of its fundamentalarchitectural flaw is well expressed by this quote from a moderately reputable source:

SQLAlchemy supports class inheritance mapped to databases but it’s not really something that scaleswell to deep hierarchies. You can actually stretch this a lot by emphasizing single-table inheritance so thatyou aren’t hobbled with dozens of joins, but this seems like it is still a very deep hierarchy even for thatapproach.

What you need to do here is forget about your whole class hierarchy, and first design the database schema.You want to persist this data in a relational database. How? What do the tables look like? For any non-trivial application, this is where you need to design things from.

OFX is a poor fit for a relational data model, as is obvious to anyone who’s tried to work with its handling of onlinebill payees or securities reorganizations. You don’t really want to map that mess directly onto your database tables. . .the heart of any ORM-based application. A better approach is to decouple your SQL data model from OFX, whichwill also allow you better to handle other financial data formats.

It’s recommended to define your own ORM models based on your needs. Import OFX into Python using the mainofxtools.Parser.OFXTree parser, extract the relevant data, and feed it to your model classes. Something likethis:

from sqlalchemy.ext.declarative import declarative_basefrom sqlalchemy import (

Column, Integer, String, Text, DateTime, Numeric, ForeignKey, Enum,)from sqlalchemy.orm import (relationship, sessionmaker, )from sqlalchemy import create_engine

from ofxtools.models.i18n import CURRENCY_CODESfrom ofxtools.Client import (OFXClient, InvStmtRq, )from ofxtools.Parser import OFXTreefrom ofxtools.models.investment import (BUYSTOCK, SELLSTOCK)

# Data model(continues on next page)

23

https://groups.google.com/d/msg/sqlalchemy/a7xeKebSgTE/6m-qdR4BBgAJ



Base = declarative_base()

class Account(Base):id = Column(Integer, primary_key=True)brokerid = Column(String, nullable=False, unique=True)number = Column(String, nullable=False)name = Column(String)

class Security(Base):id = Column(Integer, primary_key=True)name = Column(String)ticker = Column(String)uniqueidtype = Column(String, nullable=False)uniqueid = Column(String, nullable=False)

class Transaction(Base):id = Column(Integer, primary_key=True)uniqueid = Column(String, nullable=False, unique=True)datetime = Column(DateTime, nullable=False)dtsettle = Column(DateTime)type = Column(Enum('returnofcapital', 'split', 'spinoff', 'transfer',

'trade', 'exercise', name='transaction_type'),nullable=False)

memo = Column(Text)currency = Column(Enum(*CURRENCY_CODES, name='transaction_currency'))cash = Column(Numeric)account_id = Column(Integer,

ForeignKey('account.id', onupdate='CASCADE'),nullable=False)

account = relationship('Account', foreign_keys=[account_id],backref='transactions')

security_id = Column(Integer,ForeignKey('security.id', onupdate='CASCADE'),nullable=False)

security = relationship('Security', foreign_keys=[security_id],backref='transactions')

units = Column(Numeric)

# Importclient = OFXClient('https://ofxs.ameritrade.com/cgi-bin/apps/OFX',

org='Ameritrade Technology Group', fid='AIS',brokerid='ameritrade.com')

stmtrq = InvStmtRq(acctid='999999999')response = client.request_statements(user='elmerfudd',

password='T0PS3CR3T',invstmtrqs=[stmtrq])

parser = OFXTree()parser.parse(response)ofx = parser.convert()

# Extractdef make_security(secinfo):


24 Chapter 6. Using ofxtools with SQL



return Security(name=secinfo.secname, ticker=secinfo.ticker,uniqueidtype=secinfo.uniqueidtype, uniqueid=secinfo.uniqueid)

securities = {(sec.uniqueidtype, sec.uniqueid): make_security(sec)for sec in ofx.securities}

stmt = ofx.statements[0]account = Account(brokerid=stmt.brokerid, number=stmt.acctid)

def make_trade(invtran):security = securities[(invtran.uniqueidtype, invtran.uniqueid)]return Transaction(

uniqueid=invtran.fitid, datetime=invtran.dttrade,dtsettle=invtran.dtsettle, type='trade', memo=invtran.memo,currency=invtran.currency, cash=invtran.total, account=account,security=security, units=invtran.units)

trades = [make_trade(tx) for tx in stmt.transactionsif isinstance(tx, (BUYSTOCK, SELLSTOCK))] # dispatch by model class

# Persistengine = create_engine('')Session = sessionmaker(bind=engine)session = Session()session.add(account)session.add_all(securities.values())session.add_all(trades)session.commit()

25


26 Chapter 6. Using ofxtools with SQL

CHAPTER 7

Contributing to ofxtools

To start hacking on the source, see the section entitled “Developer’s installation” under Installing ofxtools.

Make sure your changes haven’t broken anything by running the tests:

python `which nosetests` -dsv --with-coverage --cover-package ofxtools

Or even better, use make:

make test

After running one of the above commands, you can view a report of which parts of the code aren’t covered by tests:

coverage report -m

Poke around in the Makefile; there’s a few developer-friendly commands there.

Feel free to create pull requests on ofxtools repository on GitHub.

If you commit working tests for your code, you’ll be my favorite person.

27

https://help.github.com/articles/using-pull-requests/

https://github.com/csingley/ofxtools


28 Chapter 7. Contributing to ofxtools

CHAPTER 8

Adding New OFX Messages

As an example, I’ll document the implementation of bank fund transfers.

Download a copy of the OFXv2.03 spec. The messages we want to implement are located in Section 11.7. Since thesemessages appear in the hierarchy under BANKMSGSETV1, we’ll put them under ofxtools.models.bank.

8.1 Request and Response

In order to implement INTRARQ (the command clients use to request a funds transfer) we’ll first need to define anyaggregates it refers to - in this case, XFERINFO.

29

http://ofx.net/downloads/OFX2.0.3.zip


Here’s how we translate the spec info Python.

from ofxtools.models.base import Aggregate, SubAggregatefrom ofxtools.Types import String, Decimal, DateTime, OneOffrom ofxtools.models.bank.stmt import (

BANKACCTFROM, BANKACCTFROM, CCACCTFROM, CCACCTTO,)

class XFERINFO(Aggregate):""" OFX section 11.3.5 """

bankacctfrom = SubAggregate(BANKACCTFROM)ccacctfrom = SubAggregate(CCACCTFROM)bankacctto = SubAggregate(BANKACCTTO)ccacctto = SubAggregate(CCACCTTO)trnamt = Decimal(required=True)dtdue = DateTime()

requiredMutexes = [["bankacctfrom", "ccacctfrom"],["bankacctto", "ccacctto"],


30 Chapter 8. Adding New OFX Messages



]

We create a subclass of ofxtools.models.base.Aggregate, where the class name is the OFX tag in ALLCAPS. We define a class attribute for each tag that can appear under XFERINFO - the attribute names must be alllowercase.

Container aggregates are defined with ofx.models.base.SubAggregate; pass in the relevant model class.

Data-bearing elements are defined as a subclass of ofxtools.Types.Element - Decimal for TRNAMT andDateTime for DTDUE, as indicated by the spec. The spec prints TRNAMT in bold, which means it is required. Thisconstraint is enforced simply by passing required=True to the attribute definition.

The spec also states that either BANKACCTFROM or CCACCTFROM must appear in XFERINFO, as well as eitherBANKACCTTO or CCACCTTO. We can’t simply pass in required=True to the relevant class attributes - that wouldrequire all of them to appear in any valid XFERINFO instance, which is clearly not right. Instead of attribute-levelvalidation, these kinds of class-level constraints are enforced by separate class attributes.

In this case, we employ the awkwardly-named ofxtools.models.base.Aggregate.requiredMutexes,which requires that exactly one of each sequence of attribute names must be passed to Aggregate.__init__().Note the lower-case naming.

With XFERINFO in hand, defining the request aggregate (INTRARQ) is simple.

class INTRARQ(Aggregate):""" OFX section 11.7.1.1 """

xferinfo = SubAggregate(INTRARQ, required=True)

Now we we move on to the corresponding server response aggregate (INTRARS). INTRARS contains a new subag-gregate (XFERPRCSTS) for the server to indicate transfer status; we’ll need to implement that first so that INTRARScan refer to it. Here’s the spec.

8.1. Request and Response 31


The XFERPRCCODE element only allows specifically enumerated values. Our validator type for that is ofxtools.Types.OneOf.

class XFERPRCSTS(Aggregate):""" OFX section 11.3.6 """

xferprccode = OneOf("WILLPROCESSON", "POSTEDON", "NOFUNDSON","CANCELEDON", "FAILEDON", required=True)

dtxferprc = DateTime(required=True)

Having XFERPRCSTS, we can define the response aggregate.



This features a new kind of constraint. While DTXFERPRJ and DTPOSTED are mutually exclusive, the ab-sence of boldface type indicates that it’s valid to omit them both, which means we can’t use Aggregate.requiredMutexes as we did for XFERINFO above.

Instead we express this class-level constraint via Aggregate.optionalMutexes, again using lower-cae attributenames within.

from ofxtools.models.i18n import CURRENCY_CODES

class INTRARS(Aggregate):""" OFX section 11.7.1.2 """

curdef = OneOf(*CURRENCY_CODES, required=True)srvrtid = String(10, required=True)xferinfo = SubAggregate(XFERINFO, required=True)dtxferprj = DateTime()dtposted = DateTime()recsrvrtid = String(10)xferprcsts = SubAggregate(XFERPRCSTS)

optionalMutexes = [["dtxferprj", "dtposted"],

]

The definition of currsymbol type refers to the three-letter currency codes in ISO-4217. Happily we’ve already definedthem in ofxtools.models.i18n.

Also note the ofxtools.Types.String validator; it takes an (optional) length argument of type int.

n addition to creating account transfers with INTRARQ, there are also messages for clients to modify or cancel existingtransfer requests. We’ll just bang these out.



class INTRAMODRQ(Aggregate):""" OFX section 11.7.2.1 """

srvrtid = String(10, required=True)xferinfo = SubAggregate(XFERINFO, required=True)

class INTRAMODRS(Aggregate):""" OFX section 11.7.2.2 """

srvrtid = String(10, required=True)xferinfo = SubAggregate(XFERINFO, required=True)xferprcsts = SubAggregate(XFERPRCSTS)

class INTRACANRQ(Aggregate):""" OFX section 11.7.3.1 """

srvrtid = String(10, required=True)



class INTRACANRS(Aggregate):""" OFX section 11.7.3.2 """

srvrtid = String(10, required=True)

Those are all the basic funds transfer commads, but we’re not quite done yet. Every request or response in OFX istransmitted in a transaction wrapper bearing a unique identifier, The structure of these wrappers is laid out in Section2.4.6.1 of the OFX spec.

This commonly-repeated pattern is factored out in ofxtools.models.wrapperbases as base classes for thevarious *TRNRQ / *TRNRS classes to inherit.

class TrnRq(Aggregate):trnuid = String(36, required=True)cltcookie = String(32)tan = String(80)





class TrnRs(Aggregate):trnuid = String(36, required=True)status = SubAggregate(STATUS, required=True)cltcookie = String(32)

Using these base classes, we just need to add attributes for each type of request/response they can wrap, along withclass-level constraints enforcing the choice of a single wrapped entity.

Note that *TRNRQ wrappers must contain a request, while the spec allows empty *TRNRS wrappers, so we setrequiredMutexes and optionalMutexes respectively.

from ofxtools.models.wrapperbases import TrnRq, TrnRs

class INTRATRNRQ(TrnRq):""" OFX section 11.7.1.1 """

intrarq = SubAggregate(STMTRQ)intramodrq = SubAggregate(INTRAMODRQ)intracanrq = SubAggregate(INTRACANRQ)

requiredMutexes = [["intrarq", "intramodrq", "intracanrq"],

]

class INTRATRNRS(TrnRs):""" OFX section 11.7.1.2 """

intrars = SubAggregate(INTRARS)intramodrs = SubAggregate(INTRAMODRS)intracanrs = SubAggregate(INTRACANRS)

optionalMutexes = [["intrars","intramodrs","intracanrs","intermodrs","intercanrs","intermodrs"],

]

8.2 Recurring Requests

In addition to one-time fund transfer requests, a bit further down the spec also details messages for creating, modifying,and canceling recurring funds transfers. This just repeats the pattern of INTRARQ and INTRARS.



class RECINTRARQ(Aggregate):""" OFX section 11.10.1.1 """

recurrinst = SubAggregate(RECURRINST, required=True)intrarq = SubAggregate(INTRARQ, required=True)

class RECINTRARS(Aggregate):""" OFX section 11.10.1.2 """

recsrvrtid = String(10, required=True)recurrinst = SubAggregate(RECURRINST, required=True)intrars = SubAggregate(INTRARS, required=True)

8.2. Recurring Requests 37


class RECINTRAMODRQ(Aggregate):""" OFX section 11.10.2.1 """

recsrvrtid = String(10, required=True)recurrinst = SubAggregate(RECURRINST, required=True)intrarq = SubAggregate(INTRARQ, required=True)modpending = Bool(required=True)

class RECINTRAMODRS(Aggregate):""" OFX section 11.10.2.2 """

recsrvrtid = String(10, required=True)recurrinst = SubAggregate(RECURRINST, required=True)intrars = SubAggregate(INTRARS, required=True)modpending = Bool(required=True)



class RECINTRACANRQ(Aggregate):""" OFX section 11.10.3.1 """

recsrvrtid = String(10, required=True)canpending = Bool(required=True)

class RECINTRACANRS(Aggregate):""" OFX section 11.10.3.2 """

recsrvrtid = String(10, required=True)canpending = Bool(required=True)

recintratrnrq.png

class RECINTRATRNRQ(TrnRq):""" OFX section 11.10.1.1 """

recintrarq = SubAggregate(RECINTRARQ)recintramodrq = SubAggregate(RECINTRAMODRQ)recintracanrq = SubAggregate(RECINTRACANRQ)

requiredMutexes = [["recintrarq", "recintramodrq", "recintracanrq"],

]

8.2. Recurring Requests 39


recintratrnrs.png

class RECINTRATRNRS(TrnRs):""" OFX section 11.10.1.2 """

recintrars = SubAggregate(RECINTRARS)recintramodrs = SubAggregate(RECINTRAMODRS)recintracanrs = SubAggregate(RECINTRACANRS)

optionalMutexes = [["recintrars", "recintramodrs", "recintracanrs"],

]

8.3 Synchronization

Besides commands to perform funds transfers, the OFX spec also defines messages for downloading funds transferactivity. The synchronization protocol and its messages are detailed in a different chapter of the spec - Section 11.12.2.



The requirement that each *SYNCRQ / *SYNCRS may contain a variable number of transaction wrappers means thatwe can’t define these wrappers with SubAggregate, which maps every child element to a single class attribute.

Contained aggregates that are allowed to appear more than once are instead defined with a validator of typeListAggregate, and accessed via the Python list API. Unique children are defined in the usual manner, andaccessed as instance attributes.

Here’s how it looks in ofxtools.models.bank.sync.

from ofxtools.Type import ListAggregatefrom ofxtools.models.bank.stmt import BANKACCTFROM, CCACCTFROMfrom ofxtools.Types import Bool

class INTRASYNCRQ(Aggregate):""" OFX section 11.12.2.1 """token = String(10)tokenonly = Bool()refresh = Bool()rejectifmissing = Bool(required=True)bankacctfrom = SubAggregate(BANKACCTFROM)ccacctfrom = SubAggregate(CCACCTFROM)intratrnrq = ListAggregate(INTRATRNRQ)

requiredMutexes = [["token", "tokenonly", "refresh"],["bankacctfrom", "ccacctfrom"]

]

class INTRASYNCRS(Aggregate):""" OFX section 11.12.2.2 """


8.3. Synchronization 41



token = String(10, required=True)lostsync = Bool()bankacctfrom = SubAggregate(BANKACCTFROM)ccacctfrom = SubAggregate(CCACCTFROM)intratrnrs = ListAggregate(INTRATRNRS)

requiredMutexes = [["bankacctfrom", "ccacctfrom"],

]

class RECINTRASYNCRQ(Aggregate):""" OFX section 11.12.5.1 """

token = String(10)tokenonly = Bool()refresh = Bool()rejectifmissing = Bool(required=True)bankacctfrom = SubAggregate(BANKACCTFROM)ccacctfrom = SubAggregate(CCACCTFROM)recintratrnrq = ListAggregate(RECINTRATRNRQ)

requiredMutexes = [["token", "tokenonly", "refresh"],["bankacctfrom", "ccacctfrom"],

]

class RECINTRASYNCRS(Aggregate):""" OFX section 11.12.5.2 """

token = String(10, required=True)lostsync = Bool()bankacctfrom = SubAggregate(BANKACCTFROM)ccacctfrom = SubAggregate(CCACCTFROM)recintratrnrs = ListAggregate(RECINTRATRNRS)

requiredMutexes = [["bankacctfrom", "ccacctfrom"],

]

8.4 Extending the Message Set

We have defined the funds transfer service, but we still need to add it to the banking message set (the top-levelwrappers). We need to edit the relevant classes in ofxtools.models.msgsets.

class BANKMSGSRQV1(List):""" OFX section 11.13.1.1.1 """

...intratrnrq = ListAggregate(INTRATRNRQ)recintratrnrq = ListAggregate(RECINTRATRNRQ)intrasyncrq = ListAggregate(INTRASYNCRQ)recintrasyncrq = ListAggregate(RECINTRASYNCRQ)





...

class BANKMSGSRSV1(List):""" OFX section 11.13.1.1.2 """

...intratrnrs = ListAggregate(INTRATRNRS)recintratrnrs = ListAggregate(RECINTRATRNRS)intrasyncrs = ListAggregate(INTRASYNCRS)recintrasyncrs = ListAggregate(RECINTRASYNCRS)...

Then we need to define the funds transfer profile.

class XFERPROF(ElementList):""" OFX section 11.13.2.2 """

procdaysoff = ListElement(OneOf(*DAYS))procendtm = Time(required=True)cansched = Bool(required=True)canrecur = Bool(required=True)canmodxfer = Bool(required=True)canmodmdls = Bool(required=True)modelwnd = Integer(3, required=True)dayswith = Integer(3, required=True)dfltdaystopay = Integer(3, required=True)

Finally, we add the funds transfer profile to the message set.

8.4. Extending the Message Set 43


class BANKMSGSETV1(Aggregate):""" OFX section 11.13.2.1 """

...xferprof = SubAggregate(XFERPROF)...

All done!

..resources:


CHAPTER 9

Additional Resources

• The OFX spec is canonical. . .

• . . . but since Quicken dominates the industry, also see the Quicken data mapping guide

• OFX Home is a great free resource to look up OFX connection information for various financial institutions

9.1 More open-source OFX code

• libofx

• ofxparse

• csv2ofx

45


https://fi.intuit.com/technical/resources_tools/QW_DataMappingGuide.pdf


https://github.com/libofx/libofx

https://github.com/jseutter/ofxparse

https://github.com/reubano/csv2ofx


46 Chapter 9. Additional Resources

CHAPTER 10

What is it?

ofxtools requests, consumes, and produces both OFXv1 (SGML) and OFXv2 (XML) formats. It converts serial-ized markup to/from native Python objects of the appropriate data type, while preserving structure. It also handlesQuicken’s QFX format, although it ignores Intuit’s proprietary extension tags.

In a nutshell, ofxtools makes it simple to get OFX data and extract it, or export your data in OFX format.

ofxtools takes a comprehensive, standards-based approach to processing OFX. It targets compliance with the OFXspecification, specifically OFX versions 1.6 and 2.03.

ofxtools Coverage of the OFX Specification

• Section 7 (financial institution profile)

• Section 8 (service activation; account information)

• Section 9 (email over OFX)

• Section 10 (recurring bank transfers)

• Section 11 (banking)

• Section 12 (bill pay)

• Section 13 (investments)

This should cover the great majority of real-world OFX use cases. A particular focus of ofxtools is full support ofthe OFX investment message set, which has been somewhat neglected by the Python community.

The major item remaining on the ofxtools “to do” list is to implement the tax schemas. It’s currently a low priorityto implement Section 14 (bill presentment) or the extensions contained in OFX versions beyond 2.03, but you’rewelcome to contribute code if you need these.

Some care has been taken with the data model to make it easily maintainable and extensible. The ofxtools.models subpackage contains simple, direct translations of the relevant sections of the OFX specification. Usingexisting models as templates, it’s quite straightforward to define new models and cover more of the spec as needed(the odd corner case notwithstanding). See Contributing to ofxtools for a detailed example.

More than 10 years’ worth of OFX data from various financial institutions has been run through the ofxtools parser,with the results checked. Test coverage is high.

47




48 Chapter 10. What is it?

CHAPTER 11

Where is it?

Full documentation is available at Read the Docs.

For ease of installation, ofxtools is released on PyPI.

Development of ofxtools is centralized at GitHub, where you will find a bug tracker.

49

https://ofxtools.readthedocs.io/

https://pypi.python.org/pypi/ofxtools

https://github.com/csingley/ofxtools

https://github.com/csingley/ofxtools/issues


50 Chapter 11. Where is it?

CHAPTER 12

Installation Dependencies

ofxtools requires Python version 3.7+, and depends only on the standard libary (no external dependencies).

NOTE: As of version 0.6, ofxtools no longer supports Python version 2, which goes EOL 2020-01-01.

51

Release 0.8.22 Christopher Singley - Read the Docs

Documents

Transcript of Release 0.8.22 Christopher Singley - Read the Docs