1

Formula Status UpdateNew Usage Patterns for FINREP

Herm FischerFormula Working Group

2008-04-20

2

Formula Status

• Status = “Proposed Recommendation”– 45 day wait before Recommendation

• Four “known” implementations– Fujitsu & UBmatrix conformant, in production use– CompSci Resources & New Lido implementations

• Major stake holders– BdE, BdF, BoJ, SEC deployed– FDIC has early-IWD formulas

3

Specifications are Extendable

• Basic usage patterns in PR spec– Producing fact items (output instance document)– Assertions for

• Consistency (produced fact vs. reported fact)

• Existence

• Value

– Single input, single output of instances

• Extension usage patterns

4

Extension usage patterns

Implementation prototypes• Message composition• Formula chaining• Tuple generation• Multi-instance processing• Linkbase & footnotes functions• Custom functions implemented in XPath• Assertion sets

Interesting, not implemented yet• Very Large Instances processing

5

Existing prototypes

• Chaining and tuple generation– XSB required prototype to assure PR supports it

• Multi-instance processing – Became core part of chaining proposal– Prototyped (partially)

• Linkbase tree walks– Prototype allows linkbase to control formulas of

• Calc linkbase & dimension aggregation roll ups

• Movements & totalling by presentation linkbase

6

Discussing by history vs. Discussing by FINREP urgency

• Message generation (needed)• Chaining (cool, but getting by without this)• Tuple generation (just for GL people?)• Multi-instance (it is the chaining solution)• Custom functions with XPath (need this)• Linkbase functions (top priority)

– Patterns in linkbase eliminate most formulas– Moves formula semantics from code to linkbase– Critical for success

• Assertion sets (in use)

7

Chaining

• Most frequently request usage case

• Least needed (based on experience)

• Two solutions implemented – Multi-instance approach most flexible– Explicity dependencies useful for tuple output

• Facilitates modularization

• Helpful to manage large formula projects

8

Chaining with explicit dependencyPrior result passed as factVariable

chained variableSet

chained variableSet e.g., nested tuple contents facts

factVariablefactVariablefilter(s)

V-F

factVariablefilter(s)

V-F

formula evaluation

factVariable

fact & gen’l variables by dependency, precondition, result, then varSet arcs to nested varSets (e.g., consis or tuple)

factVariablefilter(s)

factVariablefilter(s)

V-F

for consisAsser, parameters; for nested formula(s), factVars & genVars

location aspectfor nested results

precondition

source instance fact variables

precondition

consistencyAssertion

consisAssertion

9

Dependency is explicit

• The arc assigns prior result to a factVariable

• The \author explicitly specifies this dependency

• Important for tuple output

• Difficult to maintain in large formula sets

• Difficult to factor formulae to separate files

10

Tuple chaining must be explicit

Formula for deeper nested tuplefactVariableconceptName filter F-V

factVariableconceptName filter

V-F

factVariableconceptName filter

V-F

Formula for nested tuplefactVariableconceptName filter F-V

factVariableconceptName filter

V-F

factVariableconceptName filter

V-F

Formula for outer tuplefactVariable

conceptName filter

F-V

fact variables by their order dependency, then formula-formula arcs to nested tuple

factVariableconceptName filter

V-F

factVariableconceptName filter

V-F

11

Muti-instance chaining solution

• Common solution to multi-instance and chaining

• Let’s first look at multi-instanceand then come back to chaining that uses it

12

Other multi-instance solutions

• Complete DTS with linkbases needed– For tree walks: movements, totaling, and dimensions

• fn:doc() not useful to load instances– Does not discover DTS– Does not load linkbases– Can not handle shared DTS components

• Some referenced taxonomies common

• Some linkbases evolve

13

Prior: xfi:inst() was prototyped

• Provides a fn:doc() counterpart to load DTS

• generalVariables can access these instances– No filtering on multi-instances– factVar, filter on primary instance,

genVar, XPath on additional instances

• Requires formula execution code to load instances (instead of formula processor infrastructure)

14

New multi-instance features

• All instances loaded by infrastructure(doesn’t have to be coded)

• Filters, functions, sequence, covering work

• Should share formulas & filters - all instances

15

New multi-instance solution

• Instances are represented by instance resource

• instance-variable arc to factVariable– If present, specifies non-default source instance

• formula-instance arc from formula– If present specifies the instance to receive fact

• Instance resources are files or temporary

16

Instance resources

• Could be loaded by processor– E.g., java code in a server loads primary instance and

some prior-period or other-company instances– Or user of GUI adds ‘additional’ instances, such as

loading prior-period or other-company instances

• Default implied source and result instances

• Can be temporary in memory only– Used for chaining and modularization

17

Multi-instance solution

• A better approach to chaining

• Implements multiple instance documents

• Applies to very large instance solution

18

Multi-source and result instances

formula a = $b + $c

$d

fact

to result instance

factVariable

$c

filter(s)

V-F

factVariablefilter(s)

V-Fformula c = $d + $e

$e

fact

to result instance

fact & gen’l variables by dependency, precondition, result, then varSet arcs to nested varSets (e.g., consis or tuple)

factVariable

$d

filter(s)

V-F

factVariablefilter(s)

V-F

consisassertion

consisAsser

location aspectfor nested results

precondition

precondition

consistencyAssertion

consisAssertion

source instance (default)

temporary result

source instance (default)

result instance

source instance (default)

specifiedoutput

instance

defaultoutputinstance

output result(default)

19

Aspect sources, implicit filtering

• Formula aspects come from its variables

• Variables from different instances contribute aspects– Aspects independent of the instances they come from– Aspect “covering” is by-aspect, not by-instance

20

A=B+C; C=D+E use case (Explicit dependency chaining)

• Formula 1 (C=D+E)– Result is C, factVariables D & E– factVariables D & E are from the source instance

• Formula 2 (A=B+C)– Arc from formula 1, name $r given to Formula 1 result– Result is A, factVariables B & C– factVariable B is from source instance– factVariable $r is from result of formula 1

21

A=B+C; C=D+E (Example 0027 v-01)Explicit dependency chaining

chained formulaA = B + C

fact AfactVariablefilter(s)

V-F

factVariable$r from above

formula evaluationC = D + E

fact C

fact & gen’l variables by dependency, precondition, result, then varSet arcs to nested varSets (e.g., consis or tuple)

factVariablefilter(s)

factVariablefilter(s)

V-F

for consisAsser, parameters; for nested formula(s), factVars & genVars

location aspectfor nested results

source instance fact variables

consistencyAssertion

consisAssertion

22

A=B+C; C=D+E use case (Multi-instance chaining)

• Formula 1 (A=B+C)– Result is A, factVariables B & C– factVariable B is from source instance (default)– factVariable C is from result instance (has an arc)

• Formula 2 (C=D+E)– Result is C, factVariables D & E– factVariables D & E are from the source instance

23

A=B+C; C=D+E (Example 0026 v-01) Multi-instance chaining

Formula evaluationA = B + C (from above)

$b

fact A

to result instance

factVariable

$c

filter(s)

V-F

factVariablefilter(s)

V-F

formula evaluationC = D + E

$e

fact C

to result instance

fact & gen’l variables by dependency, precondition, result, then varSet arcs to nested varSets (e.g., consis or tuple)

factVariable

$d

filter(s)

V-F

factVariablefilter(s)V-F

location aspectfor nested results

source instance (default)

result instance

source instance (default)

source is the result instance

source instance (default)

result instance (default)

result instance (default)

24

COREP Use case 18: Weighted average of member children • Weighted average of its dimensional children

by another primary item

ii

iii

dp

dpdpdp

)(

)()()(

.12

.12.11

11

Primary items ==>PD Assigned to the obligor grade Exposure value

Capital requirements

Total Exposures (dimension) 60% 26.750,00 € 1 Assigned to obligor grade 27% 1.250,00 € Obligor grade 1 10% 100,00 € Obligor grade 2 20% 150,00 € Obligor grade 3 30% 1.000,00 € 2 Specialized lending slotting 62% 25.500,00 € Risk weight 0% 50% 500,00 € Risk weight 10% 75% 20.000,00 € Risk weight 150% 10% 5.000,00 €

25

Current single-formula solution

• Excel formulas:

• Make PD controlling fact, get PD and EV of dimensional children

• General variable for PDxEV member matching

Primary items ==>PD Assigned to the obligor grade Exposure value Capital requirements

Total Exposures (dimension) =(B3*C3+B7*C7)/C2 =C3+C7 1 Assigned to obligor grade =(B4*C4+B5*C5+B6*C6)/C3 =SUMA(C4:C6) Obligor grade 1 0,1 100 Obligor grade 2 0,2 150 Obligor grade 3 0,3 1000 2 Specialized lending slotting =(B8*C8+B9*C9+B10*C10)/C7 =SUMA(C8:C10) Risk weight 0% 0,5 500 Risk weight 10% 0,75 20000 Risk weight 150% 0,1 5000

26

<f:formula ...

value="$v:sumPDxEV div sum($v:EVkids)"

source="v:PD" />

Tests PD of input instance is within ½% of weighted average value

E-L

factVariable

PD

factVariablePdkids

(sequence)

Binds to any PD fact

Sequence of PD values for dimensional children of $PD

E-L

E-L

conceptName filterqname c:PD

label resources

explicitDimension filter, axis=childdimension qname c:ExposuresDimensionvariable c:PD

Pdkids & Evkids depend on PD, PDxEV

on both Pdkids & EVkids

factVariableEVkids

(sequence)

conceptName filterqname c:EV

Sequence of PD values for dimensional children of $PD

E-L

preconditiontest=”count($PDkids) eq count($EVkids)

and sum($EVkids) gt 0"

general variable sumPDxEVsum( for $pd in $PDkids, $ev in $EVkids[xfi:dimension-value(.,QName(‘ExposuresDimension’) = xfi:dimension-value($pd,QName(‘ExposuresDimension’)] return $pd * $ev )

Sequence of ev x pd for matching dimensions

consistencyAssertion strict="false" acceptRadius=".005"

Single formula (Example 0017 v-01)

difficultto explain

27

Exposure value formula

• Each PD x EV produced by one formula– Result factItem PDxEV is the product for each

dimension value

• Second formula binds PDxEV’s of dim-children to sequence and EV’s of dim-children to second sequence, value assertion checks result

28

New idea: multiple result instances

• The PDxEV result fact items aren’t needed for a real result instance

• Only a value assertion is really needed

• A temporary-results instance might be useful

• Also a temporary facts DTS would be needed (to define the PDxEV result fact item)

29

Chained formulas (0026 v-20)

<f:formula ... value="sum($v:PDxEVkids) div sum($v:EVkids)" source="v:PD" />

Tests PD of input instance is within ½% of weighted average value

factVariablePD

factVariableEVkids

(sequence)

V-F

Binds to any PD fact

Sequence of EV values for dimensional children of $PD

E-LconceptName filter

qname c:PD

label resources

explicitDimension filter, axis=childdimension qname c:ExposuresDimensionvariable $PD

EVkids & PDxEvkids depend on PD,

PDxEV depends on above formula

factVariablePDxEVkids(sequence)

conceptName filterqname c:PDxEV

Sequence of PDxEV values for dimensional children of $PDfrom temp instance source

preconditiontest=”count($EVkids) eq count($PDxEVkids)

and sum($EVkids) gt 0"

consistencyAssertion strict="false" acceptRadius=".005"

<f:formula ... value="$v:PD * $v:EV" source="v:PD" conceptName=v:PDxEV/> (syntax abstracted)

Produces PDxEV for each dimension

factVariablePD

conceptName filterqname c:PD

factVariableEV

conceptName filterqname c:EV

conceptName filterqname c:EV

source instance (default)

result instance

fact PDxEV

30

Implementation issues

• Multi-instance term binding– Variables can be bound to different source instances– (This already exists in xfi:inst() based solution.)– Each term in XPath ‘knows’ its instance/DTS (in the

internal model or DOM of implementation)

• Function binding– A function with item results must keep the instance/DTS

of the function result (based on the input terms)

31

Tree walking

• Current implementation– Navigation returns concepts and attributes

( (c1, c2, c3), (a11, a12, a13), (a21, a22, a23) )– take subsequences with XPath for-loops– working (geeky)

• Change idea (not prototyped yet)– Navigation returns fully resolved relationship nodes

• A relationship has reference to arc node attributes

– Attributes: rel/@weight, rel/@preferredLabel– Concepts: maybe xfi:from/xfi:to( rel-node )

32

Use of tree walking

• Calculation linkbase checking by formula– Uses xfi function for linkbase tree walk– Roll ups compared

• By threshold value

• By rounded values same as ordinary calc validation

– Extended links managed by formula• EDInet consolidated vs nonConsolidated conflicts

• Dimension aggregation by formula– Uses dimension filter child/descendant feature

33

FINREP formulas

• Most current formulas can be custom tree walk– Consider optional/required attribute– Consider fall back values by arc attribute– Consider dimension filter by arc attribute– Other attributes as needed

• Replace 72± (BdF count) with few tree walks

34

Very Large Instances Use Case

• Sizes > ½ million facts, > ½ GB DOMs– Census, Tax office, Security exchanges, etc.

• Multi-GB heaps not feasible with Java VMs– moribund at couple GB (incl code & data)

• Data almost always from Relational DBMSes

35

Very Large Instances approach

• Basic PR formula solution– All facts, all filters, all variable sets in parallel– Not feasible with very large single- or multi-instances

• Multi-instance approach– Allows modularizing processing– Stage formulas to work on parts of very large instance– Cooperative filters & (stored) SQL DB interfaces– Intermediate result instances pass between stages

36

Staged multi-instance strategy

SQL

SQL

SQL

Very LargeRelational DB

Formula Linkbase(s)

lazy loadearly GCinstance

filtersformula &variables

interiminstance

filtersformula &variables

filtersformula &variables

result

facts

facts

facts

interiminstance

37

Custom functions with XPath

• Custom functions in PR require Java code

• Examples of custom functions– Taxonomy and linkbase access– Math with exponentials and recursion (loan value calc)

• Prototype adds XPath implementation

38

a(b,c) = $b + $c (Example 0030 v01)

• <formula:formula value="my-fn:a($b, $c)“ …>• <function-impl:function xlink:type="resource"

xlink:label="cust-fn-a“ name="my-fn:a“ output="xs:decimal“ value="$b + $c" > <function-impl:input name="b“ type="schema-element(xbrli:item)" /> <function-impl:input name="c“ type="schema-element(xbrli:item)" /> </function-impl:function>

39

Precision by unit (Example 0030 v-03)

<formula:decimals>my-fn:decimals($b) </formula:decimals>

value=" for $unit in local-name-from-QName( xfi:measure-name( xfi:unit-numerator( xfi:unit( $item ))[1] )) return ( if ($unit eq 'JPY') then -5 else -2 ) " >

40

Recursion (Example 0030 v-04)

<function-impl:function xlink:type="resource" xlink:label="cust-fn-a" name="my-fn:power" output="xs:decimal" value=" if ($exp lt 0) then ( 1 div my-fn:power($y, - $exp) ) else ( if ($exp lt 1) then 1 else ($y * my-fn:power($y,$exp - 1)) ) " > <function-impl:input name="y" type="xs:decimal" /> <function-impl:input name="exp" type="xs:decimal" /> </function-impl:function>

41

Present value (Example 0030 v-05)

<formula:formula xlink:type="resource" xlink:label="formula1"

value="$amountDue *

my-fn:power((1 + $interestRate), $numYears)"

42

Back to FINREP

• Segue to – Use of tree walks to consolidate many formula