1 Formula Status Update New Usage Patterns for FINREP Herm Fischer Formula Working Group 2008-04-20.

1

Formula Status UpdateNew Usage Patterns for FINREP

Herm FischerFormula Working Group

2008-04-20

2

Formula Status

• Status = “Proposed Recommendation”– 45 day wait before Recommendation

• Four “known” implementations– Fujitsu & UBmatrix conformant, in production use– CompSci Resources & New Lido implementations

• Major stake holders– BdE, BdF, BoJ, SEC deployed– FDIC has early-IWD formulas

3

Specifications are Extendable

• Basic usage patterns in PR spec– Producing fact items (output instance document)– Assertions for

• Consistency (produced fact vs. reported fact)

• Existence

• Value

– Single input, single output of instances

• Extension usage patterns

4

Extension usage patterns

Implementation prototypes• Message composition• Formula chaining• Tuple generation• Multi-instance processing• Linkbase & footnotes functions• Custom functions implemented in XPath• Assertion sets

Interesting, not implemented yet• Very Large Instances processing

5

Existing prototypes

• Chaining and tuple generation– XSB required prototype to assure PR supports it

• Multi-instance processing – Became core part of chaining proposal– Prototyped (partially)

• Linkbase tree walks– Prototype allows linkbase to control formulas of

• Calc linkbase & dimension aggregation roll ups

• Movements & totalling by presentation linkbase

6

Discussing by history vs. Discussing by FINREP urgency

• Message generation (needed)• Chaining (cool, but getting by without this)• Tuple generation (just for GL people?)• Multi-instance (it is the chaining solution)• Custom functions with XPath (need this)• Linkbase functions (top priority)

– Patterns in linkbase eliminate most formulas– Moves formula semantics from code to linkbase– Critical for success

• Assertion sets (in use)

7

Chaining

• Most frequently request usage case

• Least needed (based on experience)

• Two solutions implemented – Multi-instance approach most flexible– Explicity dependencies useful for tuple output

• Facilitates modularization

• Helpful to manage large formula projects

8

Chaining with explicit dependencyPrior result passed as factVariable

chained variableSet

chained variableSet e.g., nested tuple contents facts

factVariablefactVariablefilter(s)

V-F

factVariablefilter(s)

V-F

formula evaluation

factVariable

fact & gen’l variables by dependency, precondition, result, then varSet arcs to nested varSets (e.g., consis or tuple)



V-F

for consisAsser, parameters; for nested formula(s), factVars & genVars

location aspectfor nested results

precondition

source instance fact variables

precondition

consistencyAssertion

consisAssertion

9

Dependency is explicit

• The arc assigns prior result to a factVariable

• The \author explicitly specifies this dependency

• Important for tuple output

• Difficult to maintain in large formula sets

• Difficult to factor formulae to separate files

10

Tuple chaining must be explicit

Formula for deeper nested tuplefactVariableconceptName filter F-V

factVariableconceptName filter

V-F


V-F

Formula for nested tuplefactVariableconceptName filter F-V


V-F


V-F

Formula for outer tuplefactVariable

conceptName filter

F-V

fact variables by their order dependency, then formula-formula arcs to nested tuple


V-F


V-F

11

Muti-instance chaining solution

• Common solution to multi-instance and chaining

• Let’s first look at multi-instanceand then come back to chaining that uses it

12

Other multi-instance solutions

• Complete DTS with linkbases needed– For tree walks: movements, totaling, and dimensions

• fn:doc() not useful to load instances– Does not discover DTS– Does not load linkbases– Can not handle shared DTS components

• Some referenced taxonomies common

• Some linkbases evolve

13

Prior: xfi:inst() was prototyped

• Provides a fn:doc() counterpart to load DTS

• generalVariables can access these instances– No filtering on multi-instances– factVar, filter on primary instance,

genVar, XPath on additional instances

• Requires formula execution code to load instances (instead of formula processor infrastructure)

14

New multi-instance features

• All instances loaded by infrastructure(doesn’t have to be coded)

• Filters, functions, sequence, covering work

• Should share formulas & filters - all instances

15

New multi-instance solution

• Instances are represented by instance resource

• instance-variable arc to factVariable– If present, specifies non-default source instance

• formula-instance arc from formula– If present specifies the instance to receive fact

• Instance resources are files or temporary

16

Instance resources

• Could be loaded by processor– E.g., java code in a server loads primary instance and

some prior-period or other-company instances– Or user of GUI adds ‘additional’ instances, such as

loading prior-period or other-company instances

• Default implied source and result instances

• Can be temporary in memory only– Used for chaining and modularization

17

Multi-instance solution

• A better approach to chaining

• Implements multiple instance documents

• Applies to very large instance solution

18

Multi-source and result instances

formula a = $b + $c

$d

fact

to result instance

factVariable

$c

filter(s)

V-F


V-Fformula c = $d + $e

$e

fact

to result instance


factVariable

$d

filter(s)

V-F


V-F

consisassertion

consisAsser


precondition

precondition


consisAssertion

source instance (default)

temporary result


result instance


specifiedoutput

instance

defaultoutputinstance

output result(default)

19

Aspect sources, implicit filtering

• Formula aspects come from its variables

• Variables from different instances contribute aspects– Aspects independent of the instances they come from– Aspect “covering” is by-aspect, not by-instance

20

A=B+C; C=D+E use case (Explicit dependency chaining)

• Formula 1 (C=D+E)– Result is C, factVariables D & E– factVariables D & E are from the source instance

• Formula 2 (A=B+C)– Arc from formula 1, name $r given to Formula 1 result– Result is A, factVariables B & C– factVariable B is from source instance– factVariable $r is from result of formula 1

21

A=B+C; C=D+E (Example 0027 v-01)Explicit dependency chaining

chained formulaA = B + C

fact AfactVariablefilter(s)

V-F

factVariable$r from above

formula evaluationC = D + E

fact C




V-F

for consisAsser, parameters; for nested formula(s), factVars & genVars


source instance fact variables


consisAssertion

22

A=B+C; C=D+E use case (Multi-instance chaining)

• Formula 1 (A=B+C)– Result is A, factVariables B & C– factVariable B is from source instance (default)– factVariable C is from result instance (has an arc)

• Formula 2 (C=D+E)– Result is C, factVariables D & E– factVariables D & E are from the source instance

23

A=B+C; C=D+E (Example 0026 v-01) Multi-instance chaining

Formula evaluationA = B + C (from above)

$b

fact A

to result instance

factVariable

$c

filter(s)

V-F


V-F

formula evaluationC = D + E

$e

fact C

to result instance


factVariable

$d

filter(s)

V-F

factVariablefilter(s)V-F



result instance


source is the result instance


result instance (default)

result instance (default)

24

COREP Use case 18: Weighted average of member children • Weighted average of its dimensional children

by another primary item

ii

iii

dp

dpdpdp

)(

)()()(

.12

.12.11

11

Primary items ==>PD Assigned to the obligor grade Exposure value

Capital requirements

Total Exposures (dimension) 60% 26.750,00 € 1 Assigned to obligor grade 27% 1.250,00 € Obligor grade 1 10% 100,00 € Obligor grade 2 20% 150,00 € Obligor grade 3 30% 1.000,00 € 2 Specialized lending slotting 62% 25.500,00 € Risk weight 0% 50% 500,00 € Risk weight 10% 75% 20.000,00 € Risk weight 150% 10% 5.000,00 €

25

Current single-formula solution

• Excel formulas:

• Make PD controlling fact, get PD and EV of dimensional children

• General variable for PDxEV member matching

Primary items ==>PD Assigned to the obligor grade Exposure value Capital requirements

Total Exposures (dimension) =(B3*C3+B7*C7)/C2 =C3+C7 1 Assigned to obligor grade =(B4*C4+B5*C5+B6*C6)/C3 =SUMA(C4:C6) Obligor grade 1 0,1 100 Obligor grade 2 0,2 150 Obligor grade 3 0,3 1000 2 Specialized lending slotting =(B8*C8+B9*C9+B10*C10)/C7 =SUMA(C8:C10) Risk weight 0% 0,5 500 Risk weight 10% 0,75 20000 Risk weight 150% 0,1 5000

26

<f:formula ...

value="$v:sumPDxEV div sum($v:EVkids)"

source="v:PD" />

Tests PD of input instance is within ½% of weighted average value

E-L

factVariable

PD

factVariablePdkids

(sequence)

Binds to any PD fact

Sequence of PD values for dimensional children of $PD

E-L

E-L

conceptName filterqname c:PD

label resources

explicitDimension filter, axis=childdimension qname c:ExposuresDimensionvariable c:PD

Pdkids & Evkids depend on PD, PDxEV

on both Pdkids & EVkids

factVariableEVkids

(sequence)

conceptName filterqname c:EV

Sequence of PD values for dimensional children of $PD

E-L

preconditiontest=”count($PDkids) eq count($EVkids)

and sum($EVkids) gt 0"

general variable sumPDxEVsum( for $pd in $PDkids, $ev in $EVkids[xfi:dimension-value(.,QName(‘ExposuresDimension’) = xfi:dimension-value($pd,QName(‘ExposuresDimension’)] return $pd * $ev )

Sequence of ev x pd for matching dimensions

consistencyAssertion strict="false" acceptRadius=".005"

Single formula (Example 0017 v-01)

difficultto explain

27

Exposure value formula

• Each PD x EV produced by one formula– Result factItem PDxEV is the product for each

dimension value

• Second formula binds PDxEV’s of dim-children to sequence and EV’s of dim-children to second sequence, value assertion checks result

28

New idea: multiple result instances

• The PDxEV result fact items aren’t needed for a real result instance

• Only a value assertion is really needed

• A temporary-results instance might be useful

• Also a temporary facts DTS would be needed (to define the PDxEV result fact item)

29

Chained formulas (0026 v-20)

<f:formula ... value="sum($v:PDxEVkids) div sum($v:EVkids)" source="v:PD" />

Tests PD of input instance is within ½% of weighted average value

factVariablePD

factVariableEVkids

(sequence)

V-F

Binds to any PD fact

Sequence of EV values for dimensional children of $PD

E-LconceptName filter

qname c:PD

label resources

explicitDimension filter, axis=childdimension qname c:ExposuresDimensionvariable $PD

EVkids & PDxEvkids depend on PD,

PDxEV depends on above formula

factVariablePDxEVkids(sequence)

conceptName filterqname c:PDxEV

Sequence of PDxEV values for dimensional children of $PDfrom temp instance source

preconditiontest=”count($EVkids) eq count($PDxEVkids)

and sum($EVkids) gt 0"

consistencyAssertion strict="false" acceptRadius=".005"

<f:formula ... value="$v:PD * $v:EV" source="v:PD" conceptName=v:PDxEV/> (syntax abstracted)

Produces PDxEV for each dimension

factVariablePD

conceptName filterqname c:PD

factVariableEV




result instance

fact PDxEV

30

Implementation issues

• Multi-instance term binding– Variables can be bound to different source instances– (This already exists in xfi:inst() based solution.)– Each term in XPath ‘knows’ its instance/DTS (in the

internal model or DOM of implementation)

• Function binding– A function with item results must keep the instance/DTS

of the function result (based on the input terms)

31

Tree walking

• Current implementation– Navigation returns concepts and attributes

( (c1, c2, c3), (a11, a12, a13), (a21, a22, a23) )– take subsequences with XPath for-loops– working (geeky)

• Change idea (not prototyped yet)– Navigation returns fully resolved relationship nodes

• A relationship has reference to arc node attributes

– Attributes: rel/@weight, rel/@preferredLabel– Concepts: maybe xfi:from/xfi:to( rel-node )

32

Use of tree walking

• Calculation linkbase checking by formula– Uses xfi function for linkbase tree walk– Roll ups compared

• By threshold value

• By rounded values same as ordinary calc validation

– Extended links managed by formula• EDInet consolidated vs nonConsolidated conflicts

• Dimension aggregation by formula– Uses dimension filter child/descendant feature

33

FINREP formulas

• Most current formulas can be custom tree walk– Consider optional/required attribute– Consider fall back values by arc attribute– Consider dimension filter by arc attribute– Other attributes as needed

• Replace 72± (BdF count) with few tree walks

34

Very Large Instances Use Case

• Sizes > ½ million facts, > ½ GB DOMs– Census, Tax office, Security exchanges, etc.

• Multi-GB heaps not feasible with Java VMs– moribund at couple GB (incl code & data)

• Data almost always from Relational DBMSes

35

Very Large Instances approach

• Basic PR formula solution– All facts, all filters, all variable sets in parallel– Not feasible with very large single- or multi-instances

• Multi-instance approach– Allows modularizing processing– Stage formulas to work on parts of very large instance– Cooperative filters & (stored) SQL DB interfaces– Intermediate result instances pass between stages

36

Staged multi-instance strategy

SQL

SQL

SQL

Very LargeRelational DB

Formula Linkbase(s)

lazy loadearly GCinstance

filtersformula &variables

interiminstance



result

facts

facts

facts

interiminstance

37

Custom functions with XPath

• Custom functions in PR require Java code

• Examples of custom functions– Taxonomy and linkbase access– Math with exponentials and recursion (loan value calc)

• Prototype adds XPath implementation

38

a(b,c) = $b + $c (Example 0030 v01)

• <formula:formula value="my-fn:a($b, $c)“ …>• <function-impl:function xlink:type="resource"

xlink:label="cust-fn-a“ name="my-fn:a“ output="xs:decimal“ value="$b + $c" > <function-impl:input name="b“ type="schema-element(xbrli:item)" /> <function-impl:input name="c“ type="schema-element(xbrli:item)" /> </function-impl:function>

39

Precision by unit (Example 0030 v-03)

<formula:decimals>my-fn:decimals($b) </formula:decimals>

value=" for $unit in local-name-from-QName( xfi:measure-name( xfi:unit-numerator( xfi:unit( $item ))[1] )) return ( if ($unit eq 'JPY') then -5 else -2 ) " >

40

Recursion (Example 0030 v-04)

<function-impl:function xlink:type="resource" xlink:label="cust-fn-a" name="my-fn:power" output="xs:decimal" value=" if ($exp lt 0) then ( 1 div my-fn:power($y, - $exp) ) else ( if ($exp lt 1) then 1 else ($y * my-fn:power($y,$exp - 1)) ) " > <function-impl:input name="y" type="xs:decimal" /> <function-impl:input name="exp" type="xs:decimal" /> </function-impl:function>

41

Present value (Example 0030 v-05)

<formula:formula xlink:type="resource" xlink:label="formula1"

value="$amountDue *

my-fn:power((1 + $interestRate), $numYears)"

42

Back to FINREP

• Segue to – Use of tree walks to consolidate many formula

1 Formula Status Update New Usage Patterns for FINREP Herm Fischer Formula Working Group 2008-04-20.

Documents

Transcript of 1 Formula Status Update New Usage Patterns for FINREP Herm Fischer Formula Working Group 2008-04-20.