Solve Production Allocation and Reconciliation Problems using the same Network

Jeffrey D. Kelly1

Industrial Algorithms LLC.

15 St. Andrews Road,

Toronto, Canada, M1P 4C3

Keywords: Allocation, reconciliation, matrix projection, network flow model and observability.

1 E-mail: jdkelly@industrialgorithms.ca

2

Lead Paragraph

Production allocation is a business accounting practice used throughout the processing world to proportionately and quantitatively

assign measurement error and production expenditures or overheads to internal and external business owners. Reconciliation is a

scientific function to vet production data of gross errors or non-random variation if it occurs and to find more precise estimates of

the measured values. The consequence of our proposed technique is to allow these two functions the capability to use the same

production network or flow-path. Only one model is required to be maintained eliminating the possibility that potentially costly

mis-allocation will occur due to business and engineering model-mismatch. Mis-allocation due to measurement errors can still be

problematic as we illustrate in an example, but should be reduced over time because of the reconciliation measurement diagnostics.

Introduction

The focus of this article is to highlight a technique to solve production allocation problems using the same network or flowsheet as

is used for the engineering function of data reconciliation. Data reconciliation is a technique to optimally adjust measured process

data so that they are consistent with known constraints such as material and energy balances. Production allocation is effectively a

financial (or non-engineering) process of fixing key flow meter readings and prorating or allocating the material imbalances onto

other measurements according to their relative measurements. This provides a simple procedure, which is easily verified manually,

to allow production and cost accounting functions to apportion or distribute resource usage to consumers within the plant, or

outside per contracted agreements.

Reconciliation tools that address plant equipment process networks have not supported the allocation process. At issue, the

reconciled measurements are engineering best estimates of true flow, not the business-defined values required. Allocation engines

utilizing the production network would be confounded by the numerous unmeasured flows present in a real plant.

Historically, allocation problems have been solved by entering transactions into spreadsheets or ledgers and applying corrections

where necessary to achieve closure. Automated approaches included hard coding the network information into real-time process

historian calculations and allocating based on designated error-free flow meters (with minimal common cause2 variation only).

There are difficulties with this approach. The first issue arises with developing complex allocation functions when the equipment

line-ups (source to destination routes) are dynamic causing the rules for allocation to vary in time. A second issue arises in

updating the calculations when physical plant changes are made as the accounting and engineering functions are typically separate.

Another problem is that allocation offers no means of diagnosing when there are measurement malfunctions. Allocation also does

not assist the accounting function in determining a best estimate value for a measurement that has failed.

It is for these reasons that motivate us to develop a more sustainable method to calculate production allocation numbers used for

the many aspects of business accounting. The first section to follow describes the technology used to perform reconciliation and

2 Common cause variation is considered as back-ground noise which is present in all real processes.

3

allocation using the same network by the method of matrix projection invented by Crowe et. al. (1983). The second section

highlights the technique in more detail using a small but representative illustrative example.

Allocation by Matrix Projection

The methods to solve steady-state data reconciliation problems are well studied and can be found in the many articles referenced in

the survey by Crowe (1996). Solving an allocation problem within a network is much less studied yet the two are similar given the

same underlying matrix operations required. For the sake of brevity and to maintain the overall clarity of the business message, we

exclude most of the mathematics involved and only discuss the key aspects of the allocation solution. First we need to define the

attributes of the streams from the perspective of allocation. In reconciliation the attributes of a stream in the network are measured,

unmeasured and fixed. Measured means there is a flow reading on the stream and the value usually contains noise or random error

(common cause). Unmeasured means there is no flow reading available and if possible the value must be determined from the

network and the measurements. Fixed means that there is a flow reading and there is no random error and is declared error-free. In

allocation a stream can take on one of the following attributes measured, unmeasured, fixed and excluded. These have much the

same meaning as in reconciliation where excluded means that it is excluded or eliminated when allocation is performed but is

included in the network when reconciliation is run. Excluded streams are those that may have useful measurements for the

reconciliation stage but are required to be removed from the allocation of production. An example of such a stream would be a

slipstream of natural gas sent to a gas turbine used to drive a compressor.

However, in reconciliation the fixed designation is used sparingly in order to help detect and identify possible gross errors or faults

in the measurements. If a stream is unmeasured then it will be unmeasured in both reconciliation and allocation. A stream is

deemed to be fixed in the allocation (from either a measured or fixed in the reconciliation) when it is to be used as one of the

streams that defines how the production will be allocated to the other connected measured streams. A measured stream can only be

allocated after allocation if it meets the criteria that it is measured and it is incident or directly connected to a node in the network

(i.e., balance equation) with either all fixed streams on the inlet or all fixed streams on the outlet but not both. Nodes which have

all fixed streams on the inlet side are called feed-side fixed and the opposite is called a product-side fixed node where it is perfectly

acceptable to have some fixed streams on the outlet of feed-side fixed nodes and visa versa. An interesting aspect of solving

allocation using the same network as the reconciliation is the situation where an allocation specified measured or unmeasured

stream becomes internally fixed based on up or downstream propagation of the explicitly fixed streams. This is handled

automatically in the allocation solver. If a node, which has incident unmeasured streams attached and these streams are not

internally labeled as fixed, then the node is deemed to be non-allocatable.

The allocation algorithm is now expressed at a high-level making use of what is known as the matrix projection (Crowe et. al.

(1983)) in Step 2 below:

4

1. Take the fixed streams and their values and solve a network balance whereby the measured and unmeasured streams

are degrees-of-freedom in the network. That is, the measured and unmeasured streams are solved for using the fixed

stream and network information only. If any of the measured or unmeasured streams are found to be observable,

solvable or determinable (Kelly (1998b)) then internally fix their values to the values found and internally re-label

them as fixed. Observable simply means that the value of a stream can be uniquely determined from the network and

the available stream values else it is unobservable.

2. Compute the matrix projection based on the unmeasured variables incidence matrix (Kelly (1998a)) and project out

the unmeasured variables from the network and call this the projected network. For each projected node determine

which are feed or product-side fixed and sum the measured flows-out and the fixed flows-in if a feed-side fixed node.

Decrement the fixed flow-in amount by the sum of any fixed flow-out’s and compute the allocated values for each

measured flow-out by the equation below:

n

m

k

f

j

f

ma

nx

kxjx

ixix

)(

)()(

)()(

where )(ixa is the allocated value for the measured flow-out’s on a feed-side fixed projected node, )(ixm is the

measured flow-out value, )( jx f is a fixed flow-in value with index j spanning the set of fixed flows-in connected to

the projected node, )(kx f is any fixed flows-out of the projected node and )(nxm is a measured flow-out

connected to the projected node with n spanning the set of measured flows-out excluding any fixed flows-out. The

equation can be easily modified for product-side fixed projected nodes.

3. After Step 2 not all of the measured flows will be allocated. The next step is to iterate or loop through each non-

allocated measured flow cycling through all of the projected nodes and recording whether or not it has been allocated

until either all of the measured flows have been allocated or there is no change in the number of allocated measured

flows from the previous loop. If a flow has been allocated either in a previous loop or in Step 2 then it is considered

to be an internally fixed flow for the next iterations. The maximum number of iterations is bounded by the number of

measured flows-in the network. We require this step given that there may many up or downstream projected nodes

which involve only measured flows and no fixed flows.

4. Perform another network solve to determine the unmeasured flow values and report any unobservable flows if they

exist by substituting in all of the allocated values including the fixed flow values.

The fundamental idea of the matrix projection method applied to the production allocation problem is to remove from the network

all unmeasured variables by altering the network (and hence reducing the number of nodes) in such a way so as to cancel out

unmeasured streams. Thereby, leaving only fixed and measured streams upon which allocation can be easily performed on each

node. It should be duly noted that without the matrix projection technology, production allocation could not be as elegantly solved

using a general network and thus being identical to the reconciliation network. Without it, a more complicated computer search

would be required to find the projected nodes (Vaclavek (1969)). A final check of the allocation procedure is to make sure that

when all of the allocation results are included in the network, each of the nodes obey the law of conservation of matter. If not, then

we must report the nodes which are in violation.

Illustrative Example

Figure 1 presents a simplified flowsheet that could be taken to represent a variety of different production processes. Two suppliers,

N1and N2 provide material to a process plant, with some amount consumed as fuel (sent to N10) in the transmission process. All

5

parties have deemed that the flow meter on S3 is correct and that each supplier should bear the fuel cost in proportion to what they

supply. Further, any flow metering discrepancy between the flow meter on S3 and the supplier flow meters should also be prorated

back to the suppliers. The net supply from N1and N2 will be calculated by a simple allocation of the flow meter on S3. Similarly

at the back end of the plant, consumers N7, N8 and N9 each have agreements. N9 is an external consumer with a high quality flow

meter and must be billed exactly per meter. N7 and N8 are company-owned and operated consumers and will each be deemed to

take a share of the production and the total loss (shown as a flow to N11) for accounting purposes. In this example the flow meter

on S3 is the official meter for allocation and a simple representation of some plant internals is highlighted around N4 and N5.

Figure 1. Common network for both reconciliation and allocation.

Table 1 provides the type of stream in the network when reconciliation and allocation are to be performed. In this example there

are no streams for reconciliation treated as fixed. Instead, the two high quality flow meters are given a smaller reconciliation

tolerance or measurement uncertainty as shown in Table 1.

Table 1. Stream Treatment for Reconciliation and Allocation.

Stream Reconciliation Tolerance (%) Allocation

S1 Measured 5.0 Measured

S2 Measured 5.0 Measured

S3 Measured 0.5 Fixed

S4 Measured 5.0 Measured

S5 Unmeasured N/A Unmeasured

S6 Measured 5.0 Measured

S7 Measured 5.0 Measured

S8 Measured 5.0 Measured

S9 Measured 0.5 Fixed

S10 Measured 5.0 Excluded

S11 Measured 5.0 Excluded

S1

S2 S9

S3 S4

S5

S6 S7

S8

S11 S10

N2

N1

N3

N10 N11

N4 N5 N8

N7

N9

N6

Could be combined for both reconciliation and allocation.

Transshipment Node Source or Sink Node Allocation Excluded Node

6

After reconciliation, the objective function for the sum of squares of weighted adjustments (measured minus reconciled squared)

was calculated at 13.05 which is above the background noise threshold value3 of 7.81indicating at least one gross error in the

system. Table 2 displays the measured, reconciled and individual error statistics for each stream.

Table 2. Stream flow values after reconciliation with gross error statistics.

Stream Measured

(m3)

Reconciled

(m3)

Gross Error

Statistic

S1 98.0 97.07 2.29

S2 109.0 114.22 2.60

S3 214.0 213.45 3.41

S4 89.0 89.00 1.37

S5 - 124.45 -

S6 210.0 213.45 0.66

S7 70.0 73.38 2.29

S8 38.0 38.99 2.29

S9 97.0 102.22 2.60

S10 3.0 2.99 2.60

S11 4.0 4.01 2.29

The bold type on stream S3 highlights that this measurement is statistically worse than the rest and is the most probable candidate

to be in gross error. In fact, a subtle gross error or defect was injected into this measurement for the sake of demonstration.

If it can be determined that the meter has indeed failed (e.g. an examination of the meter reading over time may reveal a failure

symptom of displaying precisely the same value over the time interval of interest). If so the best possible value for the meter would

be determined by treating it as an unmeasured flow and using reconciliation to calculate a value. If the measurement for S3 is

discarded and calculated using reconciliation, a value of 208 is attained. This number is the best engineering estimate of the true

value based on all the other measurements.

Table 3 shows the results from allocation with the injected error compared to the results with the S3 set to 208. The lesson from

the comparison is that allocating based on the measured value versus the best engineering estimate is that we may be overstating

the amount from each supplier. This may be contractually obligated, but in such a situation we can at minimum request

maintenance or recalibration of the critical flow meter. Allocation alone would not recognize a flow metering problem until the

injected gross error was much more substantial.

3 Determined using what is known as the Chi-Squared statistic (Crowe et. al. (1983)).

7

Table 3. Stream flow values after allocation with and without gross error in S3 flow meter.

Stream Allocated with S3 as

Measured (m3)

Allocated with S3 Fixed at

Best Estimate (m3)

Allocation Status

S1 101.31 98.47 Allocated

S2 112.69 109.53 Allocated

S3 214 208 Fixed

S4 - - Not-Allocated

S5 - - Not-Allocated

S6 214.00 208.00 (Internally) Fixed

S7 75.83 71.94 Allocated

S8 41.17 39.06 Allocated

S9 97 97 Fixed

S10 - - Excluded

S11 - - Excluded

From an end user perspective, the matrix projection used in the allocation process effectively reduces the flowsheet for the plant to

that illustrated in Figure 2. Flows into N3 are allocated to match S3 (a product-side fixed node). Nodes 4 and 5 are effectively

collapsed with a single flow-in and out, where S6 is forced to match S3. This allowed N6 to be allocated as feed-side fixed node

and given that S9 was declared to be fixed caused S7 and S8 to be allocated to match the difference between S6 and S9.

Figure 2. “Effective” network used by allocation.

It is clear from Figure 1 that stream S4 cannot be allocated because of the existence of the unmeasured stream S5 hence the

collapsing of nodes N4 and N5 in Figure 2.

It is natural to question the difference between the allocation and reconciliation processes. Both processes provide corrected

measurements that obey the laws of conservation balances around nodes. Reconciliation is normally used to detect measurement

errors and thus is operated against a rigorous plant model. Allocation operates against a simplified model and follows some

defined calculation rules. Table 4 compares the reconciliation results with the allocation results using the allocation model. This

model is described as Figure 2 with the additional condition that S3 and S9 are both treated as fixed flows set to zero (zero

tolerance versus 0.5%).

S1

S2

S9

S3

S7

S8

N2

N1

N3

N7

N6

S6

N4/5 N8

N9

8

Table 4. Reconciliation results using the allocation model (S4 removed).

Stream Reconciliation applied to

Allocation Model (m3)

Allocated with S3 as

Measured (m3)

S1 101.13 101.31

S2 112.87 112.69

S3 214 214 Fixed

S6 2140 214.00

S7 76.95 75.83

S8 40.05 41.17

S9 97 97 Fixed

S10 0.0 -

S11 0.0 -

In both instances the sum of S1 and S2 equals S3, and the sum of S7 and S8 equals S6 less S9. Note that only minor differences

between the two methods occur where both provide feasible solutions. In any environment where there is no contractual obligation

to follow a defined and simple allocation procedure, reconciliation results may be utilized instead as they will always have a lower

objective function than any other method (i.e., sum of the squares of the deviations weighted by tolerance).

Conclusions

Presented in this article are the fine points of performing allocation and reconciliation using the same network flow model. The

motivating reasons for the effort are single point of model maintenance and improved diagnostics for engineering and business

processes. The procedure of network production allocation relies heavily on the use of the matrix projection method, which is not

surprising given that unmeasured flows by definition cannot be allocated.

References

Crowe, C.M., Garcia, Y.A., and Hrymak, A., “Reconciliation of process flow rates by matrix projection, part I: linear case”, AIChE

Journal, 29, 881-888, (1983).

Crowe, C.M., “Data reconciliation - progress and challenges”, J. Proc. Cont., 6, 89-98, (1996).

Kelly, J.D., “On finding the matrix projection in the data reconciliation solution”, Computers chem. Engng., 22, 1553-1557,

(1998a).

Kelly, J.D., “A regularized solution to the reconciliation of constrained data sets”, Computers chem. Engng., 22, 1771-1788,

(1998b).

Kelly, J.D., “The necessity of data reconciliation: some practical issues”, NPRA Computer Conference, Chicago, Illinois,

November (2000).

Vaclavek, V., “Studies on system engineering III. Optimal choice of the balance measurements in complicated chemical

engineering systems”, Chem. Engng. Sci., 24, 947-955, (1969).