1

”You get what you measure” or not?

Challenges for fact-based quality management

15.3.2008

Juhani AnttilaVenture Knowledgist – Quality Integrationwww.QualityIntegration.biz , juhani.anttila@telecon.fi

These pages are licensed underCreative Commons Nimeä 2.5 lisenssi

http://creativecommons.org/licenses/by/2.5/deed.fi

2

1. Searching a balanced position to statistical quality management

2. Factual business performance

3. Facts, data, information and knowledge for business management

4. Using measurements, analyses, and knowledge within business

management

5. Example: Using ZEF methodology for business performance evaluations

6. A topical problem: Quality of networks

7. Summary and conclusions

”You get what you measure (*)” or not?

Challenges for fact-based quality management

(*) 32500 Google hitsxxxx/2.3.2008/jan

3

What am I striving for?

3462/2.3.2008/jan

StatisticsMathematical science pertaining to the

collection, analysis, interpretation orexplanation, and presentation of data

Quality management (Quality of management)Degree to which coordinated activities of management to direct and control the organization

fulfill the needs and expectations of organization’s all stakeholders

Principles of good managementFactual approach to decision making (ISO 9000 - Quality management principles)

Management by fact (Malcolm Baldrige - Core values and concepts)Management by processes and facts (EFQM - Fundamental concepts of excellence)

Business methodology

Business performance management

Measuring business process performance

QualityDegree to which a set of inherentcharacteristics fulfils needs and

expectations

4

Positioning to statistics in quality management

3461/16.2.2008/jan

Quality managementtraditions:-Statistical processcontrol-SixSigma-Problem solving-Taguchi techniques

Constraints:- Theoretical /mathematicaldifficulties- Problems- Warnings

Walter Shewhart

1931

Darrell Huff

1954

How to find balance?

5

Information problems and danger of empirism

(Ref.. Jaana Venkula, Huff, etc.)3465/14.2.2008/jan

• Data hell – Chaotic data disturbance

• Data bulimia or data anorexia

• Data mechanization – overly emphasized

information technology and impoverished

skills

• Knowledge splintering into objective

information pieces

• Data pedantry and data naiveté

• Popularization – Complex realities overly

simplified and information contents

impoverished

• Information insecurity

• Information capitalism or information

proletariat – misuse of information power

• Information terror and data crime

6

Warning about the danger of information (*)

Information is not knowledge. Knowledge comes from theory. Without theory, there is no way

to use the information that comes to us on the instant. There is no true value of any

characteristic, state, or observation.

Warning about people rewarding based on quantitative measurements, deadly disease #3 of

leadership:

“Evaluation of performance, merit rating, or annual review nourish short-term performance,

annihilates long-term planning, builds fear, demolishes teamwork, nourishes rivalry and

politics. It leaves people bitter, crushed, bruised, battered, desolate, despondent, dejected,

feeling inferior, some even depressed, unfit for work for weeks after receipt of rating, unable

to comprehend why they are inferior. It is unfair, as it ascribes to the people in a group

differences that may be caused totally by the system that they work in.”

More important than to create scrupulous and even harmful business measurement systems,

is to get tacit knowledge of business leaders and workers effectively interact with each other

for creating extensive profound knowledge within an organization.

3466/14.2.2008/jan (*) W. Edwards Deming (e.g. in "The new economics")

7

Pragmatic (*) genuine knowledge

Pragmatism (C.S. Peirce): Action based information / knowledgeapproach:

• Genuine knowledge makes possible to act in a newmeaningful way.

• Knowledge can be genuine only if it has been tried andjustified in operation.

• Knowledge is a basis of an organization and a communityand their practical operations.

• Knowledge cannot be objective.

”The knowledge you take

is equal to the knowledge you make.” (**)

2475/4.3.2008/jan (Ref.: Jaana Venkula) (**) Lennon, McCartney

Charles Sanders Peirce

(*) Ref. the other paradigms of knowledge theories:

- Heuristic (internal comprehension based knowledge)

- Empirical (external observation and experience based knowledge)

8

Quantitative indication of performance:

- Phenomenon to be considered of a business interest

and its character – especially in business processes

- Measure / indicator, characteristic, quantity

- Metrics, unit of measure

- Numerical values and value range of the quantity

Practicalities of performance management:

- Target / required values, observed values

- Meter, gauge, means to measure / observe the quantity

- Measurement, assessment technique

- Uncertainty

- Conclusions, decisions and actions based on

measurements

- Approach and practices for performance improvement

Quantitative approach to driving business

performance

1613/10.3.2008/jan

9

Management areas:

- Operational management

- Strategic management

Scope of measurements:

- Performance of business processes

- Performance of products (goods andservices)

Related aspects:

- Measurements management

- Business performance management

3467/2.2.2008/jan

Performance measurements for businessmanagement – Need of business information

Purpose of measurements:

- Research activities for getting newknowledge

- Acquisition of information for planningbusiness or operations

- Controlling operations and processes

!Performance monitoring

!Determination measurements

!Verification measurements,hypothesis testing (AQL / LQ)

!Validation testing

- Measurements for problem solvingand performance improvement: Kaizen,breakthrough and learning

- Measurements for quality assurance

10

From business activities to overall performance of

an organization and to performance excellence

0877/22.3.2008/jan (Ref.: Malcolm Baldrige Criteria)

Financialmeasures (data)

Non-financialmeasures (data)

DOMAIN OF ACTION - BUSINESS ACTIVITY

(Operational business processes and projects)

Customer-focused

performance

Operationalperformance- processes

- workforce

Financial andmarketplaceperformance

Overall business performance

References:Competitive performance

PERFORMANCEEXCELLENCE!

Excellence is originated from the business activity not only being superficially outstanding

Productperformance

- goods - services

}

11

Interesting information on business performance

• Information available from all key business areas and processes and relatedrequirements:

– Consistent and balanced quantitative data through measures andindicators of business related facts

• Level of performance

• Rate of improvement, or slope of trends in data demonstrating developmentof business performance

– ”Three generation reporting”: Past, now, future

– Pattern of development: Linear, non-linear, spiral

• Comparisions of performance levels and their development with owntargets, competitors, and World-Class benchmarks

• Relationships of achieved quantitative business results with managementand process activities: Cause-effect linkages

3468/2.2.2008/jan (Ref.: Malcolm Baldrige Criteria)

12

Temporal dimensions of the present moment “Now”

2583/5.3.2008/jan

The present “Now” always includes the

impacts of the past events and the

challenges of the future events!

The ‘Now’ of future events and performance

The ‘Now’ of past events and performance

The ‘Now’ of present events and performance } Kairos Kronos

Time

13

What is information security all about?

2476/11.2.2008/jan

Basic concepts characterizing security of the information consist of:

• Integrity – information that one is using for his/her actions is accurate

• Availability – one has access to the relevant information when and so long it is

needed

• Confidentiality – information one is using is not manipulated by anybody else.

Additionally authenticity (information is authentic, trustworthy, or genuine) and

authority (one has right to use the information) aspects are significant especially

when using the means of electrical communication.

The most essential information security

aspect is privacy!

(Ref.: ISO/IEC 27001 / 27002)

14

You don’t get necessarily what you measure

Measurements in all kinds of business activities are important for management but in

practice:

• Measurements are often excessive, unfocused or wrong (*)

• Business results you get don’t necessarily relate to what you measure

• What you get through measurements is not necessarily meaningful or good for

organization or people being managed• Business leaders may not know what they don’t know (“Tacit ignorance”)

Measurements results represent explicit knowledge of the business performance but

business intentions and real business results are principally of tacit knowledge of

business leaders and workers in the organization.

No measurements can be objective but they are always affected by the intension and

awareness of somebody – even in physics objectivity is impossible. What is being

measured, by what kinds of means or methodology, what is obtained through

measurements, and how the measurements results are understood – they all depend on

somebody’s intention and awareness.

3469/5.3.2008/jan

(*) Dr. Juran:

“Vital few” / “Trivial many”

15

Facts & knowledge for managing business activities

Measuring

Data

Analysing

Information

Reflecting and decidingIntervention

Plan / Act

Effects

0609/25.3.2006/jan

Knowledge- explicit records- tacit knowledge(know-how, competence)

The performance reality of the company business processes

A P

DC

...

Wisdom- myths- values

”Ba”

Environments

Facts

You get what

you measure

16

2343x/15.8.2007/jan

PDCA model (*) for a good management: Coordinatedactivities to direct and control an organization (**)

ACTING:• Preventing actions• Improving actions• Re-engineering• Communicating• Recognizing andrewarding

PLANNING:• Business andmanagement models• Business plan• Approaches andmethodology

CHECKING:• Assessing theperformance• Reviewing theperformance

DOING:• Deploying the approachand achieving the results• Controlling operationalperformance• Corrective actions

A P

C D

P = Planning

D = Doing

C = Checking

A = Acting

Applying PDCA model:

• Rational control (operational)

• Continual rational small step

improvement (operational),

“Kaizen” approach

• Innovative breakthrough

changes (strategic) (***)

(*) Deming / Shewhart Cycle (**) ISO 9000 (***) Dr. Shiba

All three PDCA loops modes and

four PDCA phases are strongly

information / knowledge dependent

17

Control of the business activities

- feedback

Operations

(Business processes)

Feedback

Tfb

Top

Business

requirement

input

Business output

Notes:

• Requirements for control :

Throughput times Tfb<<Top

• ”A decision must be made

at lowest possible level in

the organization and as close

as possible to where its

outcome will be executed.”

(Peter Drucker)Target

1847/26.2.2008/jan (Ref.: Bode, Asby)

Business

control

input

18

Traditional ”quality tools” have a long history and are very strongly related with data prosessing andmanagement of business operations. Assortment of the methods may seem confusing, despite that theyhave been arranged into different groups such as Seven basic quality tools, Seven new management andplanning tools, 127 tools, Kaizen tools, SixSigma tools, etc. tools. These tools include e.g.:

Simple basic tools:•Data collection tools•Idea creation, brainstormingtools, nominal group technique•Check lists / sheets•Survey methods•Affinity diagram•Matrix diagram•Histogram / bar graph•Scatter plots•Relations diagram•Cause-effect, fishbone orIshikawa diagram•Pareto chart•Evaluation and decision-making tools, decision three,force-field diagram•Voting methods

3470/26.2.2008/jan

Process management tools:•Control charts•IDEF0•Process capability methods•Flow charts

Project management tools:•Pert / Gantt chart

Reliability (dependability) tools:•Failure mode effects andcriticality analysis (FMECA)•Fault tree analysis (FTA•Reliability prediction•Reliability testing

Inspection, testing oracceptance methods:•Attribute testing•Variable testing

Data analysis methods:•Statistical tests•Variance analysis•Regression analysis•Design of experiments•Z-score analysis•Value analysis

Traditional quality tools

Recommendation: Organizations should establish and maintaintheir tool sets for quality management, “Business excellence toolkit”, as well as the measurement and information management atlarge.

19

Enablers criteria

Category 4: Measurement, analysis, and knowledge management:

!Measurement, analysis, and improvement of organizational performance

(Assessment area 4.1)

!4.2 Management of information, information technology, and knowledge

(Assessment area 4.1)

Results criteria:

Category 7: Results

!Product outcomes (Assessment area 7.1)

!Customer-focused outcomes (Assessment area 7.2)

!Financial and market outcomes (Assessment area 7.3)

!Workforce-focused outcomes (Assessment area 7.4)

!Process effectiveness outcomes (Assessment area 7.5)

!Leadership outcomes (Assessment area 7.6)

Using measurements, analyses, and knowledge

within business management (*)

(*) Malcolm Baldrige Criteria 20083472/.20.3.2008/jan

20

Enablers criteria:

!Criterion 2: Policy and strategy

- Policy and strategy are based on information from performance measurement,

research, learning and external related activities (Criterion part 2b)

!Criterion 4: Partnership and resources

- Information and knowledge are managed (Criterion part 4c)

Results criteria:

!Customer results (Criterion 6)

!People results (Criterion 7)

!Society results (Criterion 8)

!Key performance results (Criterion 9)

Using measurements, analyses, and knowledge

within business management (*)

(*) EFQM Criteria3473/.20.3.2008/jan

21

Information technology serves our knowing and

learning as an extension of the body

Impact of technology:

The form of a medium imbeds itself in the message,

creating a symbiotic relationship by which the medium

influences how the message is perceived, creating

subtle change over time.

We shape our tools and afterwards our tools shape us.

Use of technology:

– Collecting data

– Processing data and information

– Storing data and information

– Sharing / communicating information and

knowledge

(IT = Interactive technology)

3474/2.1.2008 /jan

Marshall McLuhan 1911-1980

22

Collaborative knowledge working / learning

environment with Web 2.0 tools, “Enterprise 2.0”

3372/15.9.2007/jan

Collaborative knowledge working environment for activity XXX at organization XYZ:

Group-members

Knowledge feeds from group-members'findings, observations, and reflection,and comments(Blog)

Automatic informationfeeds from external sources, e.g. Internet (Aggregators)

Documents files

Discussion forums

Common shared knowledge-base accumulated,discussed, and updated by the group-members(Wiki)

" Links to• Other tools (calendar, to-do list, diary,Skype, virtual meeting room, etc.)• Administration of the environment

23

Web 2.0 challenges

3460/2.3.2008/jan

Web 2.0: Second-generation of Internet-based services - such as social networking sites,

wikis, communication tools, and folksonomies - that let people collaborate and share

information online in ways previously unavailable

RSS: (Rich Site Summary) Really Simple Syndication

Mashup: A website or application that combines content from more than one source into

an integrated experience

Tag cloud: A visual depiction of content tags used on a website

Facebook: a social utility that connects people with friends and others who work, study

and live around them

RTE – Real Time Economy: Speeding up information flow between companies, each

having capabilities to monitor their businesses continuously and quickly react to changes

and exceptions

(Ref.: http://www.oreilly.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html )

24

Example: ZEF evaluation methodology

3336/.20.3.2008/jan

ZEF is an advanced interactive evaluation methodology and tool that is originated from strong core

competences of evaluation technique and human interface technology.

Origin:

• ZEF methodology was developed in Finland at Oulu University as Z-scored Electronic

Feedback (= ZEF) tool. Effectiveness of its two dimensional evaluation structure was

validated by University of Lapland.

• Commercial ZEF solution is a product of Oulu based company ZEF Solutions Inc.

Usage:

• ZEF is an all-purpose tool that is suitable for all kinds of organizations.

• ZEF evaluation services are web-based solutions operating as SaaS (Software as a Service).

With ZEF on-line tool one may collect, compare and evaluate data effectively and efficiently

from even a big group of people.

• ZEF tool has been used by a lot of organizations for business information and hundreds

thousands of ordinary citizens in several survey-cases of general societal interest.

(Ref.: http://zefsolutions.com/www/english , http://qiblog.blogspot.com/2007/06/carry-out-appropriate-surveys.html )

25

ZEF tool is applicable for:• All kinds of quality-related performance evaluation, feedback and surveyapplications

!E.g. 3-In-1 assessment scheme that combines Malcolm Baldrige model,EFQM model and ISO 9000 and is easy to tailor to specific business aspectsor organizational strategic priorities. Examples include modifications for

!Health services quality management (CEN/TS 15224)!Food safety management (ISO 22000)!Information technology service management (ISO/IEC 20000)

!Process performance evaluations!Customer / workforce satisfaction

• Other business related assessments needs including!Innovations evaluation!Risk, vulnerability and threat analyses!Strategy analyses!Customer lead surveys

• Other general e.g. social, entertainment, etc. applications including!National innovation policy surveys!Parliamentary / presidential election comparisons!Education and school evaluations!Idols rating

ZEF evaluations

3476/20.3.2008/jan

26

3288/20.3.2008/jan (*) ZEF Solutions Inc., Finland: www.zef.fi

Improving effectiveness / efficiency of self-assessmentswith an advanced tool (ZEF)

27

Performance evaluation for excellence

through a multi-faceted and consistent approach

1. Leadership

4. Measurement, analysis,and knowledge management

7. Businessresults

6. Processmanagement

3. Customer andmarket focus

5. Humanresource focus

2. Strategicmanagement

3325/26.3.2008/jan

Overall business performance (e.g. 3-In-1 criteria)

Board operationsStrategic performance

Corporate securityRisk management

Networking capability

Process performanceProject performance

Supplier performanceQuality Assurance:

ISO 9001, AQAPEnvironmental management

People satisfaction360 feedback

OHS assessment

Information securityInnovations

IT Governance

Customer satisfactionMarket communication

Leadership performanceSocial responsibility Product

performance

28

Category 4: Measurement, analysis, andknowledge management1. We follow up fulfilment of targets anddevelopment of performance on company andprocess levels with adequate measures andindicators.2. We follow up effectiveness and efficiency ofour business processes.3. We have easily available analyzed andcurrent information on performance of ourcompany and its development against targetsand other references.4. Our real-time internal accounting operatesin advance.5. Our employees measure, analyze andevaluate quality of their own work in order tomake decisions and initiate improvementactions for their work in line with ourcompany-wide plans. …

… Etc. other assessment items

Assessesment scheme “3in1”: EFQM, Malcom

Baldrige and ISO9000 combined in one scheme

Results of ZEF evaluations (“3-in-1”)

3475/20.3.2008/jan

29

Results of ZEF evaluations

3334/.20.8.2007/jan

30

Statistical inferences from ZEF evaluations

3471/.20.3.2008/jan

Detailed business information is obtained from ZEF raw data through statistical treatments.All individual respondent answers to the query items in the ZEF evaluations are recorded asseparate data pieces that are then used for statistical analyses as needed:

! The basic ZEF report shows the means and standard deviations of all respondents’answers per query items in one or two dimensions as needed.

! The result data may be normalized according Z-scored transformation that is useful toemphasize relative differences among query items.

! Answers from respondents may be grouped in arbitrary way or presented asindividual answers.

! Results of different ZEF evaluations, e.g. of different points in time, may be comparedby using ZEF’s “comparison engine” function.

! Raw data of evaluations may be presented in the formats of Excel (.xls) or"Concurrent Versioning System" (.cvs) in order to facilitate the use of moresophisticated statistical tools, e.g. SPSS, for data analyses.

ZEF tool’s interactive user-interface enhances user’s deliberation when he/she is respondingto the query items and also enhances understanding the relative differences among queryitems. That makes answering more motivated and justified.

31

Z-score transformation

3478/20.3.2008/jan

Z-score for an item indicates how far and in what direction that item deviates from its distribution's

mean expressed in units of its distribution's standard deviation. The z-score transformation is such

that if every item in a distribution is converted to its z score, the transformed scores will have a

mean of zero and a standard deviation of one.

Z-score transformation is especially useful when seeking to compare the relative standings of

items from distributions with different means or different standard deviations.

(Ref.: http://www.sysurvey.com/tips/statistics/zscore.htm )

32

Z-scored normalized evaluation results (ZEF)

3477/.20.3.2008/jan

33

Summary and conclusions

3479/.20.3.2008/jan

1. Competitiveness and success of all organizations are based on right business relatedinformation and knowledge on time.

2. Impressive business phenomena and their nature including stochastic features should betaken into account in planning business performance measurements.

3. Business related measurements and information may have many different purposes.4. One should recognize vital few valid subjects for business performance measurements.5. Measurements should be based on sound methodologies and effective measuring tools.6. Observed data should be understood in the business context and also data’s statistical

viewpoints should be noted.7. Tacit knowledge and explicit information should be brought together in making

conclusions and activating managerial measures.8. Interactive information technology gives remarkable advantage and serves as an

extension of our body and senses both in making observations and in using informationwithin business communities.

9. Cooperation of quality and statistics experts is obviously useful.

34

Overall quality of a network

3464/15.2.2008/jan

S

A A

Si

j

Overall quality of a network Qvm:

Degree to which the network fulfils needs andexpectations of all network members:

- m members in the network

- Each member gets benefits (Si) but alsoloses something (Ai).

Knots with

few links

Knots with

many links

Power law of a scale-free network:

Number of links (k)

Nu

mb

er o

f kn

ots

wit

h k

lin

ks

A topical problem:

How to measure in practice

the overall quality of a network?

Win / Win in network

Network

member i

Network

member j

35

xx/15.2.2008/jan

Please, evaluate my presentation

and material and

give your feedback with ZEF:

http://www.zef.fi/service/user/?q=597

Thank you!