Why Measure Quality?

Copyright ©2015 Ron Dolin1

Why Measure Quality?

Chicago Legal Innovation

Ron DolinStanford Law School

Notre Dame Law School

April 16, 2015


Quality Control (QC) at Google Purpose

− Improving core service (ads, search results)− Efficiency gains w/o harm− Preventing/discovering hidden problems

Method− Manual setup− Automated testing− Scoring Result Quality (subjective/objective)− Internal (absolute)− External (relative)


Legal Services QC Goals Clients

− compare (component) services− value-driven purchasing− ROI

New Entrants− validation− market segmentation

Incumbants− high standards (not via proxy)− efficiency (w/o loss of quality)− staff performance metrics


Current QC Inadequate

Subjective Coarse grain

− outcome aggregates (e.g., annual pay-out) Proxy

− reputation− pedigree

Manual


Necessary Methodology

Manual− hierarchical, direct, per-item− peer− sampling− fuzzy

Automated− training data− scoring function (e.g. bar exam essay scores)− tuning and testing procedures


ODR Evaluation – New Approach

Online divorce mediation: # MSA's eBay:

− 80% (of 60M!) resolved via automation− closed ecosystem amenable to optimization− implementing feedback is within eBay's

control− lower # disputes per transaction (via

feedback) DSD – Dispute Systems Design – prevention!

− negotiating interests, rights, or power


Benchmarks

Value = Quality / Cost Measurable quality metrics Can't compare without the same quality metrics Who has the money and power to drive this? Empirical legal studies: benchmarks Human evaluation → metrics → tuning All components of the legal system


Question

When is doing nothing better than doing something?

Examples:− bad wills worse than probate− bad contracts worse than UCC defaults− wrong party pays worse than no one pays


Legal Technology Framework

Factor 1: When/where to use

Factor 2: Legal and user requirements, design

Factor 3: How to evaluate, compare to non-tech method

Factor 4: Impact on the legal system


Evaluation – Beyond Coin Tosses

Goals:− efficiency− quality (e.g. fairness – process, outcome)− accuracy− accessibility

How much quality is efficiency worth?


ODR Evaluation – The Literature Literature:

− Outcome, process− Alternative systems− % settling− Cost (time, $) to participants and system

Problems− Comparing online with offline

measured offline? (legal tech generally) we can't send same case through both

− Subjective vs. objective criteria


What Is Success?

Rebecca Sandefur: “The problem is those people who think they understand what to do about their problem... but don't."

Is procedural fairness sufficient?

In order to apply machine learning, we must pick an objective function.


Factor 4: Impact On The Legal SystemOverly Simplistic Approach

How to apply feedback to the legal system? Which metrics to optimize – reduce # disputes?

− less incarceration? (death penalty for tickets)− less lawsuits? (shut down the judiciary)− less divorces? (outlaw marriage – the cause

of the problem is outside the institution that has to deal with it)

Why Measure Quality?

Law

Transcript of Why Measure Quality?