Regression Estimates in Federal Welfare Quality Control Programs: CommentAuthor(s): Thomas CoxSource: Journal of the American Statistical Association, Vol. 85, No. 411 (Sep., 1990), pp. 868-869Published by: American Statistical AssociationStable URL: http://www.jstor.org/stable/2290029 .

Accessed: 14/06/2014 22:50

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact support@jstor.org.

.

American Statistical Association is collaborating with JSTOR to digitize, preserve and extend access to Journalof the American Statistical Association.

http://www.jstor.org

This content downloaded from 185.2.32.134 on Sat, 14 Jun 2014 22:50:49 PMAll use subject to JSTOR Terms and Conditions

Comment THOMAS COX*

It is a pleasure to comment on the work of Hansen and Tepping and the public welfare quality control (QC) effort. Although I am a statistician, I have also worked as a public welfare caseworker and as a VISTA (Volunteer In Service to America) in a welfare rights organization. I hope, in my comments, to pass on some insights based on those experiences. On the whole, the work of Hansen and Tep- ping has been careful, precise, and sound. They deserve credit for trying to develop a public welfare QC program. There are better ways to approach the problem, but they have made important contributions to measurement and estimation that will serve us well.

My concern is the overall effect on program quality of efforts to reduce the payment error rates (PER's). Based on my knowledge of the public welfare system, I believe that a deeper level of statistical inquiry is appropriate. Statisticians can bring unique skills to public welfare re- search and to the public debate on public welfare funding. These skills include asking the right questions, challenging incorrect assumptions, collecting the right data, and mak- ing accurate and useful inferences. If we do not use these skills, we will not improve the quality of decision making or the public debate. My comments refer to areas in which the use of these skills is lacking.

CAUSALITY, THE PER, AND PUBLIC WELFARE PROGRAM OBJECTIVES

The public welfare programs exist to distribute money to people who do not have enough money. If the public welfare programs fail to distribute enough money to the poor, or if they distribute too much, then they are failures. Of the two, the first is the most serious. The PER only measures whether they deliver too much money and thus addresses a small aspect of the delivery process. Not ad- dressing the broader purpose of the public welfare pro- grams is the greatest failing of the current program.

The PER is one of many variables to use in public wel- fare QC. Is the reason for selecting only the PER the assumption that massive waste, fraud, and abuse occurs? If so, we should verify this assumption. What is the evi- dence? Since the public welfare programs are politically volatile, the scientific community should use scientific principles and methods to confirm or refute such assump- tions. The QC variables selected should be the best pos- sible, and they should pass a scientific rather than a political litmus test.

In selecting QC measures we should consider the chance that the use of the PER alone will result in unintended and undesirable consequences. There has been a dramatic

* Thomas Cox is a statistical and social work consultant, currently enrolled in the Ph.D. program in statistics at Temple University. The author thanks Alan Izenman for introducing him to the subject and for helpful guidance along the way. Thanks also to M. Sobel, J. E. Nowa- kowski, John Rolph, and the referees for their valued comments and suggestions.

rise in homelessness during the last eight years as the ef- forts to reduce the PER's increased; is there a connection? If the PER reduction effort has had a negative impact on important public welfare goals, we should correct the problem.

EARLY PROBLEMS AND DESIGN ISSUES

In their early work, Hansen and Tepping apparently misunderstood the eligibility and benefit determination processes of the public welfare system. They made an error that, though corrected in 1982, highlights a serious prob- lem with the current program. They assumed that over- payment errors and eligibility errors were distinct types. Consequently, they created the aberrant points that laid along the axes of the associated scatterplots and used two separate regressions for fitting.

These errors are of one type. Overpayment errors ap- proach eligibility errors as the overpayment approaches 100% of the benefit paid. Though obvious to someone who has conducted an eligibility interview or calculated a welfare grant, this merging is not readily apparent to peo- ple unfamiliar with the public welfare system. If we want to improve the public welfare system, first we need to fully understand it. Underestimating the complexity of our task and of the system will cause the QC programs we design to worsen rather than improve the public welfare system.

OF NUMERATORS, DENOMINATORS, AND OF CABBAGES AND KINGS

The PER is not an effective tool for management, con- trol, or improvement. A new, two-component error rate (TCER) is more effective for these purposes. The first component would measure the occurrence of rate of cases with errors per 100 sampled cases. The second component would measure the average size of an error. Whereas the PER does not distinguish between 20 $6.00 errors and 2 $60.00 errors, the TCER would. Unlike the PER, the TCER improves our ability to diagnose and resolve spe- cific problems. With additional refinements, the TCER could pinpoint major types of error with separate occur- rence components and facilitate rapid correction of prob- lems.

For any error rate there is still a question of how much quality is enough? Is a 0% error rate necessary? Is a rate of 1%, 2%, 3%, or 5% acceptable? How much should we spend to reduce the error rate? We need to develop suit- able yardsticks that vary from state to state, agency to agency, and office to office as error control resources vary.

? 1990 American Statistical Association Journal of the American Statistical Association

September 1990, Vol. 85, No. 411, Quality Control of Welfare Programs

868

This content downloaded from 185.2.32.134 on Sat, 14 Jun 2014 22:50:49 PMAll use subject to JSTOR Terms and Conditions

Cox: Comment 869

IS THE CURRENT PROGRAM UNBIASED FOR CLIENTS?

There are two types of errors that result in underpay- ments to clients. The first is when public welfare agencies deny all benefits to an eligible client. The second is when a client receives some, but not all, benefits. Failing to deal with underpayments has led to a biased program. Bias occurs because clients can be hurt but not helped by "im- provements" in program quality. It is inappropriate to impose biased penalties on the states and no less inappro- priate to impose a "no win" situation on clients. Fortu- nately, despite much neglect of this topic, the results of client-initiated fair hearings provide data for such an analysis.

The fair hearing is a quasi-judicial, administrative pro- cess. Clients may request a fair hearing if they believe that their benefits are lower than appropriate. As a VISTA I represented, or advised, hundreds of clients in such ac- tions. Most of those actions resulted in decisions that fa- vored the client. Although there are problems with the

use of the fair hearing data, both the fair hearing results and the underpayment problem deserve more attention than they have had. At the minimum, we should study the number of fair hearing complaints before and after the current QC program began. By analyzing the outcome of fair hearing complaints we will be able to quantify the underpayment phenomena and any adverse effects of the PER reduction program.

CONCLUSION

A sense of purposelessness hangs heavily over the cur- rent QC program. The data we are collecting and analyzing do not meet the needs of the Congress, program admin- istrators, or agency personnel. A comprehensive quality improvement program that addresses all of the purposes of public welfare programs is urgently needed. The Na- tional Academy of Sciences report gives some useful sug- gestions in this direction. If we conscientiously move in this direction, we will improve the public welfare programs and, in the long run, reduce their costs.

omment

Hansen and Tepping have given us an article that clar- ifies several issues in sampling for quality control in federal welfare programs, in particular the helpful role of double sampling. The article, however, is written from a tradi- tional design-based viewpoint and ignores the possible rel- evance of model-based theory, which in recent years has offered an alternative to, and challenged the foundations of, design-based sampling. See Royall and Cumberland (1981) and references therein; for a counterview, see Han- sen, Madow, and Tepping (1983).

In particular, the design viewpoint underlies the sharp dichotomy claimed by Hansen and Tepping in Section 4 between classical regression analysis and regression-like procedures in sampling. Design-based considerations ren- der as naive the unease we feel in applying the regression estimator to data typified in Figure 1 of Hansen and Tep- ping. A model-based approach regards that unease as well founded.

From a model-based viewpoint, some observations that may be of interest follow. For simplicity, "single" sampling is assumed, that is, the total enumeration of the auxiliary variable x (state estimate of overpayments). This does not affect the basic thrust of the remarks.

A natural model for the data of Figure 1 is Y = Y(x)

* Alan H. Dorfman is Assistant Professor, Department of Mathe- matics, Tulane University, New Orleans, LA 70118.

given by three components:

Y(O) = VU, V- Bernoulli(p), U- (,u, z2);

Y(x) = V*U*(x), V* - Bernoulli(p.*),

U*(x) - (a + A3x, u2), 0 < X < Xo;

Y(x) - (a + /AX, c2), Xo C X.

Descriptively, the three components correspond to (a) the large group of 0 overpayments, as assessed by the state, of which the large proportion is 0, as assessed by the fed- eral government, and the rest is distributed over a range of positive values; (b) a group of low overpayments as assessed by the state, of which a portion are judged non- overpayments by the federal government, and for the re- mainder of which, federal and state estimates are in substantial agreement; (c) a group of large overpayments, for which there is substantial agreement.

Then it is straightforward to show

E(Y I x) = pp, X = O, = p*a + p*3x, O < X < X0,

a + flx, XO ' X

? 1990 American Statistical Association Journal of the American Statistical Association

September 1990, Vol. 85, No. 411, Quality Control of Welfare Programs

This content downloaded from 185.2.32.134 on Sat, 14 Jun 2014 22:50:49 PMAll use subject to JSTOR Terms and Conditions