Lessons in Data and Statistical AnalysisPitfalls and Problems Topics for Consideration: Source Bias...
Transcript of Lessons in Data and Statistical AnalysisPitfalls and Problems Topics for Consideration: Source Bias...
Lessons in Data and
Statistical Analysis
NCSL Fiscal Analysts Seminar
October 5, 2017
Data types, relevance and reliability
Useful data sources
Tools
Examples
Pitfalls and Problems
Topics for Consideration:
Source Bias
Tips for assessing the credibility of online information
(Center for Public Education, based on Stanford Guidelines for Web Credibility)
1. Does the Web site provide references for research that can be independently verified?
2. Are authors identified? Are their affiliations, credentials, and contact information provided?
3. Who owns or is responsible for the Web site? Is a physical address and complete contact information provided?
4. Does the site describe its mission? Are staff members identified?
5. Does the site carry advertising? If it is run by a not-for-profit organization, are its sources of funding identified?
6. Is the site professional in appearance and quality? How recently has it been updated? Is it free from typographical and grammatical errors?
Evaluating reliability and relevancy
Time Period
Evaluating reliability and relevancy
Unit of Analysis
Evaluating reliability and relevancy
individuals
households
Common Adjustments
Inflation (what year = 100?)
Seasonally-Adjusted (SA)
Seasonally-Adjusted at an Annual Rate (SAAR)
Per Capita
Evaluating reliability and relevancy
Flaws/Errors
Statewide In-Migration vs Utah School Age Enrollment
Statewide In-Migration = 11,000
School Age Enrollment
Growth = 12,161 (School Migration Est. 5,355)
Under estimate = 4,243
Cost = $17 million
Evaluating reliability and relevancy
Useful data sources
• Bureau of Economic Analysis
• Consumer Expenditure Survey
• Bureau of Labor Statistics
• U.S. Census Bureau
• National Bureau of Economic Analysis
• American Economic Association
• State Vital Statistics Offices
• Global Insight
• Moody’s
• Federal Reserve
Data Sources: BLS Consumer Expenditure Survey (CES)
• Cross-section of spending characteristics of US households
• Useful for analysis of sales tax proposals for narrow taxes/ exemptions:
• Exemption bill on diapers/ feminine hygiene products
• Bill expanding sales tax to services
• Federal Reserve Economic Data (time series)
• BEA, BLS, Census, numerous other data releases
• Always up-to-date on latest data releases
• Example: LA Sales Tax
• Able to manipulate data as needed
Data Sources: FRED
Data Sources: Other Agencies
• Federal Reserve Banks and Board of Governors
• Continuously research policy issues and monitor economic trends
• Timely research on net impacts of sales tax holidays
• Crucial for analysis of LA Second Amendment Sales Tax Holiday bill
Data Sources: Other Agencies
• US Energy Information Administration (EIA)
• Continuous research and monitoring
• Timely research on impact of gasoline prices on consumer response
• Useful in analysis of LA Gasoline Tax increase bill
Data Sources: Other Agencies
• US Department of Agriculture• Extensive data on agricultural production
• Current data on farm size and production by state• Vastly improved estimate reliability on proposal exempting
irrigation materials from sales tax
Data Sources: Market Research Firms
• Example: Statista• Limited amount of
free data, but enough US-level info to be useful
• Examples:• Expansion of sales
tax to services including digital media
• Diapers/feminine hygiene products sales tax exemption
Other Data Sources
• Professional or Industry Associations
• Often can be a valuable source of very relevant data (but be aware of their mission/motives)
• Example• AOPA: Aircraft painting
service tax exemption
• Think Tanks or Professional Research Groups
• Sometimes can function similarly to an industry association; same caveats apply
• Example• University of New
Orleans Hospitality Research Center: Louisiana Tax Free Shopping Program for foreign tourists
Analytical Tools
Tools
Application Pros Cons
Excel Ubiquitous
Easy to present
results on the fly
“Intuitive”
Hard to track iterative
process
Size limits (better)
Can be slow
Statistical Packages
(R, STATA, SPSS,
SAS, EVIEWS, …)
Good documentation
of process
Sophisticated
modeling techniques
Large datasets/faster
Steeper learning
curves
Expensive
Output presentation
can be tricky
Statistical Packages: STATA Example – “freduse”
Example
• Utah Tax Reform Bill of 2006• Modified Flat Tax
• Went from 6 bracket to one with deductions being converted to credits.
• Tax Commission had access to the filer data but LFA and Governor did not.
• Tax converted the data into an Excel spreadsheet.
• After the bill passed the Tax Commission discovered an error in the spreadsheet understating the revenue loss by $200 million.
• Resulted in a change in statute to allow LFA to access taxpayer data as long as the taxpayer could not be identified.
• Transitioned from Excel to SAS for analysis.
Taxpayer ReceiptTools | budget.utah.gov
Sales Tax Revenue
Total General Fund Revenue
Individual Income Tax Revenue
Total Education Fund Revenue
Enter your income tax paid: $15,000
Enter your sales tax paid (or "x" to estimate from income tax): x
Total: $21,100 $6,100
Public Education48%
Higher Education15%
Social Services14%
Infrastructure10%
Criminal Justice7%
Economic Development
2%
Environment and Natural Resources
3%
Elected Officials1%
General Government
0%
State Sales and Income Tax Uses
Taxpayer Receipt
Statewide
Agency (in M) Your Share
Grand Total 6,266.4$ 21,100.00$
Public Education Total 3,052.6$ 12,315.75$
Higher Education Total 916.8$ 3,240.75$
Social Services Total 884.4$ 2,113.92$
Infrastructure Total 638.4$ 1,545.76$
Buildings -$ -$
Debt Service 643.5$ 1,557.97$
Transportation (5.1)$ (12.21)$
Criminal Justice Total 427.7$ 1,022.49$
Economic Development Total 130.2$ 344.41$
Environment and Natural Resources Total 101.6$ 242.84$
Elected Officials Total 86.6$ 206.97$
General Government Total 28.0$ 66.97$
$11,961
$3,337 $2,072 $1,515 $1,032
$337 $608 $174 $66 $0
$2,000
$4,000
$6,000
$8,000
$10,000
$12,000
$14,000
Revenue Simulator
Pitfalls and Problems
La Legislative Fiscal Office
Reliance on External Forecasts Can Be ProblematicHere, La wages & salaries income. History sourced to the BEA, but forecasts provided by Moody’s Analytics. The Moody’s forecast of LA income (La variables in general) has been persistently more optimistic than has actually prevailed. Thus, the Moody’s series has to be adjusted throughout the forecast horizon to something more realistic/likely. A simple extrapolation of underlying trend path or near-term historical performance provides a more likely forecast of this variable for use in sales tax forecasting. {Moody’s forcast implies annual growth rates up to 5% before tapering off; Adjusted rate no more than 2.5%}
0.00
20000.00
40000.00
60000.00
80000.00
100000.00
120000.00
140000.00
1990Q1
1990Q4
1991Q3
1992Q2
1993Q1
1993Q4
1994Q3
1995Q2
1996Q1
1996Q4
1997Q3
1998Q2
1999Q1
1999Q4
2000Q3
2001Q2
2002Q1
2002Q4
2003Q3
2004Q2
2005Q1
2005Q4
2006Q3
2007Q2
2008Q1
2008Q4
2009Q3
2010Q2
2011Q1
2011Q4
2012Q3
2013Q2
2014Q1
2014Q4
2015Q3
2016Q2
2017Q1
2017Q4
2018Q3
2019Q2
2020Q1
2020Q4
LaWages&SalariesIncomeHistoricalandForecasts
qtrlyobs,SAAR
Moody'sForecastAdjusted Moody'sForecast
La Legislative Fiscal Office
Reliance on External Forecasts Can Be ProblematicSame issue here, but with regard to oil prices. Moody’s doesn’t have to live on any of its forecasts, but we have to live on ours. External forecasts of oil price seem to be more aggressive than what we are comfortable budgeting. So we adjust them down (an average of Moody’s, EIA, our Natural Resources Dept. – low price scenario, and sometimes NYMEX futures).
0.00
20.00
40.00
60.00
80.00
100.00
120.00
140.00
90:1
91:1
92:1
93:1
94:1
95:1
96:1
97:1
98:1
99:1
00:1
01:1
02:1
03:1
04:1
05:1
06:1
07:1
08:1
09:1
10:1
11:1
12:1
13:1
14:1
15:1
16:1
17:1
18:1
19:1
20:1
21:1
Moody'sAdjusted
Moody'sOilPrice
La Legislative Fiscal Office
Even When Good Data Is Available, It Can Be Used BadlyThe 2015/16 Executive Budget proposed to balance the total proposed spending with existing baseline revenue + the recommended additional revenue below from curtailing usage of certain tax credits. The additional revenue figures were taken from the Revenue Dept. annual publication titled “Tax Exemption Budget”, reporting actual credit usage.
from2014teb
AmountOffsettingTax
AmountRefunded
Inventory Tax Credit
Ad Valorem
Credit for Certain
Natural Gas
Ad Valorem Credit
for Offshore
Vessels
Telephone Co
Property Tax Credit
452,676,421 4,534,210 56,406,978 22,643,842
75,961,191 551,179 13,356,040 17,381,873
376,715,230 3,983,031 43,050,938 5,261,969
A bill addressing these 4 credits expected a fiscal note reflecting $429.1 million of additional revenue for FY15/16 (82% of the $525.9 million total recommendation).
This estimate, used in the budget, came directly from Revenue Dept. data concerning the utilization of these credits. To the left, the latest year available of a multi year table of credit use for these 4 credits and all the others mentioned above (from the Revenue Dept.).
∑ = $429.1 million
La Legislative Fiscal Office
Even When Good Data Is Available, It Can Be Used BadlyThe budget expected $429.1 million for FY15/16 from these 4 credits. This is the fiscal note they got. A little short of that.
What went wrong? The administration looked at the figures provided by the Dept in the Tax Exemption Budget. That document reports exemption losses (credit losses in this case) by state fiscal year. However, the bill reducing the credits was made effective (appropriately so) on the basis of tax/calendar year, “Applicable to tax years beginning on and after January 1, 2015”. No one in the administration seemed to realize this until the fiscal note was generated. That note explained it, as nicely as possible, as follows:
So what did we do? In addition to many other changes that made the bill almost unrecognizable from the Original version, the effectiveness / applicability language was changed to: “Applicable to all claims for these credits on any return filed onor after July 1, 2015, regardless of the tax year to which the return relates”.
La Legislative Fiscal Office
Even When Good Data Is Available, Implementation Can Make Estimates UselessTable below is a sample of near-term history of tax credit usage going into the 2015 legislative session. For many credits history actually went back many years prior. Very good data on actual credit use, annual growth & variation.
Citizens Insurance Credit Inventory Tax Credit
Ad Valorem
Credit for Certain
Natural Gas
Ad Valorem Credit
for Offshore
Vessels
Telephone Co
Property Tax Credit Milk Producers
Technology
Commerciali-
zation
Historic
Residential
Rehab Angel Investor Broadway South
School
Readiness
ChildCare
Provider
Sugarcane
Trailer
Conversion
Research &
Development
from2014teb
AmountOffsettingTax
AmountRefunded
from2013teb
AmountOffsettingTax
AmountRefunded
from2012teb
AmountOffsettingTax
AmountRefunded
from2011teb
AmountOffsettingTax
AmountRefunded
from2010teb
AmountOffsettingTax
AmountRefunded
41,100,487
33,546,054
7,554,433
40,697,004
33,012,766
7,684,238
42,078,539
33,813,798
8,264,741
43,527,560
35,421,468
8,106,092
46,284,427
37,818,083
8,466,344
452,676,421 4,534,210 56,406,978 22,643,842
75,961,191 551,179 13,356,040 17,381,873
376,715,230 3,983,031 43,050,938 5,261,969
419,306,842 4,022,555 41,457,576 24,097,188
77,639,885 1,216,471 5,263,692 15,204,986
341,666,957 2,806,084 36,193,884 8,892,202
381,004,799 3,276,447 43,768,033 24,425,749
65,086,822 866,684 8,394,741 17,817,892
315,917,977 2,409,763 35,373,292 6,607,857
373,990,196 5,407,465 37,454,767 69,304,257
83,704,067 1,108,439 10,254,776 26,628,016
290,286,129 4,299,026 27,199,991 42,676,241
319,710,932 3,825,605 29,795,877 23,306,138
57,553,014 1,129,622 10,430,960 18,386,476
262,157,918 2,695,983 19,364,917 4,919,662
1,555,702 201,377 275,657 1,568,555 8,754,604
112,401 27,030 149,345 690,898 185,497
1,443,301 174,347 126,312 877,657 8,569,107
1,810,000 104,735 303,818 1,819,274 4,950,016
107,934 2,137 180,448 890,545 201,943
1,702,066 102,598 123,370 928,729 4,748,073
1,737,500 104,924 643,476 3,386,710 5,261,983
125,346 20,577 516,380 1,276,181 76,631
1,612,154 84,347 127,096 2,110,529 5,185,352
1,778,750 91,849 250,435 3,493,531 588,663
133,587 8,397 147,909 1,371,238 29,583
1,645,163 83,452 102,526 2,122,293 559,080
1,885,000 317,332 428,553 2,664,283 30,024
148,048 1,851 360,483 1,037,563 21,888
1,736,952 315,481 68,070 1,626,720 8,136
4,662,556
256,260
4,406,296
5,506,820
278,417
5,228,403
5,617,109
261,312
5,355,797
4,685,290
221,578
4,463,712
2,818,400
143,711
2,674,689
2,744,431
23,053
2,721,378
5,733,693
367,868
5,365,825
1,746,142
147,202
1,598,940
880,335
91,732
788,603
197,642
32,656
164,986
24,380,813
794,852
23,585,961
19,703,087
1,444,225
18,258,862
10,795,238
970,834
9,824,404
12,449,259
680,029
11,769,230
2,262,509
62,683
2,199,826
Table 1Summary of Major Revenue Bills, 2015 Session
(in millions)
FY16 Duration Applicable RecoupmentAct 125 / HB 629 Income & Franchise Tax Credits Cut 28% $31.5 3 years All Returns From July 1 3 yearsAct 123 / HB 624 Corporate Income Tax Exclusions and Deductions Cut 28% $122.0 3 years All Returns From July 1 3 yearsAct 133 / HB 805 Five Year Carry-forward of 25% of Inventory Credit $129.0 Permanent All Returns From July 1 None
HCR 8 Suspend Business Utilities Exemption to 1% of Sales Tax $107.2 8/ 27/ 16 Transactions From July 1 NoneAct 94 / HB 119 Increase Cigarette Tax by 50¢/ pack plus vapor products $106.4 Permanent Transactions From July 1 None
Act 109 / HB 402 Equalize Credit for Taxes Paid to Other States $34.0 3 years All Returns From July 1 3 yearsAct 103 / HB 218 Eliminate Net Operating Loss Carry-Backs $29.0 Permanent All Returns From July 1 NoneAct 131 / HB 779 Cap Solar Tax Credit Program $19.0 Permanent All Claims From Jan 1 NoneAct 126 / HB 635 Enterprise Zone Restrictions $5.0 Permanent All Claims From July 1 NoneAct 134 / HB 829 Modify / Cap Film Tax Credit Program $77.0 3 years All Claims From July 1 NoneAct 110 / HB 445 Certificates Of Title Tax Increase $59.5 Permanent Transactions From July 1 NoneAct 147 / SB 271 Reduce Motor Fuels Tax Remitance Discounts $6.0 Permanent Transactions From July 1 NoneAct 109 / SB 93 Prohibits Education Credit If Tuition Deduction Taken $2.3 Permanent From Tax Year 2015 None
Interaction Between Act 123 and Act 103 ($8.0)
Total Additional Revenue Generated $719.9 plus $46.8 million in new fees = $766.6M new money raised
This data and other data on actual usage of credits, deductions, programs was used to pass legislation with estimated effects below.
Ultimately, we couldn’t tell if any of this revenue actually materialized in FY16. Much of it ($404M) was to show up in corporate tax collections.Beginning-of-year forecast = $790M vs end-of-year actual collections = $248M. Look at implementation provisions:(a) 3-year temporary duration (wait us out and receive 100¢ on the dollar), (b) applicable to returns received after 7/1/2015 regardless of tax year of the filing (file quickly and receive 100¢ on the dollar), (c) 3 years to recoup extra tax payments if return filed after 7/1/15 was a final return of an earlier extension (this made the accountants happy).
La Legislative Fiscal Office
Lack of data can lead to data collection effort (FY 2016/17)• Sales tax rate increase and base expansion to tax previously exempt transactions estimated based on current collections from the traditional
base, and on data from the Tax Exemption Budget for exempt transactions. • Only a handful of sales tax exempt transactions are reported in the Exemption Budget b/c there are so many of them.• Some 90+ exempt transactions reported in a catch-all line, “All Other Exempt Sales”. A very large number implying a very large amount of
tax receipt should those transactions be taxed.• Fiscal note on the bill assumed only 10% of the implied tax amount from taxation of those transactions. Very controversial assumption.• Total sales tax collections for the year came in $30+M less than forecast, inclusive of the fiscal note.• Revenue Dept implemented a tax form that attempts to capture all the catch-all transactions individually.