EXPERT REBUTTAL REPORT IN THE MATTER OF Moussouris v....

CONTAINS CONFIDENTIAL MATERIALS

EXPERT REBUTTAL REPORT IN THE MATTER OF Moussouris v. Microsoft

SUBMITTED BY:

Henry S. Farber, Ph.D.

February 9, 2018

214

Case 2:15-cv-01483-JLR Document 359-10 Filed 02/21/18 Page 2 of 108

TableofContents

I. SUMMARYOFCONCLUSIONS..........................................................................................2II. DATA......................................................................................................................................4III. DR.SAAD’SPREDICTEDPAYANALYSESAREFLAWED......................................6A. OmittedFixedEffects................................................................................................................6B. OtherTechnicalErrors.............................................................................................................8C. Analysisofpayvariationbysupervisor.............................................................................9D. AnalysisofpayvariationforNamedPlaintiff................................................................11

IV. DR.SAAD’SPREDICTEDPROMOTIONSANALYSESAREFLAWED...............12A. Dr.Saad’sanalysesof“businessjustification”aremeaningless.............................21B. Dr.Saad’sz-score“selectionpool”analysis....................................................................26C. Dr.Saad’sinclusionofStandardTitleinStockLevelanalyses................................30

V. DR.SAAD’SANALYSESOFPAYISFLAWED.............................................................32A. ItisinappropriatetoincludeStockLevelasacontrolvariableinapayregression............................................................................................................................................32B. DiscussionofModels1-3.......................................................................................................37C. DisaggregationbetweenITandEngineering.................................................................37D. GenderdifferencesbyDiscipline.......................................................................................38E. CareerStage..............................................................................................................................39F. Dr.Saad’sothercriticismsofpayanalysis......................................................................401. Parttimeworkandleavesofabsence..........................................................................................402. Patents........................................................................................................................................................413. Collegehiresversuslateralhires....................................................................................................414. BusinessProducts.................................................................................................................................43

VI. DR.SAAD’SANALYSISOFCENSUSDATA.............................................................45VII. CONCLUSION.................................................................................................................45

215


-2 -


I. SUMMARY OF CONCLUSIONS

1. I have been asked to review the expert report and deposition transcript of the

Defendant’s expert, Dr. Ali Saad,1 and to determine whether these provide any reason to alter the

conclusions I described in my expert report.2 I find that the arguments made by Dr. Saad are

generally either incorrect, misleading, or irrelevant. I find no reason to alter the conclusions I

described in my expert report. My specific conclusions are summarized here:

• Female Technical Employees within the Engineering and IT Operations Professions employed at Microsoft (“Technical Employees”) are paid statistically significantly less than their male counterparts even after accepting Dr. Saad’s criticisms.

• Dr. Saad’s predicted pay analyses include several technical flaws. When predicting pay, he ignores each worker’s Standard Title and instead predicts pay based on the average pay across all Standard Titles. He makes two additional technical errors that cause him to (1) miscalculate each prediction’s standard error and (2) understate each person’s predicted pay in dollars.

• My analysis is designed to provide a statistical estimate of the amount by which

Technical Employees are underpaid, on average. It is not designed to identify which particular women are paid more or less than average, which is what Dr. Saad’s flawed analyses attempt to do.

• Dr. Saad reports the percent of supervisors who supervise women who are not

underpaid relative to their predicted pay (but-for gender discrimination). However, his analysis is flawed. I estimate a separate gender pay gap for each supervisor, and find that nearly all women work for supervisors under whom women earn less than men.

1 Expert Report of Ali Saad, Ph.D., in the matter of Moussouris et al., v. Microsoft Corporation, January 5, 2018. “Saad Report.” Dr. Saad also provided an errata to this report on February 5, 2018. 2 I previously submitted a report dated October 27, 2017 in this matter. On December 5, 2017, I submitted a corrected report. All references to “my report” or “Farber Report” refer to the latter.

216


-3 -


• Dr. Saad’s asserts that Plaintiff Moussouris was not paid less than the amount he predicts. This ignores Plaintiffs’ claim that Ms. Moussouris (and others) was denied a promotion or promotions on the basis of her gender. I find that Ms. Moussouris was likely underpaid relative to the amount that she would have earned if she had been promoted.

• Dr. Saad also presents an analysis of actual versus predicted advancement rates

for each supervisor, but this analysis is not informative as to the behavior of any supervisor. I present an analysis that identifies a separate advancement rate for each supervisor, and show that 90% of class-member years are worked for supervisors under whom women are less likely than men to advance.

• Dr. Saad reports that the Named Plaintiffs and Declarants did not receive fewer

advancements than they “should have” received. This is based on an assertion that if an advancement was predicted with greater than 50% probability, then it “should have” happened, and if not, then it “should not have.” This is simply an incorrect inference and an incorrect use of statistical techniques.

• Dr. Saad divides Class Members in a number of ways that are not appropriate.

Likewise, Dr. Saad arbitrarily divides promotions into small date ranges (including one analysis where he looks at 14 time periods per year). It is inappropriate to divide the data into small groups essentially in order to find groups where there are not gender pay differences.

• Dr. Saad presents a “selection pool” model of promotions. In order to perform

this analysis, Dr. Saad has broken the data into tens of thousands of “pools.” The majority of these pools are comprised of only men (or only women), which render them useless for determining anything about the difference in promotion rates by gender.

• Dr. Saad presents an analysis of so-called promotion “velocity.” This analysis is based on a small sample of the data, and fails to control for important factors that are likely to influence the probability of a promotion.

• Dr. Saad wrongly claims that Stock Level is not pay band. Stock Level is an

inappropriate control variable in a pay regression.

• Dr. Saad’s analysis of Census data is not informative.

217


-4 -


II. DATA

2. In my original report, I analyzed data provided by Microsoft that ran from January 1,

2010 through May 31, 2016. Microsoft had provided additional FY 2016 data on October 24,

2017, but these data were provided too late to be analyzed for my original report.3 Since filing my

original report, I have had an opportunity to review and analyze these data, and I will incorporate

them into this report.

3. Based on the instruction of counsel, I have also excluded observations in which the

employee was a “managers of managers” from the analyses throughout this report.4

observations ( among men and among women) refer to employee-years where the

worker was a manager of managers, and have been excluded.5

4. My original report indicated that there were 8,630 members of the compensation class,

who worked person-years, and 8,037 members of the advancement class, who worked a

total of person-years. After incorporating the new data for FY 2016 from Microsoft, and

excluding employee years where an individual was a manager of managers, these data identify

8,435 members of the compensation class, who worked total person-years, and 7,976

members of the advancement class, who worked total person-years. 90% of class-member

years are worked in Washington state, and 72% of class-members years are worked at Microsoft’s

Redmond, Washington headquarters.

3 See Farber Report at fn 4. 4 An observation is an employee-year. 5 I also exclude all employee-years where a worker is in Stock Levels 68 or higher. This exclusion was also in effect for my original report.

218


-5 -


5. Dr. Saad describes my inclusion of data prior to the beginning of the class period as

“inexplicable,” and performs his analyses using data restricted to the class period.6 I have

included data prior to the class period because I understand that the class period is based on a

statute of limitations, not based on a claim that gender discrimination at Microsoft only began in

2012. Thus, data prior to 2012 has economically relevant information regarding the claim of

gender discrimination at Microsoft. I further understand based on instruction from counsel that

data prior to 2012 are legally relevant.

6. I have repeated the analyses from my original report using three alternative data sets,

one that includes data from January 1, 2010 through May 31, 2016, a second that includes data

from January 1, 2010 through August 31, 2016 and a third that is limited to the period from

September 1, 2012 though August 31, 2016, which roughly corresponds to the class period.7

These analyses are contained in Appendix 1.8 Tables 1A-8A and Figures 1A-3A in this report are

parallel to Tables 1-8 and Figures 1-3 from my original report, and cover the same time period

(January 1, 2010 through May 31, 2016).9 Tables 1B–8B and Figures 1B–3B in this report are

6 Saad Report at fn 11. 7 The class period begins on September 16, 2012. Many of Dr. Saad’s analyses begin on September 1, 2012. 8 In reconciling my advancement indicator with Microsoft’s promotions indicator, I have made a small change in the way that I defined “advancements” based on changes in Stock Level. I have based my advancement indicator on the last observed stock level in a given compensation year. Previously, I had based my advancement indicator on the first observed stock level in a given compensation year. The Stock Level advancement analyses in the Appendix tables and throughout the report reflect the updated advancement indicator. 9 These tables differ from those contained in my original report based on the manager of managers exclusion. In this and all following analyses, I have also excluded six employee-years where Microsoft’s payroll data indicate that (though the worker worked in the United States), he or she was paid in a currency other than U.S. Dollars.

219


-6 -


parallel to Tables 1-8 and Figures 1-3 from my original report, but include data from January 1,

2010 through August 31, 2016, the last date for which Microsoft has provided payroll data.

Tables 1C – 8C and Figures 1C – 3C in this report are likewise parallel to Tables 1-8 and Figures

1-3 from my original report, but are restricted to September 1, 2012 through August 31 2016.

Each of these analyses excludes managers of managers.

7. There are only very minor differences between each of these sets of tables, which

indicates that my results are not sensitive to differences in which years of data I use. Throughout

the remainder of this report, unless otherwise specified, I will use the broadest time period of data

available to me: January 1, 2010 to August 31, 2016.

III. DR. SAAD’S PREDICTED PAY ANALYSES ARE FLAWED

8. Dr. Saad claims that my analysis (Model 5 of Table 3) shows that 33.7% of women

earned more than my model predicted that they would have earned if there were no gender pay

gap in 2015. Further, he asserts that for 5.9% of female technical employees at Microsoft in 2015,

this difference is statistically significant.10 However, not only does Dr. Saad misunderstand my

assignment and the purpose of my analysis, but his calculations are also performed incorrectly.

A. Omitted Fixed Effects

9. Dr. Saad describes his exercise in predicting pay for individuals as being based on

Model 5 of Table 3 of my report, with the exception of a control for gender.11 He describes this

10 Saad Report at ¶ 28. 11 He did not include gender as a control variable so that “predicted pay does not systematically separate men and women but instead models the impact their other non-gender characteristics have on pay.” (Saad Report at ¶ 25)

220


-7 -


analysis as controlling for year, age (and its square), tenure at Microsoft (and tenure squared), the

state and city in which the employee works, their pay scale type, performance rating, discipline,

and Standard Title.12 However, when Dr. Saad calculates but-for compensation for the purposes

of this comparison with actual pay, he mistakenly omits the standard title fixed effects.13 In effect,

Dr. Saad’s calculation essentially assumes that, in the but-for world, workers in each standard title

receive the same average compensation, assuming all other variables are fixed. This introduces a

great deal of noise into Dr. Saad’s calculations. Consider the case of Plaintiff’s declarant Ms.

Sowinska, who earned in compensation year 2015. Dr. Saad calculates her predicted

compensation but-for gender discrimination (purportedly based on my Model 5) as ,14

whereas he would have calculated if he had used my whole model including the standard

title fixed effects (as well as correcting for the “smearing factor,” discussed below). In other

words, if Dr. Saad had actually implemented my model, including Standard Title fixed effects, he

would have found that Ms. Sowinska’s actual pay was below her predicted pay but-for gender

discrimination. This is because by ignoring Standard Title fixed effects, Dr. Saad has predicted

Ms. Sowinska’s pay (and indeed, all women’s pay) assuming that she earns the average of what all

Technical Employees in all positions earn (after controlling for age and other human capital

factors).

12 Saad report at ¶ 23 13 In Stata, the computer program that Dr. Saad used to perform this analysis, the built-in command that he used to calculate his but-for compensation excludes fixed effects by default. See Stata 15 documentation for –areg postestimation-. 14 If Dr. Saad were to calculate Ms. Sowinska’s predicted but-for compensation using my updated sample, which excludes employee-years where workers are managers of managers, Dr. Saad’s prediction method would yield a lower predicted but-for compensation of .

221


-8 -


B. Other Technical Errors

10. In addition to excluding fixed effects, Dr. Saad made two additional technical errors in

his calculation. Dr. Saad’s first error is that he incorrectly calculates the standardized residual that

he uses to test for statistical significance. Just as he improperly ignores standard-title fixed effects

in his prediction of but-for compensation, he improperly includes standard-title fixed effects in his

computation of the standardized residuals. That is, he treats standard-title fixed effects as if they

were a part of the error term, which is simply wrong; these fixed effects are part of the explained

variation, not part of the unexplained residual.15

11. Additionally, in predicting pay, Dr. Saad has made a mathematical error that causes

him to systematically underestimate the but-for pay implied by my model. This error is that he

failed to account for the “smearing factor,” which is necessary when using estimates of a natural

log model of pay to predict the dollar amount of pay.16

12. These technical errors notwithstanding, Dr. Saad also misinterprets my assignment. As

described in my original report, my assignment was to study whether there is “statistical evidence

of (1) discrimination in compensation between male and female Technical Employees in Stock

Levels 59-67, and (2) discrimination in Stock Level or Career Stage advancement rates between 15 The built-in command that Dr. Saad uses to predict standardized residuals is proper when used after a regression performed by the command –reg- but is not proper when used after the command –areg- command that Dr. Saad employed. 16 My regression model directly predicts the natural logarithm (“log”) of total compensation rather than compensation in dollar terms, as is the standard practice in labor economics. Transforming the natural log into dollar terms will systematically underestimate predicted total compensation in dollars unless multiplied by what is referred to as a “smearing factor.” Dr. Saad does not apply any smearing factor. I have used the basic Duan smearing estimate in my investigation of Dr. Saad’s analysis. See Duan, Naihua, “Smearing Estimate: A Nonparametric Retransformation Model.” Journal of the American Statistical Association., Volume 78, No. 383 (September 1983), pp. 605-610.

222


-9 -


male and female Technical Employees in Stock Levels 59-64.”17 My analysis is designed to

provide a statistical estimate of the amount by which female Technical Employees at Microsoft are

underpaid, on average. It is not designed to identify which particular women are paid more or less

than average (after controlling for various factors), which is what Dr. Saad’s analyses attempt to

do. Any statistical estimate is going to contain some amount of unexplained variation (captured in

the error term of the regression). This term may be negative (indicating that the person is paid less

than the model would predict given her characteristics) or positive (indicating the opposite). In

performing this predicted pay analysis, Dr. Saad has ignored this feature of all regression models

and has assumed each individual’s unexplained variation in pay does not apply in the but-for

world. This is incorrect.

C. Analysis of pay variation by supervisor

13. Dr. Saad argues that I did not take into account variation among supervisors.18 As an

attempt to measure this variation, Dr. Saad presents a series of pie charts that purport to show the

proportion of supervisors (at various levels in the supervisory hierarchy) who supervise (1) women

who all have a non-significant difference in actual and but-for predicted pay; (2) an equal number

of women who earn statistically significantly more than predicted and statistically significantly

less than predicted; (3) more women who earn statistically significantly more than the but-for

prediction; (4) or more women who earn statistically significantly less than the but-for prediction.

17 Farber Report at ¶ 4. 18 Saad Report at ¶ 30.

223


-10 -


This is done for Level 2, 3, and 4 supervisors for 2015.19 Dr. Saad does not present these figures

for Level 1 supervisors, which is the top of the supervisory hierarchy provided by Microsoft.

14. These charts are created using Dr. Saad’s flawed predictions of but-for pay. Namely,

he did not control for standard title fixed effects, he did not correctly compute the standardized

residual, and he ignored the “smearing factor” when transforming log pay into levels of pay.

These flaws render this analysis unreliable.

15. Furthermore, Dr. Saad’s analyses focus on the count of supervisors, not the women

who they supervise. That is, the size of each “slice” of these pie charts is not scaled by the number

of class members (or even by the number of all employees), but rather is scaled by a count of

supervisors. Dr. Saad notes that “Level 2 supervisors can supervise anywhere from 1 to almost

3,000 technical employees in the data that I have….Level 3 supervisors can supervise anywhere

from 1 to 1,065 employees in the data…[and] Level 4 supervisors…supervise from 1 to over 600

employees in the data.”20 Because he has not adjusted for the number of female employees who

work under each supervisor, the pie charts on pages 26 and 27 of Dr. Saad’s report give equal

weight to supervisors who supervise 10 employees as it does to supervisors who supervise

thousands. As a result, these pie charts are misleading about the number of affected women.

16. Technical errors notwithstanding, Dr. Saad’s analysis is not informative to answer the

question of whether certain supervisors tend to underpay women. In order to answer this question,

it is possible to identify a separate gender pay gap for each supervisor. I have performed the

analyses described as Model 4 and Model 5 in my original report, with one change: instead of 19 Saad Report at ¶ 35 and Figures on pages 27-28 20 Saad Report at ¶ 33.

224


-11 -


calculating a single gender effect, I calculate a separate gender effect for each Level 1 supervisor.

I present the results of this analysis in Table 1. In any year during the class period,21 more than

98% of class members report to one of 4 Level 1 supervisors.22 The analyses reported in Table 1

demonstrate that virtually all woman years (more than 99%) are worked under Level 1 supervisors

under whom women earn less than men, on average, after controlling for the factors in the model.

This average pay difference is statistically significant for Level 1 supervisors representing

approximately 98% (Model 4) or 99% (Model 5) of woman years.

D. Analysis of pay variation for Named Plaintiff

17. Dr. Saad also discusses Plaintiff Moussouris at length and concludes that she is paid

more than her predicted compensation and is therefore uninjured.23 In addition to the technical

issues discussed above, Dr. Saad’s analysis ignores Plaintiffs’ claim that Ms. Moussouris was

denied a promotion or promotions based on her gender In 2012 and 2013, Ms. Moussouris had the

title of “Senior Program Manager.” In 2014, Ms. Moussouris had the title of “Senior PM

Manager.” In both of these positions, she was in Stock Level 64. The most common promotion

from a Level 64 “Senior Program Manager” is to a Level 65 “Principal Program Manager,” and

the most common promotion from a Level 64 “Senior PM Manager” is to a Level 65 “Principal

21 During the class period, the supervisor hierarchy is available for compensation years 2014-2016, but not for 2013. 22 All women in my analysis work for one of ten Level 1 supervisors. 23 Saad Report at ¶¶ 40-42. I have limited my attention to Ms. Moussouris in this paragraph based on Dr. Saad’s report.

225


-12 -


PM Manager.”24 Table 2 presents Ms. Moussouris’ actual pay and her predicted but-for pay

(based on the corrections to Saad’s prediction method of my model, as described in paragraphs 9-

12 above), had she had the most-common promotion from her actual job title. In each year, Ms.

Moussouris received lower compensation than my model predicts that she would have, had she

received the most-common promotion from her position. Insofar as all class members are subject

to discrimination in promotion, similar considerations apply to their pay predictions as well. 25

IV. DR. SAAD’S PREDICTED PROMOTIONS ANALYSES ARE FLAWED

18. Dr. Saad presents a comparison of actual versus predicted promotion rates by

supervisor.26 Dr. Saad treats the predicted promotion rate for a supervisor as a rate that is not

subject to any variation, which is not true. Predictions (by their nature) include a degree of

uncertainty in their estimation, which Dr. Saad ignores. Furthermore, as with Dr. Saad’s analysis

of gender pay discrepancy by supervisor, Dr. Saad has misunderstood the purpose of my analysis,

which is to measure the difference in promotion rates between men and women and see if there is

statistical evidence of gender disparity in advancement rates.

19. As with his analysis of pay by supervisor, Dr. Saad’s analysis of advancement by

supervisor is not informative to answer the question of whether certain supervisors tend to

24 For the purposes of finding the most common promotion, I searched for the most common standard title associated with an increase in Stock Level from Ms. Moussouris’ actual standard title and Stock Level. 25 I have performed this analysis for only Ms. Moussouris (and not any other class members), based on Saad’s specific criticisms regarding Ms. Moussouris. More broadly, Plaintiffs claim that female Technical Employees at Microsoft are discriminated against with respect to promotions, and I do find that women are less likely to advance (whether measured as an increase in Stock Level or Career Stage) than men. 26 Saad Report at pp. 33-36.

226


-13 -


disadvantage women with respect to advancements. I can identify a separate gender advancement

gap for each supervisor. I have performed the advancement analyses described in my original

report adding a separate gender effect for each supervisor. I do this separately for each Level 1

supervisor. I present the results of this analysis in Table 3. Among female Technical Employees,

approximately 90% of woman-years are worked for Level 1 supervisors who under-advance

women, and the difference in advancement rate is negative and statistically significant for

supervisors who supervise 89% of woman-years.

20. Dr. Saad compares “predicted” and actual advancements among Named Plaintiffs and

declarants but his analysis is deeply flawed. As with the pay analysis above, Dr. Saad has

assumed that the point estimates generated for each Named Plaintiff or Declarant are exactly true,

and he ignores the variation in these predictions.

21. Troublingly, Dr. Saad’s analysis assumes that if the predicted probability of

advancement for a particular worker in a particular year is greater than 50%, then that

advancement “should have occurred,” and implicitly assumes that any person-year where the

predicted probability of advancement is less than 50%, then that is an advancement that should not

have occurred.27 This method of identifying advancements that “should have occurred” is based

on a fundamental misunderstanding of probabilities.

22. Consider the following hypothetical example. Imagine that there are 4 workers, all of

whom are identical and equally qualified for an advancement, or promotion. Exactly one of the

four will be promoted (and the other three will not) so that each of these workers has a 25%

27 Saad Report at ¶ 49.

227


-14 -


probability of promotion. Among these four workers, I would predict that one promotion should

occur. Dr. Saad’s method of identifying promotions that “should have occurred” will identify zero

promotions that should have occurred, even though one out of the four workers will be promoted

(as I predicted). As another example, consider a case where three individuals (perhaps out of a

larger pool) each have a 2/3 chance of promotion, and out of this group of three workers, two are

in fact promoted. Note that the predicted number of promotions for this group is two (because

3*2/3 = 2). Dr. Saad’s method, however, would predict that each of these workers should have

been promoted, and would calculate a promotion shortfall of one, even though the predicted

number of promotions and actual number of promotions are exactly the same.

23. In Row 1 of Table 4, I present the number of advancements that actually occurred

(separately by gender, and overall). In Row 2 of Table 4, I present the number of advancements

that Dr. Saad believes “should have occurred” based on his assertion that any advancement that

was predicted to occur with greater than 50% probability “should have occurred” (and by

extension, any advancement that was predicted to occur with less than 50% probability should not

have happened). Dr. Saad’s count of advancements that “should have occurred” is 23,379,

whereas 40,812 advancements actually did occur. Dr. Saad’s “should have occurred”

advancement prediction rule predicts less than 60% of the actual number of advancements.

24. The appropriate approach to calculate the advancement shortfall, as I present in my

original report, is to sum up the predicted probability of advancement across the entire class, and

to compare that to the number of female advancements that actually occurred.28

28 Farber Report at ¶¶ 72-77.

228


-15 -


25. Dr. Saad criticizes the use of the advancement indicator that I constructed based on

changes in Stock Levels. This advancement indicator compared an employee’s Stock Level in a

compensation year t+1 to the stock level in the compensation year t. I repeat my analysis using

the Microsoft-provided promotions indicator.29 The results of my analysis are presented in Table

5. In footnote 76 of his report Dr. Saad admits that the “proportion of promotions that go to

women is virtually identical under the two definitions.” It is not surprising therefore that I find

largely similar results using either measure of Stock Level advancement—women are about 2.1

percentage points less likely than men to advance using the Stock Level advancement indicator I

previously defined, and they are about 2.4 percentage points less likely than men to advance using

the Microsoft-provided promotions indicator. In addition, in my original report I also considered

an alternative definition of advancement: an increase in Career Stage. Whether I use advancement

indicators (based on either advancement in stock level or advancement in career stage) or the

Microsoft-provided promotion variable, my conclusion remains unchanged – women advance at

lower rates than men based on each of these measures of advancement.

26. Dr. Saad argues that I should have examined IT Operations and Engineering

promotions separately. However, doing so ignores the fact that it is possible for workers to move

back and forth between these two Professions (as Plaintiff Muenchow did in 2016, when she

moved from IT Program Manager in the IT Operations Profession to Senior Program Manager in

the Engineering Profession). In all, I observe approximately 800 advancements that also include

workers moving between Professions, mostly from IT Operations to Engineering, and

29 This variable is provided by Microsoft and indicates an increase in Stock Level, which Microsoft terms a “promotion.”

229


-16 -


approximately 25% of all advancements among workers in the IT Operations Profession include a

move to the Engineering Profession. Further, approximately 61% of individuals who ever worked

in the IT Operations Profession (during the period January 1, 2010 to August 31, 2016) also

worked in the Engineering Profession at some point during the same period.30

27. Lastly, note that my advancement analysis controls for Discipline, which is a finer

categorization than Profession, so that controlling for Discipline also serves to control for

Profession. Because examining IT Operations and Engineering separately ignores the possibility

(and reality) of advancements and other movement between these two Professions, and because I

do control for each of these Professions, I reject Dr. Saad’s assertion that it is inappropriate to

consider them jointly.

28. Dr. Saad additionally argues that I should have included time in stock level and

standard title, not time at Microsoft as the tenure variable in this analysis. However, time in Stock

Level is a variable that is based on prior advancement decisions. Based on the plaintiffs’ claim

(and my findings) that women are less likely to advance than men, and are under-leveled relative

to men, women will tend to have longer time in Stock Level than men, all else equal. This is a

natural consequence of women having lower Stock Level advancement rates than men.

Therefore, inclusion of time in Stock Level (or time in Standard Title) will introduce bias into my

30 Saad claims that only 4.3% of employees transfer between professions. His analysis misses some transfers because he ignores pre-class period data, and misses some transfers because he only looks at the annualized data, which obscures some transfers, whereas I make my calculation based on the underlying data. I find that 8.5% of employees (4,883 individuals) work in both IT Operations and Engineering at some point in the discovery period. Put another way, 61% of people who ever worked in IT Operations also worked in Engineering at some point during the discovery period.

230


-17 -


analysis.31 Despite these considerations, I re-estimated my advancement models, presented in

Table 5, including time at relevant level and its square (in addition to other tenure variables

previously included in my analysis). These results are presented in Table 6. Comparing the

results in Row A of Tables 5 and 6 show that the marginal effect (i.e.. the gender difference in the

probability of advancement in stock level) falls from about 2.1 percentage points to about 1.7

percentage points when controlling for time in Stock Level and remains statistically significant.

Comparing the results of Row B of Tables 5 and 6 show that after controlling for time in Stock

Level, the marginal effect increases slightly, from 2.4 percentage points to 2.7 percentage points. I

make a parallel change to my analysis of the promotions based on change in Career Stage. I

include time in Career Stage and time in Career Stage squared in addition to other tenure variables

previously included in my analysis. These results are presented in Rows C and D in Table 6. After

controlling for time in Career Stage, I continue to find that women are less likely to be promoted

than men, and this difference is statistically significant.

29. Dr. Saad also argues that I should have examined in- and out-of-cycle advancements

separately. His argument is twofold: first, advancements at different times of year may be subject

to different annual budgets; and second that I have ignored performance review timing.

30. There is no principled reason to divide advancements based on whether they are given

in September or during a different time of year. As a labor economist, the standard method of

analyzing advancements is to examine advancements that occur over a fixed period of time,

31 I have not performed an analysis that includes time in Standard Title as a control, as I believe that it is inappropriate to control for Standard Title when doing analyses gender differences in Stock Level. In particular, because each Standard Title typically spans only a small number of Stock Levels, including Standard Title will over-explain the variance in the model.

231


-18 -


commonly over a particular year. Dr. Saad is merely subdividing the available data into smaller

portions, and it is unsurprising that by subdividing the available data in this manner, he is able to

construct groups with no advancements gap.

31. Notably, even after artificially dividing advancements based on time of year, Dr. Saad

still finds a shortfall for women among out-of-cycle advancements (i.e. those that happen at any

time of year other than September 1). Dr. Saad asserts that these non-September advancements

are not connected with the annual performance review process, and suggests that perhaps there are

other, unspecified data and factors that I should have controlled for (beyond the performance

review information, tenure at Microsoft, age, Stock Level prior to advancement, Discipline,

location, and compensation year).32 Note also that in paragraph 102 of his report, Dr. Saad argues

that I have paired off-cycle (non-September 1) advancements with the wrong annual review, but in

paragraph 106, he claims that off-cycle advancements have no connection to the annual review

process.

32. Dr. Saad argues that by combining advancements that occur at different times of the

year, I have mismatched performance ratings and advancement decisions for mid-year and other

advancements.33 He goes on to argue that “at the time mid-year promotions decisions are made

(mid-fiscal year), the end-of-fiscal-year performance review cycle has not yet been completed.

Therefore, the only official information that would be available to decision-makers regarding

performance ratings would be from the prior year.”34 While it may be the case, as Dr. Saad

32 Saad Report at ¶ 106. 33 Saad Report at ¶ 102. 34 Saad Report at ¶ 102, bolded emphasis added, italic emphasis in original.

232


-19 -


writes, that this is the only official information available to decision-makers regarding off-cycle

advancements (depending on the timing of said advancement), the employee’s managers may have

additional information about that employee’s performance that will eventually appear in the

performance review, even if it has not yet as of the day of the advancement. For example, if an

individual is being considered for advancement in May of 2015, the performance review for that

year is incomplete, but has likely begun.35 More importantly, however, the employee’s manager

has an idea of his or her performance and skills, both as an absolute matter and relative to other

comparable individuals. Furthermore, Dr. Saad’s analysis likely mis-matches some performance

reviews: for advancements that happen between the end of the performance review cycle (which

closes to managers at the end of July) and September 1, managers would have “official”

performance review data, but Dr. Saad ignores these data and uses only the previous year’s

performance reviews.36 Nevertheless, based on Dr. Saad’s suggestion that the prior year’s

performance reviews may be relevant for some employees who are advanced outside of September

of a particular year, I have added controls for each employee’s performance review outcomes for

the prior year. This allows me to control for performance reviews in the recent past as well as the

current year, both of which may influence the probability of advancement. The results of this

analysis, which include both current and one-year lagged performance reviews, are presented in

35 Johnson Declaration at ¶ 6. Ritchie Ex. 21 also explains that in the performance system in use beginning in 2014, “Connects,” which are periodic review meetings between an employee and his or her manager, are not pegged to a rigid time schedule like mid-year check-ins and annual reviews. 36 Johnson Declaration at ¶ 6. Ritchie Tr. 458:9-10 (“Our annual rewards process takes place on September 1”); Johnson Decl. ¶ 5 (“The annual rewards cycle begins on May 1st of each year.”).

233


-20 -


Table 7. After adding additional controls for prior performance reviews, I continue to find the

gender difference in the probability of advancement of about 2.1 percentage points, a difference

which is statistically significant.

33. I also performed a version of my shortfall analysis based on the model that includes

current and lagged performance reviews. I present the results of this analysis in Table 8. I find that

the advancement shortfall for women in stock level 59 through 64 between January 1, 2010

through August 31, 2016 is equal to 534. These results are similar to those reported in Table 7B in

Appendix 1 (which did not control for lagged performance reviews but is otherwise identical to

this analysis).

34. In short, Dr. Saad’s attempts to disaggregate advancements based on the time of year in

which they occur distract from the fact that women at Microsoft advance at lower rates than men.

35. Dr. Saad concludes that off-cycle (non-September 1) promotion justifications are

“equally thorough” relative to in-cycle promotion justifications based on an exercise where he

compared the number of words in in- and out-of-cycle promotion justification comments.37 There

is no foundation in economics of which I am aware regarding any relationship between word

counts and the thoroughness of promotion review processes. In addition to counting the number

of words in promotion comments, Dr. Saad has also attempted to identify promotions as either

based on “business need justifications” or not. I discuss this below.


234


-21 -


A. Dr. Saad’s analyses of “business justification” are meaningless

36. Dr. Saad concluded that out-of-cycle promotions (where he finds gender bias) were

more likely to be based on “idiosyncratic ‘business need’ justifications” than during on-cycle

reviews. I do not see how this observation is relevant to an analysis of gender discrimination in

advancement. In any case, this analysis upon which this conclusion is based is so flawed as to be

completely useless. On a conceptual level, this analysis is flawed because he has identified a

small number of concepts as being “initiated for business reasons (e.g. expected product growth or

retention concerns),” he identifies others as being based on the “individual characteristics of the

promotee (e.g. personal skills and accomplishments).”38 This is an arbitrary division. The non-

“business reason” justifications that Saad identified are business justifications—promotions based

on an employee’s personal skills and accomplishments are promotions based on an employee’s

ability to fit into a particular role, or based on a need to avoid retention problems from not

promoting high-achieving workers.

37. In addition to this arbitrary and meaningless division of promotions into “business

need” versus “non-business need,” Dr. Saad does an extremely poor job of identifying “business

needs” justifications based on the criteria he attempted to establish.

38. Dr. Saad created a list of 229 particular words or phrases (known as “strings”), and

then used a computer program to search through tens of thousands of promotion comments for

exact matches for these strings, including spelling and punctuation. Some of these strings are

relatively simple (e.g. “business need”), and may appear in that exact form in many records.


235


-22 -


Others are somewhat more complicated: Dr. Saad searches for the term “need:” which, though it

consists of a single word, will only identify a record as a business need if it includes the colon

after the word “need”. Dr. Saad does not explain why he thinks that the word “need” standing

alone does not indicate a business need justification, but the same word followed by a colon does.

Many strings are extremely complicated: Dr. Saad searches for the term “dsp team has the need

for senior software engineers”. In order for this term to identify a promotion comment as being

based on “business needs,” the promotion comment would need to include this exact string. This

issue is discussed at greater length in paragraph 40, below.

39. Below find a list of examples of problems with his analysis, though this is by no means

exhaustive.

• Dr. Saad identified the phrases “there is need” and “there is a need” as indicative

of business justification. However, the very similar phrase “there’s a need” is not

identified by Dr. Saad as being indicative of a business justification. The phrase

“there’s a need” appears multiple times in promotion justification comments that

Dr. Saad has identified as being not based on business needs.39 Dr. Saad does not

39 Dr. Saad identifies the promotion of Personnel Number 152199 as based on a business need based on a comment that includes

In contrast, Dr. Saad identified the promotion of Personnel Number 367072 as not based on a business need, based on a promotion justification comment that includes

236


-23 -


explain why the phrase “there is a need” indicates a business need justification, but

the phrase “there’s a need” does not.

• Though one of the enumerated “business needs justifications” Dr. Saad cites is

retention of employees who receive outside job offers, Dr. Saad only searched for

two strings that contained the names of two likely competitor firms:

Dr. Saad suggested that searching for the phrases

“competitive offer” or “received offer” would be likely to identify competing

offers from rival firms.40 However, Dr. Saad did not search for either of these

terms.41

• Dr. Saad testified that it would be “quite a surprise if the phrase “competitive

offer” and the word “retain” were not always together.”42 The word “retain”

appears without the phrase “competitive offer” in 198 promotion comments. Of

these, almost exactly half (101) are identified as promotions that are not based on

business needs, while the rest (97) are identified as promotions that are based on

business needs.

40. Dr. Saad testified that he based his list of 229 strings on a randomly-selected sample of

1000 promotion justification comments. He testified that he then selected particular phrases that

40 Deposition of Ali Saad, Ph.D., January 30, 2018 (“Saad Tr.”) at 39:16-19 41 At his deposition, after being asked whether he searched for either “competitive offer” and “received offer” Dr. Saad argued that because he had searched for the term “offer” these phrases would be captured anyway. Dr. Saad did not search for the term “offer.” See Saad Tr. at 47:17-22. 42 Saad Tr. at 46:25-47:2. He based this statement on “an inference based on having looked at hundreds of these things. Saad Tr. at 47:4-7.

237


-24 -


appeared in these comments, and used those phrases to have a computer program identify other

“business needs” elsewhere. Dr. Saad apparently did very little analysis or checking to ensure that

these strings were well-suited to identifying “business needs” outside of the small sample on

which he trained his list. Many of the strings for which Dr. Saad searches are extremely specific.

For example, Dr. Saad searches for the strings “the need to have a strong senior ic band grows

with it”; “key to keeping Bing Ads Private Lab infastructure running ";43 and "the level of

complexity in that space requires a senior level pm”. Complex strings such as these are so

narrowly tailored to the “training” random sample that they are capable of identifying no more

than one promotion as being based on “business needs.” However, Dr. Saad attempts to extract

information from the larger promotions data set based on these very narrowly tailored strings, and

fails to do so. In fact, of the 229 strings selected by Dr. Saad, 111—nearly half—identify exactly

one promotion comment as a business justification. Presumably in these cases, the identified

comment is a record from Dr. Saad’s small random sample.

41. Furthermore, Dr. Saad did not produce the random sample on which he purported to

base his string search list until partway through his deposition on January 30, 2018. 44 Based on a

review of this sample, I do not see how Dr. Saad could have derived the list of strings that appear

in his BN_Dictionary.R file (the file that contains the strings he ultimately searches for in the

promotion justifications) from this random sample. There are many strings in the

BN_Dictionary.R file that are not found in the random sample Dr. Saad provided on January 30, 43 The misspelling of “infrastructure” appears in Dr. Saad’s code, but apparently does not appear in any promotion justification comment, as this string identifies zero promotions as based on “business needs.” 44 This file is called “MRT Sample 1000.csv”

238


-25 -


2018. 45 In addition to the broader concerns about this analysis outlined above, I have no

understanding of the source of many of the strings from BN_Dictionary.R, and I cannot evaluate

the accuracy of the analysis Dr. Saad performed.

42. At his deposition, Dr. Saad claimed that the purpose of this analysis was to identify a

“sufficiently large” number of business reasons and compare them between September and non-

September promotions. He testified that it is “not an issue for me and did not concern me” whether

he had identified too many or too few promotions as being based on business justifications.46 In

spite of the fact that Dr. Saad made no attempt to measure or understand the error rates in this

categorization, he “could say probably with certainty that the relationship between the annual

review and other times of the year with respect to the incidents of business justification would not

be changed by any other sort of search you would do.”47 It is not standard statistical or

econometric practice to ignore serious errors in an analysis and assert “probably with certainty”

that your findings would be unchanged if these errors were corrected.

43. Dr. Saad also attempts to draw conclusions about the relative prevalence of promotions

based on “business justifications” at various Career Stage levels. However, as with the in- and out-

of-cycle promotions analysis, in addition to being completely arbitrary, his attempted

identification of “business justifications” is so poorly constructed as to be meaningless.

45 Examples of strings that appear in his BN_Dictionary file but not in the random sample upon which he purported based this list upon include, but are not limited to: “the business is demanding ever more of this skill”; “dsp team has the need for senior software engineers”; “important we make these partner teams successful”; and “this promotion justification will allow team to continue”. See Appendix 2 for a complete list. 46 Saad Tr. at 51:23-25. 47 Saad Tr. at 52:4-9.

239


-26 -


B. Dr. Saad’s z-score “selection pool” analysis

44. Rather than the probit analysis I used in my report, Dr. Saad analyzes promotions using

a “selection pool model” he describes as a “Z-model.” He describes selection pool models as

“probably more commonly used in statistical analysis of selection practices in employment

litigation than any other statistical technique.”48 Regardless of the truth of this statement,49 the

selection pool model that Dr. Saad has described is inappropriate for use in this context.

45. In support of his claim that Z-models are commonly used, he cites to a handbook

chapter that he co-authored.50 The general idea of the Z-score selection model, as described by

Dr. Saad, is to construct “presumed homogenous pools” (where the pools are homogenous except

with respect to gender) and to compare the proportion of female promotions within the pool to the

overall proportion of females in the pool.51 In order to construct groups that are homogenous to

the level of detail he asserts is necessary, Dr. Saad divides the data into tens of thousands of

pools.52 In particular, in order to be considered in the same pool, Dr. Saad requires that workers

have identical: Profession and Discipline, High/Low Performance (a variable he constructs based

on performance review scores), review cycle, review month (a variable he constructs based on

48 Saad Report at ¶ 116. 49 I do not have an opinion on whether or not selection pool models are more commonly used in employment litigation than any other statistical technique. 50 Saad Report at ¶ 116, fn.87 (citing Haan, C., E. Reardon, and A. Saad, “Employment Discrimination Litigation,” chapter in Litigation Services Handbook, ed., by Roman Weil, et al., 2012 (“Litigation Services Handbook”)). 51 In his handbook chapter, Dr. Saad used the term “pool.” In his report he uses the terms “strata” and “pool” interchangeably. In his deposition, he uses the term “strata.” 52 This is the number of strata (within the Engineering Profession) that contain a positive number of promotions.

240


-27 -


promotion dates, and supervisors. The “review cycle” is delineated into “On-Cycle,” which is all

promotions that occur on September 1; “Off-Cycle March,” which is all promotions that occur on

March 1; and “Off-Cycle Other,” which is all promotions other than those recorded on September

1 or March 1. The “review month” is constructed based on the month of the promotion. For

example, a promotion that occurred on September 2, 2014 would have a review month of

September 2014. Not only does Dr. Saad consider promotions that occur in September separately

from promotions that occur in October, but he considers promotions that happen on September 1

separately from promotions that occur on September 2, and promotions that occur on March 1

separately from promotions that occur on March 2. All told, each year Dr. Saad employs fourteen

separate criteria based on the “month” that promotions occur: September 1; all other dates in

September; March 1; all other dates in March; and each other calendar month (separately). Dr.

Saad has provided dubious support for the assertion that promotions that occur in September

should be considered separately from promotions that occur at other times of the year (i.e., his

poorly-formed business justification analysis, described at length above), and he has provided no

support at all for the assertion that promotions that occur in October of a given year should be

considered separately from promotions that occur in November of the same year.

46. Dr. Saad’s requirement that members of each pool have the same supervisor has a

peculiar effect. Because each worker may have several direct and indirect supervisors (each

worker has an average of 4.3 supervisors, though some workers have as many as 7),53 each worker

enters into several “pools.” As a result, there are many more “pools” than there are promotions,

53 This is based on a review of backup materials provided by Dr. Saad.

241


-28 -


because the same employee’s promotion is analyzed in multiple pools. Furthermore, Dr. Saad

assigns weights to each pool, with greater weight given to the promotion decision of direct line

managers and lower weight to higher-level managers. This weighting scheme is arbitrary and is

unjustified by Dr. Saad’s presumed notion that direct managers’ decisions are more important than

the decisions made by the higher-level managers in Microsoft’s organizational hierarchy.54

47. All told, Dr. Saad divides the data into 58,851 selection pools for engineering, and

4,983 selection pools within IT Operations.55 The average size of these pools is 6.6 people in

Engineering, and 4.6 people in IT Operations.56 The majority of these pools (nearly 60% within

each Profession) have zero female employees, and an additional 8% (Engineering) to 12% (IT

Operations have only female employees. Within the Engineering Profession, 39,482 (or 67%) of

Dr. Saad’s selection pools are comprised of a single gender. Within IT Operations, 3,509 (or

70%) of selection pools are comprised of a single gender. Further, 39% (Engineering) to 43% (IT

Operations) of Dr. Saad’s selection pools contain a single individual. As Dr. Saad acknowledged

in his deposition, pools with no gender variation will provide no useful information.57 As a result,

Dr. Saad’s analysis of advancements arbitrarily omits a substantial fraction of relevant individuals

(and advancements/non-advancements).

54 See Johnson Declaration at ¶ 6 for an explanation of the approval hierarchy for rewards. See Ritchie Tr. 495:10-497:16 for an explanation of the approval hierarchy for promotions. 55 As described in ¶ 26 above, given the high degree of advancement and movement between IT Operations and Engineering, I do not think it is appropriate to segregate these two Professions for the purposes of an advancement analysis. 56 These numbers represent individual people and do not account for the supervisor weights described above. 57 Saad Tr. at 204:16-205:3.

242


-29 -


48. Dr. Saad has constructed selection pools such that two thirds of these pools provide no

useful information. Though Dr. Saad created tens of thousands of selection pools, he testified that

he combined performance ratings into groups “to reduce the number of strata; otherwise, you

would get far more strata than we already had.”58 Given the tens of thousands of strata he

created—the majority of which exhibit zero gender diversity and thus contain no useful

information—Dr. Saad does not seem to be concerned with this issue elsewhere.

49. In his handbook chapter, Dr. Saad acknowledges that for the selection pool model to

produce reliable results, there must be a certain sample size and number of expected selections.59

The data provided by Microsoft provided more than employee-year observations, which

contained information on fewer than 27,000 promotions (using MS promotions indicator) during

the class period. However, since each promotion is present in multiple pools, Dr. Saad has

constructed nearly 64,000 selection pools (between IT Operations and Engineering), each of which

is defined by having a nonzero number of promotions. Based on the method by which he

weighted many layers of supervisors, Dr. Saad has created thousands more “selection pools” than

there are actual promotions in the data.

50. Lastly, Dr. Saad’s own handbook chapter asserts that situations where many factors

(such as experience, seniority, and education) influence the probability of promotion often require

58 Saad Tr. at 219:9-17. 59 Litigation Services Handbook at 32.5(f).

243


-30 -


multiple regression analysis (of which the probit analysis I performed in my original report is an

appropriate analog for discrete choice models).60

51. In short, Dr. Saad’s z-score “selection pool” model is designed to break the sample into

pools that are so small—and which usually have no gender variation—that no meaningful

information can be gleaned from this analysis. It should be ignored in its entirety.

C. Dr. Saad’s inclusion of Standard Title in Stock Level analyses

52. Dr. Saad argues that it was inappropriate for me to not include Standard Title in my

advancement and Stock Level analyses. I disagree. Most Standard Titles (85%) span only two or

three Stock Levels.61 For this reason, controlling for Standard Title will leave only a small

amount of variation between Stock Levels. Therefore, controlling for Standard Title will

“overcontrol” an analysis of Stock Level variation. I believe that including Standard Title in an

analysis of Stock Level placement (as does Dr. Saad) will understate the discrepancy in Stock

Level placement between men and women.

53. At page 83 of his report, Dr. Saad re-estimates the stock level distribution graph I

presented as Figure 3 in my original report, but adds controls for Standard Title. Even after

controlling for each worker’s Standard Title, Dr. Saad finds women are over-represented in low

60 “Suppose, however, that many factors influence the probability of a promotion. Such a situation often requires multiple regression analysis. This method controls for differences in multiple individual characteristics. A regression analysis identifies the relationship between each of the explanatory variables (in this case, promotion), while controlling for all other variables. Thus, an analyst using regression techniques can assess the relation between sex and promotion, while adjusting for the influence of other variales, such as experience, seniority, and education.” Litigation Services Handbook at 13 (32.5(j)).. 61 78% of workers work in Standard Titles that span three or fewer Career Stages according to the Job Title Taxonomy.

244


-31 -


Stock Levels (60-62) and under-represented in higher Stock Levels (63 and higher), and the

female coefficient is negative and statistically significant. These results are similar to the results

that I described in my original report. It is notable women are significantly and systematically

under-placed with respect to Stock Level even after controlling for Standard Title.

54. However, Dr. Saad is wrong in his interpretation of this graph. He asserts that only

3.8% of women are “misplaced” in the wrong stock level. He arrives at this figure by summing up

the value of the difference between the “true female” and “predicted, no discrimination” bars for

each stock level for which women are over-represented. However, Dr. Saad’s use of this technique

demonstrates that he does not understand how to interpret ordered probit analyses. Consider the

following example. Imagine that there are four men and four women, each of whom could be

sorted into one of five levels (1-5). Further imagine that there are pairs of workers (one of each

gender) who are identical in every way except for gender. Absent discrimination, there should be

one man and one woman in each of levels 2-5 (with no workers in level 1). However, women face

gender discrimination and each woman is placed one level below where she would be placed

absent discrimination, and each man is correctly placed.62 In this case, there is one woman and no

men in level 1, one man and one woman in each level 2-4, and no women and one man in level 5.

According to Dr. Saad’s accounting procedure, only one woman (the woman in level 1) is

misplaced, even though in actuality, each and every woman is one stock level below where she

would be absent discrimination.

62 That is, the woman who should be placed in Level 2 is instead placed in Level 1, the woman who should be placed in Level 3 is in Level 2, and so on.

245


-32 -


55. Dr. Saad also presents an analysis of new hires’ promotion “velocity,” which is the

difference between each employee’s initial and current Stock Level divided by the number of

years at Microsoft.63 Dr. Saad presents an analysis of raw “velocity” by gender, only for workers

who were first hired between September 1, 2012 and August 31, 2015. This analysis therefore

excludes 87.7% of Technical Employees. Additionally, this analysis does not control for any of

the various factors that Dr. Saad insists elsewhere are important in analyzing promotions, such as

performance review, supervisor, or the type of work each person is performing.

V. DR. SAAD’S ANALYSES OF PAY IS FLAWED

A. It is inappropriate to include Stock Level as a control variable in a pay regression

56. On July 12, 2016, counsel for Microsoft wrote a letter to the U.S. Department of

Labor’s Office of Federal Contract Compliance Programs (OFCCP) in connection with OFCCP’s

investigation into This letter

contains information about

”64 Despite Microsoft’s testimony and

documents to the contrary, Dr. Saad insists that Stock Level is not a pay band, and is therefore

appropriate to use as a control variable. He is incorrect.

57. In addition to these documents and testimony that Microsoft uses “Stock Level” as a

pay band, I have also analyzed the relationship between (a) annual salary and Stock Level and (b)

total compensation and Stock Level. Each of these regressions include only indicators for stock

63 Saad Report at ¶ 131 64 MSFT_MOUSSOURIS_00859173. See also ¶ 20 and footnotes 26 and 27 of my original report.

246


-33 -


level and year as control variables. The R-squared for the annual salary regression is .90,

indicating that stock level and year explain 90% percent of the variation in annual salary. The R-

squared for the total compensation regression is 0.81, indicating that stock level and year explain

81% of the variation in total compensation. These high R-squared values indicate that, contrary to

Dr. Saad’s assertions (and in accordance with Microsoft’s documents and testimony), Stock

Levels are functionally pay bands.

58. On pages 103 and 104, Dr. Saad presents graphs plotting the annual base salary (page

103) and total compensation (page 104) by stock level.65 He has plotted this range as a solid bar,

which suggests that pay at any point within the bar is equally common. This is not true. I have re-

created these graphs as Figures 1 and 2, but using a “box and whisker” plot. These box and

whisker plots allow for a visual representation of the range of a value, and the amount of variance.

The “box” represents the middle 50% of the range of data (the 25th to 75th percentile) also known

as the “interquartile range”. Within each box is a vertical line that represents the median value.

The “whiskers,” centered on the median, represent 1.5 times the interquartile range and are a

visual representation of the additional spread beyond the box. Figure 1 presents this plot for

annual salary, and inspection of this figure makes it clear that these stock levels (depicted on the

vertical axis) are pay bands. Each level’s “box” is non-overlapping with the next level’s “box,”

indicating that the middle 50% of individuals in each stock level earn less than the middle 50% of

individuals in the next stock level. Figure 2 is similar, but presents these plots for total

compensation. In this case, there are overlapping boxes between 59 and levels as high as 62, and

65 He describes these graphs as “Average” base salary or total compensation, but they are not.

247


-34 -


60 overlaps slightly with 61. However, in all cases, the median total compensation (represented

by the vertical line within each “box”) are strictly increasing in stock level, indicating that median

total compensation is higher at each successively higher stock level.

59. Dr. Saad performs several analyses where he includes stock level as a control, but this

is simply wrong, as stock level is a pay band. By controlling for a pay band (which is a close

proxy for pay) in a pay regression, Dr. Saad is very nearly regressing pay on itself. It is therefore

unsurprising that he finds a very small gender pay gap.66 He has essentially asked the question,

“after controlling for how much each worker earns, do women earn less than men?” This question

is meaningless, as are all of the analyses that follow.

60. Dr. Saad asserts that it is an “extreme position” to exclude Stock Level in my pay

analyses because there is a shortfall in advancement among women.67 This is nonsense. It is

simply inappropriate as a matter of econometrics to include a proxy for pay as a control variable in

a pay regression. This is not an “extreme position.”

61. It may be instructive to consider an example outside of the context of this case to

explain why it is inappropriate to regress a variable on a proxy for itself. Rather than examining

the question of gender differences in pay, imagine that you are interested in studying the height of

66 It is equally unsurprising that he finds very high R-squared values in pay regressions that include pay bands. After adding Stock Level to my Model 5, Dr. Saad finds an R-squared of 0.91. R-squared measures the percent of variation in the dependent variable (here, log total compensation) that is explained by the independent control variables. Stock Level and year alone (with no other controls) explain 81% of the variance in log total compensation. See ¶ 57, above. 67 Saad Report at ¶ 133.

248


-35 -


men and women.68 You could answer the question “what is the average difference in height

between men and women” by regressing height in inches on a gender indicator variable. The first

row of Table 9 presents the results of this analysis, and shows that women are, on average 5.1

inches shorter than men. If you were to follow Dr. Saad’s suggested method for determining the

difference in height between men and women, you would instead regress height in inches on a

gender dummy and a series of height bands.69 The second row of Table 9 presents the results of

this analysis and shows that, after controlling for what height band each individual is in, women

are 0.8 inches shorter than men on average.

62. All of Dr. Saad’s regression analysis of pay that include Stock Levels are deeply

flawed. This includes the analyses where he has randomly re-assigned some men and women to

different stock levels, based on analyses showing that women are under-advanced. Simply

changing some observations’ Stock Level does not make it appropriate to include Stock Level,

which is a pay band, as a control for a pay regression. In addition to repeating his incorrect

assertion that it is somehow appropriate to include a proxy for pay as a control in a pay regression,

Dr. Saad has also made numerous errors. Namely, in order to construct the p-values and t-

statistics, Dr. Saad simply averages the estimated p-values (and t-statistics), which is simply

incorrect.70

68 The data for this analysis are based on height data reported by Francis Galton, available at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/T0HSJ1. 69 I have used the following height bins: shorter than 62 inches; taller than or equal to 62 inches, but shorter than 66 inches; taller than or equal to 66 inches, but shorter than 70 inches; taller than or equal to 70 inches, but shorter than 74 inches; taller than 74 inches. 70 Saad Tr. at 184:4-185:6.

249


-36 -


63. Dr. Saad also presents a table and a series of graphs that he purports to be based on my

Model 5, but “with additional controls.” These include the table on page 109, the pie charts on

page 110, the scatterplots on pages 125 and 126, and the bar chart on page 127 of Dr. Saad’s

report. Each of these tables or figures includes Stock Level (which is a pay band) as a control in a

pay regression, and should be disregarded as meaningless.

64. Additionally, Dr. Saad asserts that the results of his analyses that include Stock Level

as a pay band that show that women earn statistically significantly less than men, but that this

difference is not “practically significant.”71 In support of this claim, Dr. Saad cited a hypothetical

example written by Daniel Rubinfeld, which focused on a $0.10 per hour pay difference between

men and women’s pay, which Dr. Rubinfeld argued would be “likely to be deemed practically

insignificant.”72 As noted in a paper cited by Dr. Saad, practical significance “is not a statistical

concept, but rather an imprecise term whose meaning is determined on a case-by-case basis.”73

The American Bar Association’s Econometrics handbook uses a coefficient size of 0.0001 percent

as an example of an effect that may not be practically significant.74 Applying even Dr. Saad’s

estimate of a 0.4% gender pay gap, which is much larger this threshold, implies a class-wide

damages effect of

71 Saad Report at ¶ 167. 72 Saad Report at ¶ 167, citing Rubinfeld, Daniel, “Reference Guide on Multiple Regression,” Reference Manual on Scientific Evidence: Third Edition, page 318. 73 Piette, Michael and White, Paul, “Approaches for Dealing with Small Sample Sizes in Employment Discrimination Litigation,” Journal of Forensic Economists, 12(1), 1999, pp. 43-56 at 53, cited by Saad at ¶ 116, n.87. 74 ABA Section of Antitrust Law, Econometrics (2005) at p. 15.

250


-37 -


B. Discussion of Models 1-3

65. Dr. Saad spends many pages discussing how Models 1-3 are not sufficient to show that

there is a meaningful gender pay gap after controlling for reasonable differences.75 I agree that

these models are likely to mis-state the portion of the gender pay gap that is due to gender

discrimination and not other factors (such as differences in tenure, job performance, or type of

work performed). It is for this reason that I presented additional models, which control for each

worker’s Discipline (Model 4) and Standard Title (Model 5).

C. Disaggregation between IT and Engineering

66. Dr. Saad has suggested that when examining the difference in pay between men and

women, I should disaggregate between IT and Engineering.76 This argument is based on (1)

differences in the distribution of Stock Levels (and hence pay bands) by Profession;77 (2)

differences in average pay by Profession;78 (3) differences in education level by Profession

(though his chart on page 92 shows that these data are missing for nearly half of both groups).79

He also asserts that there is little crossover between these professions, though Microsoft’s data

show that 61% of individuals who ever work in the IT Operations Profession during the period

75 Saad Report at ¶¶ 136-147. 76 Note that much of Dr. Saad’s criticism of my combining IT Operations and Engineering Professions centers around Models 1-3, which do not control for Profession. Models 4 and 5 (which control for Discipline and Standard Title, respectively, also serve to control for Profession). See Saad Report at ¶¶ 142-147. 77 Saad Report at ¶ 145. 78 Saad Report at ¶ 143. 79 Saad Report at ¶ 144.

251


-38 -


January 1, 2010 – August 31, 2016 also work in the Engineering Profession at some point during

the same time period.

67. Dr. Saad is obfuscating the fact that for Models 4 and 5, I do control for Discipline and

Standard Title (respectively) each of which already control for Profession, and I believe that these

controls are sufficient to account for the differences described by Dr. Saad.80 Nevertheless, in

Table 10, I present the results of an analysis where I interact the gender effect with a profession

indicator. This will estimate separate gender pay gaps within the IT Operations and Engineering

Professions. Other than this modification, there are no other changes to the model specifications

relative to those reported in Table 3 of my original report.

68. Column [1] presents the estimated gender pay gap for the Engineering Profession, and

Column [2] presents the t-statistic for this estimate. For each model specification, women earn

less than men and this difference is statistically significant.81 Columns [3] and [4] present the

estimated gender pay gap and its t-statistic for the IT Operations Profession. Models 4x and 5x

(which control for Discipline and Standard Title, respectively, in addition to human capital

factors) show that women earn statistically significantly less than men.82

D. Gender differences by Discipline

69. Dr. Saad notes that “women and men differ in their distributions across disciplines, and

this accounted for almost 15% of Dr. Farber’s estimated pay difference reported in his Model

80 Each Profession is associated with a mutually-exclusive list of Disciplines and Standard Titles. 81 This difference ranges from 2.6 log points in Model 5x to 7.5 log points in Models 2x and 3x. 82 Models 1x-3x do not show any statistically significant difference in pay between men and women in IT Operations, however these models do not include any controls for the type of work that each person does.

252


-39 -


3….It is notable that inserting discipline does not increase R-squared in Dr. Farber’s analysis.”83

Including “Discipline” as a control variable does, in fact, increase R-squared, from 0.51 to 0.55, so

I am unsure why Dr. Saad claims otherwise. Nevertheless, Models 4 and 5 both control for each

worker’s Discipline, which will account for the difference in pay across Disciplines. By

controlling for these factors, I have accounted for differences in the distribution of men and

women across Disciplines.

E. Career Stage

70. Dr. Saad argues that I should have included controls for each worker’s Career Stage.84

Career Stage and Discipline are largely, but not entirely, determinative of each worker’s Standard

Title. Though most Standard Titles are associated with two or three Career Stages, there are some

Standard Titles (such as Software Engineering in 2016) that span a wider range of Career Stages.

In Table 11, I present the previous analyses and include Model 6, which adds a series of Career

Stage indicator variables to Model 5.85 This analysis shows a gender pay gap of 0.019 log points

or 1.9% after controlling for human capital factors, Standard Title and Career Stage.

71. However, I reiterate my finding from the original report that Career Stage (whether

entering the regression analysis separately or as information contained within Standard Title) is a

tainted variable and therefore the inclusion of Career Stage or Standard Title as a control variable

in a pay regression will likely understate the gender pay gap. This conclusion is based on the

83 Saad Report at ¶ 147 84 Saad Report at ¶¶ 150-156. 85 In all, Model 6 controls for gender, compensation year, age and its square, tenure at Microsoft ant its square, state in which the employee works, city in which the employee works, PayScaleType, performance rating, Standard Title, and Career Stage.

253


-40 -


analysis of gender disparity in Career Stages discussed in paragraphs 58-62 and Figures 1-2 of my

original report. This analysis shows that women are systematically in lower Career Stages than

would be predicted in the absence of discrimination.

72. Dr. Saad asserts that Figures 1 and 2 of my original report show that “only” 7.3% of

women are in a Career Stage lower than my model predicts.86 However, this analysis is based on

the same flawed logic and misunderstanding of ordered probit analyses as infected his calculation

of underplaced women at pages 82-83 of his report. Namely, simply comparing the difference

between the heights of the “true” and “predicted” bars does not answer the question of how many

women are underleveled. See the discussion at paragraph 54, above.

F. Dr. Saad’s other criticisms of pay analysis

1. Part time work and leaves of absence

73. Dr. Saad argues that my analysis should have accounted for the fact that some workers

at Microsoft work part time. As noted in Dr. Saad’s report, part time work is rare among

Technical Employees at Microsoft, with approximately 0.26% of employee-years reflecting part

time work. Likewise, Dr. Saad argues that I should have accounted for the fact that some workers

take leaves of absence (LOA). I agree with Dr. Saad that it is appropriate to control for both of

these factors. Table 12 presents the results of my analysis when I alternately control for part time

work, control for leave of absences, and control for both of these.87 I make these modifications to

86 Saad Report at ¶ 151. 87 In particular, I control for part time work by adding to the regression an indicator for whether the employee is designated as a part time worker at any point in the compensation year. I control for leaves of absence by adding two indicators to the regression: one indicates whether the employee took leave during the current compensation year, while one indicates whether the employee took leave during the previous compensation year.

254


-41 -


both Model 4 and Model 5. In either case, these modifications have a small impact on my

estimate of the gender difference in pay. The alternative estimates for Model 4 range from 5.1%

to 5.3%, compared to the baseline of 5.5% without controlling for part-time work or leaves of

absence. Likewise, the alternative estimates for Model 5 range from 2.3% to 2.5%, compared to

the baseline of 2.6% without modifications.

2. Patents

74. Dr. Saad also criticizes my analysis for not including a number of factors that are not

possible to control for. For instance, Dr. Saad argues that Microsoft may pay individuals who

hold patents more than individuals who do not.88 Microsoft did not provide any data on patents

held by Microsoft employees, without which it is impossible to control for patents in any analysis.

3. College hires versus lateral hires

75. Dr. Saad argues that I should have controlled for differences in college and lateral

hires. He proposes a method for doing so, based on the “hire source” and “candidate source” in

the data provided by Microsoft.89 Note that, though hire source information is available for 90%

of observations, Dr. Saad limited his analysis to only those workers who were hired after

September 1, 2012. He does not explain why he excluded class members who were hired prior to

September 1, 2012 from this analysis.90 Note also that he performed this analysis only for those

workers in the Engineering Profession and does not analyze IT Operations workers.

88 Saad Report at ¶ 183. 89 Saad Report at ¶ 186. Note that “hire source” is based on the requisition form, and is not based on the characteristics of the individual who was actually hired for the position. 90 Note that of the approximately employee-year observations available in the data, of whoich include hire source data, Dr. Saad bases this analysis on approximately

255


-42 -


76. Before turning to my own version of this analysis, it is worth discussing the final row

of the table on page 119 of Dr. Saad’s report. In this analysis, Dr. Saad is regressing total

compensation on the Stock Level (i.e. pay band) at hire. Regressing pay on Stock Level (i.e. pay

band) remains inappropriate for all of the reasons discussed in paragraphs 56-61 above, even if is

past (rather than current) Stock Level. Because any worker’s current pay is heavily influenced by

their past pay—especially pay in the recent past—this is again simply regressing pay on a proxy

for pay. Furthermore, because Dr. Saad is examining only new hires, the problem of controlling

for on-hire Stock Level in a pay regression will be more pronounced than controlling for on-hire

Stock Level for people who have been with Microsoft for a longer period of time.91

77. I have replicated Dr. Saad’s analysis reported on page 119 of Dr. Saad’s report, but

rather than limiting my analysis to new hires, I have included all Engineering workers for whom I

have sufficient data namely those with known hire source92 and known initial stock level93,

observations. At his deposition, Dr. Saad was asked why he limited his analysis to only individuals who were hired after September 1, 2012. In his response, he explained that he wanted to compare individuals who came directly from college (and presumably had no prior experience) with those who came to Microsoft laterally (and who presumably do have prior experience). This answer is not responsive to the question of why he only examined individuals who were hired after September 1, 2012. See Saad Tr. at 186:8-187:5. 91 In order to be included in Dr. Saad’s analysis on page 119, each individual must have been hired after September 1, 2012. This means that by definition, workers in this analysis can only have been employed by Microsoft for at most 4 years. The average tenure of the lateral hires Dr. Saad includes in this table is 10.5 months. 92 A , representing roughly of employee years, have a hire source of , while all other non-missing hire source values are “College” or “Industry.” I follow Dr. Saad in reclassifying the hire source as “Industry” for purposes of this analysis. I likewise follow Dr. Saad in reclassifying hire source as “College” if the field Candidate Source Category includes the words “College” or “University”.

256


-43 -


regardless of hire date. The results of this analysis are presented in Table 13. I find, among

college hires, women earn 2.5% less than men under Model 4, or 1.4% less under Model 5.

Among industry hires, women earn 5.9% less than men under Model 4, or 2.7% less under Model

5. If I follow Dr. Saad in controlling for initial stock level when analyzing industry hires, I still

find that women are underpaid by 1.2% (in both Models 4 and 5). All these differences are

statistically significant. This analysis shows that when the analysis is not limited to the small

fraction of workers who were hired during the class period, for both “college hires” and “lateral

hires,” women earn statistically significantly less than men. This remains true for lateral hires

even after controlling for on-hire Stock Level (though I believe that it is inappropriate to do so, as

Stock Level is a pay band).

4. Business Products

78. Dr. Saad argues that I should have controlled for different areas of the company in

which Technical Employees may work (in addition to controlling for Discipline or Standard Title,

which are controlled for in my Models 4 and 5).94 Dr. Saad also asserts that “we have little

information on what types of business products and specific areas within those products

employees work on while at Microsoft.”95 However, we do have information from Microsoft on

14 organizational taxonomies, which constitute the various levels of 3 distinct organizational

hierarchies, though I understand that Microsoft has not provided any evidence that these

93 Initial stock level is known for all employees hired during the discovery period, as well as for employees who were hired before the discovery period but whose “on-hire” stock level has been provided by Microsoft. 94 Saad Report at ¶ 141. 95 Saad Report at ¶ 187.

257


-44 -


organizational divisions are meaningful distinctions in the type of work each employee performs.

Nevertheless, I have added controls for these organizational hierarchies.

79. In Tables 14 and 15, I present the results of analyses that add control variables for

various levels of each organization hierarchy to Model 4 (Table 14) and Model 5 (Table 15),

respective. In each Table, Row 1 is baseline for that model, while subsequent rows add additional

information for a particular level of one of Microsoft’s organization hierarchies. For instance, the

second row of Table 16 controls for “Org Summary”, which is the highest level of the Channel

hierarchy and divides employees into 11 categories. Meanwhile, the seventh row controls for

“Function Detail”, which is second-finest96 level of the Channel Hierarchy and divides employees

into 1,600 categories. With these additional controls, the estimated pay gap under Model 4 ranges

from 4.8% to 5.4% (compared to a baseline of 5.4%), and the estimated pay gap under Model 5

ranges from 2.5% to 3.1% (compared to baseline of 2.6%).

80. While I maintain that Microsoft’s Job Title Taxonomy (Profession, Discipline, and

Standard Title) is the preferred control for what an employee does at Microsoft, these tables

demonstrate that I can additionally control for different measures of organizational structure at

Microsoft and still find a nearly identical pay differential.97 In fact, once I’ve controlled for

Standard Title in Model 5, controlling for additional organization information often increases the

estimated pay gap.

96 The base, or finest level, of both the Channel Hierarchy and the Executive Hierarchy is “Cost Center” (see the 14th row of the table) which divides employees into 3158 categories. 97 At pages 122-123 of his report, Dr. Saad compares employees with different “functions” but the same Standard Title. “Function” is one of the controls in this analysis.

258


-45 -


VI. DR. SAAD’S ANALYSIS OF CENSUS DATA

81. Dr. Saad performs an analysis where he compares the “female representation” rate at

Microsoft as compared to a benchmark that he constructed based on data provided by the

American Community Survey. This analysis provides no information as to whether or not

Microsoft disfavors female technical employees with respect to pay or advancement.

82. This analysis should be ignored in its entirety.

VII. CONCLUSION

83. I have reviewed the Expert Report and deposition of Ali Saad. I find that his analyses

are generally incorrect, misleading, or irrelevant. Several of his analyses are so deeply flawed or

irrelevant that they should be ignored in their entirety.

84. I maintain the conclusions from my original report, that members of the Compensation

Class at Microsoft (that is, female Technical Employees in Stock Levels 59-67) receive lower

compensation, on average, than otherwise-similar men, and this difference is statistically

significant I also find that female Technical Employees in Stock Levels 60-64 at Microsoft have

a lower probability of advancement through Stock Levels and Career Stages than otherwise-

similar men, and this difference is statistically significant.

__________________

Henry Farber

February 9, 2018

259


CONTAINSCONFIDENTIALMATERIALS

ModelPercentofWoman-YearsWorkedUnderSupervisorsThatUnderpay

Women

PercentofWoman-YearsWorkedUnderSupervisorsThatUnderpay

Women,andthePayGapisStatisticallySignificant

[1] [2]4s 99.72% 98.33%5s 99.72% 98.79%

Notes:ThistablesummarizestheresultsofregressionswhereaninteractionofanindicatorforeachLevel1Supervisorand"Female"replacesthesingle"Female"dummyinModel4(firstrowofthetable)andModel5(secondrowofthetable).AnindicatorforeachLevel1Supervisorisalsoaddedtoeachmodel.Bymakingtheseadjustmentstothemodels,Icandetermineunderwhichsupervisorsagenderpaygapexists.Thetablereports[1]thepercentofwomen-yearsworkedundersupervisorsforwhomIfindapaygap,aswellas[2]thepercentageofwomenworkingundersupervisorsforwhomIfindastatisticallysignificantpaygap.

Womeninmyanalysisworkfor1of10Level1Supervisors.

PercentagesaremeasuredoutofwomenforwhomIhavedataonLevel1Supervisor.Ihavethisdataforover99%ofwomenyears.

Model4s:ControlsforGenderinteractedwithSupervisor,Supervisor,aswellas"compensationyear",employeeage(anditssquare),employeetenureatMicrosoft(anditssquare),stateinwhichtheemployeeworks,cityinwhichtheemployeeworks,PayScaleType,employees'performanceratings,andDisciplineModel5s:Model4s+addcontrolsforStandardTitle

Table1SupervisorInteractedwithGender

Howmanywomenworkundersupervisorswhoexhibitagenderpaygap?

46

260



CompensationYear

TrueStandardTitle AdjustedStandardTitleTrueTotal

CompensationPredicted

CompensationDifference

2012 SeniorProgramManager PrincipalProgramManager2013 SeniorProgramManager PrincipalProgramManager2014 SeniorPMManager PrincipalPMManager

Notes:ThistableassignsKatherineMoussouistotheStandardTitleshemostlikelywouldhavebeeninhadshebeenpromoted(AjdustedStandardTitle),andpredictshertotalcompensationbasedonModel5.

Table2CompensationPredictionsforMoussourisifshewasnotUnder-AdvancedinStockLevel

47

261



Model

[1]

PercentofWomen

WorkingUnderSupervisors

whoUnderadvance

Women

[2]

PercentofWomenWorking

UnderSuperisorswho

UnderadvanceWomenandthe

GenderGapisStatistically

Significant

4s 90.01% 89.19%

Howmanywomenworkundersupervisorswhounderadvancewomen?

SupervisorInteractedwithGender

Table3

Notes:Thistablesummarizestheresultsofregressionswhereaninteractionofan

indicatorforeachLevel1Supervisorand"Female"replacesthesingle

"Female"dummyinModel4(firstrowofthetable)andModel5(secondrow

ofthetable).AnindicatorforeachLevel1Supervisorisalsoaddedtoeach

model.Bymakingtheseadjustmentstothemodels,Icandetermineunder

whichsupervisorsagenderadvancementgapexists.Thetablereports[1]the

percentofwomen-yearsworkedundersupervisorsforwhomIfindan

advancementgap,aswellas[2]thepercentageofwomenworkingunder

supervisorsforwhomIfindastatisticallysignificantadvancementgap.

Womeninmyanalysisworkfor1of10Level1Supervisors.

PercentagesaremeasuredoutofwomenforwhomIhavedataonLevel1

Supervisor.Ihavethisdataforover99%ofwomenyears.

Model4s:Controlsfor:GenderinteractedwithSupervisor,Supervisor,aswellas"compensationyear",employeeage(anditssquare),employeetenureat

Microsoft(anditssquare),stateinwhichtheemployeeworks,cityinwhich

theemployeeworks,PayScaleType,StockLevelasofSeptember1ofthe

previousyear,employees'performanceratings,andDiscipline

48

262



Men Women Total

Promotions 33,772 7,040 40,812

PredictedPromotionsusingDr.Saad'sMethod* 19,111 4,268 23,379

PredictedPromotionsasaPercentofActual

Promotions56.59% 60.63% 57.28%

Table4

Dr.Saad'sPredictedPromotionsvs.ActualPromotions

*TotalnumberofpromotionsbasedonDr.Saad'smethodofanalysingpromotionsthat

"shouldhaveoccured."Apredictedprobabilityofpromotiongreaterorequalto0.5countsas

apromotion,whileaprobabiltylessthan0.5doesnotcountasapromotion.

Basedonaprobitanalysisofanadvancementmeasureontenure,tenuresquared,age,age

squared,year,performancemetrics,location,Discipline,andpriorStockLevel.

49

263



AverageAdvancement

Rate

GenderDifference

(MarginalEffect)Z-statistic Employee

Years

[1] [2] [3] [4][A]StockLevelAdvancement 0.317 -0.021 -8.897[B]MicrosoftPromotionIndicator 0.275 -0.024 -9.105[C]CareerStageAdvancementpriorto2014 0.143 -0.025 -7.586[D]CareerStageAdvancementpost2014 0.231 -0.033 -3.519

Table5StockLevelandCareerStageAdvancementDifferencesforMenandWomen

Basedonaprobitanalysisofanadvancementmeasureontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevelinRow[A],theMicrosoftPromotionsIndicatorvariablein[B],orCareerStageinRows[C]and[D].

50

264



AverageAdvancement

Rate

GenderDifference

(MarginalEffect)Z-statistic Employee

Years


Table6StockLevelandCareerStageAdvancementDifferencesforMenandWomen

ControllingforTimeinStockLevelorCareerStage

Basedonaprobitanalysisofanadvancementmeasureontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevelinRow[A],theMicrosoftPromotionsIndicatorvariablein[B],orCareerStageinRows[C]and[D].Probitanalysesin[A]and[B]alsoincludetimeinStockLevelandtimeinStockLevelsquared.Probitanalysesin[C]and[D]alsoincludetimeinCareerStageandtimeinCareerStagesquared.

51

265



AverageAdvancement

Rate

GenderDifference

(MarginalEffect)Z-statistic

EmployeeYears


Table7

Controllingfortwomostrecentperformancereviews.StockLevelandCareerStageAdvancementDifferencesforMenandWomen

Basedonaprobitanalysisofanadvancementmeasureontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,twomostrecentperformancemeasures,andpriorStockLevelinRow[A],theMicrosoftPromotionsIndicatorvariablein[B],orCareerStageinRows[C]and[D].

52

266



StockLevel NumberofObservations

NumberofWoman-Years

NumberofAdvancements,

Women

AdvancementRate,Women

ExpectedAdvancementRate,Women

NumberofExpected

Advancements,Women

ShortfallofAdvancements,

WomenT-Statistic

[1] [2] [3] [4] [5] [6] [7] [8]59 9,516 965 0.53 0.55 1,012 -47 -3.2360 16,847 1,503 0.44 0.45 1,526 -23 -0.9361 24,814 1,756 0.34 0.37 1,905 -149 -5.1162 30,434 1,411 0.24 0.28 1,617 -206 -10.8963 25,359 878 0.22 0.25 994 -116 -7.6164 18,381 362 0.16 0.16 378 -16 -2.69

-534

Note:

ReportedCalculationsandResults:

[5]:ResultfromProbitModel ExpectedAdvancementRate,Women[6]=[5]x[2] NumberofExpectedAdvancements,Women[7]=[3]-[6] ShortfallofAdvancements,Women

BasedonaprobitanalysisoftheStockLeveladvancementindicatorvariableontenure,tenuresquared,age,agesquared,year,twomostrecentperformancemetrics,location,Discipline,andpriorStockLevel.ThismodelisestimatedformenonlyandtheprobabilityofStockLeveladvancementispredictedforbothmenandwomen.

Column: Reports:[4]=[3]/[2] AdvancementRate,Women

Table8StockLevelAdvancementShortfall

TotalShortfall,StockLevels59-64:

AStockLeveladvancementisdefinedasachangefromalowerStockLevelintoahigherStockLevel,comparingStockLevelonSeptember1ofyeartwithStockLevelonSeptember1ofyeart-1.

Controllingfortwomostrecentperformancereviews.

53

267



DifferenceinHeight

T-Statistic R2

Model1 -5.12 -30.55 0.51Model2 -0.80 -7.68 0.91

Notes

Model1istheregressionofgender(female)onheightininches.Model2=Model1,plus"HeightBand",describedbelow.

1 Lessthan62"2 Greaterthanorequalto62",butlessthan66"3 Greaterthanorequalto66",butlessthan70"4 Greaterthanorequalto70",butlessthan74"5 Greaterthanorequalto74"

Table9RegressingHeightonGender,HeightBands

Source:FrancisGalton,2017,"Galtonheightdata",doi:10.7910/DVN/T0HSJ1,HarvardDataverse,V1,UNF:6:2ty+0YgqR2a66FlvjCuPkQ==;galton.dta

HeightBandsortsheightsofindividualsinthedataintothefollowingcategories:

54

268



[1] [2] [3] [4] [5] [6]

LogPointDifference,Engineering

T-statisticonLogPointDifference

LogPointDifference,

ITOperations


AdjustedR2EmployeeYears

Model1x -0.082 -26.50 0.002 0.26 0.022Model2x -0.075 -28.32 -0.005 -0.64 0.346Model3x -0.075 -33.19 -0.012 -1.57 0.510Model4x -0.055 -26.32 -0.044 -6.14 0.566Model5x -0.026 -22.60 -0.027 -5.79 0.761

Table10AnalysisofGenderDifferenceinTotalCompensationEstimatingSeparateGenderEffectsforeachProfession

Model1x:GenderinteractedwithProfessionModel2x:Addscontrolsfor"compensationyear",employeeage(anditssquare),employeetenureatMicrosoft(anditssquare),stateinwhichtheemployeeworks,cityinwhichtheemployeeworks,andPayScaleTypeModel3x:Addscontrolsforemployees'performanceratingsModel4x:AddscontrolsforDisciplineModel5x:Model4x+addcontrolsforStandardTitle

Byinteractinggenderwithprofession,IamabletouseasinglemodeltoestimateseperategenderpaygapsforeachProfession.Columns[1]and[2]reporttheestimatedpaygapforEngineering,whilecolumns[3]and[4]reporttheesimatedpaygapforITOperations.

55

269



LogPointDifference


P-ValuePercent

Difference AdjustedR2EmployeeYears

Model1 -0.075 -25.49 0.00 -7.2% 0.009Model2 -0.068 -26.39 0.00 -6.5% 0.344Model3 -0.069 -30.99 0.00 -6.6% 0.507Model4 -0.056 -26.93 0.00 -5.5% 0.557Model5 -0.026 -22.66 0.00 -2.6% 0.761Model6 -0.019 -18.80 0.00 -1.9% 0.806

Model1:GenderistheonlyexplanatoryvariableModel2:Addscontrolsfor"compensationyear",employeeage(anditssquare),employeetenureatMicrosoft(anditssquare),stateinwhichtheemployeeworks,cityinwhichtheemployeeworks,andPayScaleTypeModel3:Addscontrolsforemployees'performanceratingsModel4:AddscontrolsforDiscipline.Model5:AddscontrolsforStandardTitleModel6:AddscontrolsforCareerStage

Table11AnalysisofGenderDifferenceinTotalCompensation

IntroducingCareerStageControls

56

270



LogPointDifference


P-valuePercent


UnmodifiedModel4(Baseline) -0.056 -26.93 0.000 -5.46% 0.557Model4+PartTimeDummy -0.054 -25.92 0.000 -5.22% 0.559Model4+LOADummies -0.054 -24.86 0.000 -5.29% 0.587Model4+PartTimeDummy&LOADummies -0.052 -23.99 0.000 -5.07% 0.589UnmodifiedModel5(Baseline) -0.026 -22.66 0.000 -2.61% 0.761Model5+PartTimeDummy -0.024 -21.13 0.000 -2.38% 0.763Model5+LOADummies -0.025 -20.63 0.000 -2.51% 0.793Model5+PartTimeDummy&LOADummies -0.023 -19.31 0.000 -2.28% 0.795

Note:Foreachofthemodels4and5,IpresentmybaselinefiguresaswellasthreevariationsThefirstvariationaddsadummyvariableforemployee-yearsthatincludeparttimework.Thesecondvariationaddsdummiesindicatingwhethertheemployeetookaleaveofabsencein(1)thecurrentyearand(2)thepreviousyear.Thethirdvariationcombinesbothmodificaitons.

Table12AnalysisofGenderDifferenceinTotalCompensationControlforPartTimeWorkandLOAinModels4and5

57

271



Table13

Hire Source LogPointDifference


P-value PercentDifference AdjustedR2

EmployeeYears

College -0.025 -9.49 0.00 -2.49% 0.692Industry -0.061 -20.47 0.00 -5.88% 0.545

Industry,withControlforInitialStockLevel -0.013 -8.96 0.00 -1.25% 0.859

Hire Source LogPointDifference


P-valuePercent


College -0.014 -8.04 0.00 -1.42% 0.773Industry -0.028 -18.08 0.00 -2.74% 0.786

Industry,withControlforInitialStockLevel -0.012 -9.95 0.00 -1.21% 0.876

Table13VariationsonSaad'sHire-SourceRegressions

Model4

Model5

Notes:ThistablepresentstwoversionsofDr.Saad'stableonpg119ofhisreport.WhereDr.SaadlimitstheanalysistoengineeringemployeeshiredafterSept1,2012,IincludeallengineeringemployeesforwhominitialStockLevelisknown(thiswillincludeallengineeringemployeeshiredduringthediscoveryperiod,aswellasolderhiresforwhom"onhirelevel"isnotmissinginthedata.)

ThetoppanelshowstheresultsforModel4,whilethebottomrowshowstheresultsforModel5(themodelDr.Saadpresentedinhistable).

Ineachpanel,thefirstrowpresentsresultsforModel5foremployeeswhose"hiresource"is"College",themiddlerowpresentsresultsforemployeeswhose"hiresource"is"Industry"(whatDr.Saadreferstoas"lateral"hires),andthelastrowisalsoforemployeeswhose"hiresource"is"Industry",withthechangethatdummiesfor"onhirelevel"havebeenaddedtothemodel.

58

272



CategoriesinHierarchy

LogPointDifference

T-statistic PercentDifference AdjustedR2

EmployeeYears

Baseline N/A -0.056 -26.932 -5.46% 0.557OrgSummary 11 -0.056 -26.917 -5.45% 0.557Org 24 -0.056 -26.920 -5.45% 0.557OrgDetail 164 -0.054 -26.487 -5.30% 0.567FunctionSummary 381 -0.056 -22.028 -5.42% 0.552Function 1019 -0.052 -26.123 -5.06% 0.586FunctionDetail 1624 -0.051 -26.023 -5.00% 0.592OrgExecSummary 11 -0.056 -26.840 -5.43% 0.560OrgExc 45 -0.056 -23.911 -5.45% 0.543OrgExecDetail 243 -0.055 -23.887 -5.36% 0.549FunctionExecSummary 750 -0.054 -23.583 -5.23% 0.563FunctionExec 1360 -0.052 -22.999 -5.04% 0.573FunctionExecDetail 1943 -0.051 -22.643 -4.93% 0.579

BothHierarchies

CostCenter 3201 -0.049 -25.491 -4.81% 0.603

Standalone Organization 8 -0.056 -26.983 -5.46% 0.557

ChannelHierarchy

ExecutiveHierarchy

Notes:EachrowofthistablereportsesimatesbasedonamodifiedversionofModel4,wheredummiesforthespecifiedhierarchyvariablehavebeenaddedtoourstandardModel4.

ThereareseveraloverlappinghierarchiesatMicrosoft:Ihavefulldataformostlevelsofthe"Channel"hierarchy(exceptforthe"functionsummary"level,whichisonlyavailableinlimitedyears).Ihavefulldataforonelevelofthe"Executive"hierarchy,whileavailabilityofdataforotherlevelsarelimitedtoparticularyears.

"CostCenter"formsthebaseofbothhierarchies,while"Organization"ispartofneitherhierarchy.

ChannelHierarchydata(excludingFunctionSummary)andOrgExecSummy,CostCenter,andOrganizationcomefromMSPeople.TheremainingfieldscomefromDr.Saad'sbackupmaterials(archived_people_review_RE2.dta).

Table14Model4withOrganizationalControls

59

273



CategoriesinHierarchy

LogPointDifference

T-statistic PercentDifference AdjustedR2

EmployeeYears

BaselineModel5 N/A -0.026 -22.656 -2.61% 0.761OrgSummary 11 -0.026 -22.691 -2.61% 0.761Org 24 -0.026 -22.682 -2.61% 0.762OrgDetail 164 -0.026 -22.556 -2.58% 0.765FunctionSummary 381 -0.032 -18.184 -3.11% 0.724Function 1019 -0.026 -22.757 -2.56% 0.771FunctionDetail 1624 -0.026 -22.548 -2.53% 0.773OrgExecSummary 11 -0.026 -22.527 -2.59% 0.763OrgExc 45 -0.030 -19.190 -2.92% 0.732OrgExecDetail 243 -0.029 -19.362 -2.90% 0.733FunctionExecSummary 750 -0.029 -19.165 -2.85% 0.741FunctionExec 1360 -0.028 -18.909 -2.79% 0.745FunctionExecDetail 1943 -0.028 -18.770 -2.76% 0.747

BothHierarchies

CostCenter 3201 -0.025 -22.231 -2.47% 0.777

Standalone Organization 8 -0.026 -22.685 -2.61% 0.762

ChannelHierarchy

ExecutiveHierarchy

Notes:EachrowofthistablereportsesimatesbasedonamodifiedversionofModel5,wheredummiesforthespecifiedhierarchyvariablehavebeenaddedtoourstandardModel5.

ThereareseveraloverlappinghierarchiesatMicrosoft:Ihavefulldataformostlevelsofthe"Channel"hierarchy(exceptforthe"functionsummary"level,whichisonlyavailableinlimitedyears).Ihavefulldataforonelevelofthe"Executive"hierarchy,whileavailabilityofdataforotherlevelsarelimitedtoparticularyears.

"CostCenter"formsthebaseofbothhierarchies,while"Organization"ispartofneitherhierarchy.

ChannelHierarchydata(excludingFunctionSummary)andOrgExecSummy,CostCenter,andOrganizationcomefromMSPeople.TheremainingfieldscomefromDr.Saad'sbackupmaterials(archived_people_review_RE2.dta).

Table15Model5withOrganizationalControls

60

274


CONTA

INSCO

NFIDE

NTIALM

ATER

IALS

Annu

al S

alar

y

596061626364656667

Figu

re 1

Annu

al S

alar

y by

Sto

ck L

evel

, 201

4

61

275


CONTA

INSCO

NFIDE

NTIALM

ATER

IALS

Tota

l Com

pens

atio

n

596061626364656667

Figu

re 2

Tota

l Com

pens

atio

n by

Sto

ck L

evel

, 201

4

62

276


APPENDIX1

63

277


APPENDIX1ATimePeriod:January1,2010–May31,2016

64

278



Level EmployeeYears CountofWomen PercentWomen CountofMen PercentMen

59 3,317 19.6% 13,626 80.4%60 5,156 21.0% 19,351 79.0%61 7,452 20.9% 28,232 79.1%62 8,040 19.8% 32,615 80.2%63 6,094 16.3% 31,300 83.7%64 3,813 13.7% 23,990 86.3%65 1,795 11.4% 13,946 88.6%66 920 10.0% 8,297 90.0%67 363 6.7% 5,043 93.3%68 151 7.2% 1,938 92.8%69 65 6.3% 970 93.7%70 41 7.2% 532 92.8%80 11 6.7% 152 93.3%81 0 0.0% 23 100.0%82 0 0.0% 8 100.0%83 0 0.0% 4 100.0%0 326 25.3% 964 74.7%

Overall 37,544 17.2% 180,991 82.8%

Note:StockLevel0isnotdirectlycomparabletootherStockLevels.MicrosoftassignsemployeestoStockLevel0whiletheyareawaitingassignmenttoaStandardLevel.

ThisisallworkersemployedinEngineeringorITProfessionsinthelistedStockLevelswhoseannualsalaryisgreaterthan$1.AnnualSalaryiscodedas"1"duringperiodsinwhichanemployeeisengagedinaJointVenturewithanothercompany,andtheothercompany,ratherthanMicrosoft,processesthatemployee'spayroll.

TABLE1AGenderCountsbyStockLevel

DroppingAllEmployeeYearsinwhichanEmployeewasaManagerofManagers

In-ClassStockLevel

Out-of-ClassStockLevel

65

279



Year DifferenceT-StatisticonDifference PctDifference

T-StatisticonPctDiff EmployeeYears

[2] [3] [4] [5]2011 -18.53 -7.7% -20.532012 -20.08 -6.9% -20.592013 -20.83 -6.9% -20.142014 -19.70 -6.8% -19.712015 -19.90 -7.1% -19.902016 -19.75 -7.3% -19.27Overall -26.80 -7.2% -47.08

TABLE2ADifferencebetweenWomen'sandMen'sTotalCompensationbyYear

ThissampleincludesworkersinEngineeringorITOperationsProfessionsinStockLevels59-67whoarenotemployedinanyprofessionsotherthanEngineeringorITOperationsatanypointinthesalaryyear,andwhobegintheyearinStockLevels59-67.Additionally,thissampleislimitedtoemployee-yearswithnon-missingvaluesfortenureandage.Ihavealsodroppedemployee-yearsinwhichannualsalaryis1,orinwhichCareerStageisoneofthefollowing:ATR-C;ATR-D;ATR-E;IC-0;orMA.

AnnualSalaryiscodedas"1"duringperiodsinwhichanemployeeinengagedinaJointVenturewithanothercompany,andtheothercompany,ratherthanMicrosoft,processesthatemployee'spayroll.

NegativevaluesinColumns[1]or[3]representloweraveragecompensationforFemaleTechnicalEmployees.


66

280



LogPoint

Difference

T-statisticon

LogPoint

Difference

P-ValuePercent

DifferenceAdjustedR

2Employee

Years

Model1 -0.075 -25.54 0.00 -7.2% 0.009

Model2 -0.068 -26.44 0.00 -6.5% 0.345

Model3 -0.068 -31.05 0.00 -6.6% 0.508

Model4 -0.056 -26.94 0.00 -5.4% 0.559

Model5 -0.026 -22.71 0.00 -2.6% 0.764

Model1:GenderistheonlyexplanatoryvariableModel2:Addscontrolsfor"compensationyear",employeeage(anditssquare),employee

tenureatMicrosoft(anditssquare),stateinwhichtheemployeeworks,cityinwhichthe

employeeworks,andPayScaleType

Model3:Addscontrolsforemployees'performanceratings

Model4:AddscontrolsforDisciplineModel5:AddscontrolsforStandardTitle

TABLE3A

AnalysisofGenderDifferenceinTotalCompensation


67

281



CoefficientonFemale

T-statistic EmployeeYears

ContributionRanking -0.018 -0.98CommitmentRating -0.035 -1.84PerformanceRating 0.006 0.46RewardOutcome 0.005 0.35

RelevantTimePeriodforEachMetric:ContributionRanking: 2011CommitmentRating: 2011PerformanceRating: 2012-2014RewardOutcome: 2015-2016

Thistablereportstheresultsoforder-probitanalysesofemployeeperformanceratings.EachrowofthistableconcernsadistinctperformancemetricusedbyMicrosoft.Theonlyexplanatoryvariableineachanalysisisgender.

TABLE4AAnalysisofGenderDifferenceinPeformanceMetrics


68

282



StockLevel Women Men

59 52.6% 52.7%60 44.0% 44.4%61 33.7% 37.3%62 24.0% 29.7%63 22.0% 25.3%64 15.8% 17.5%


StockLevelAdvancement,

Womenvs.Men

TABLE5A


69

283



AverageAdvancement

Rate

Difference(MarginalEffect)

Z-statistic EmployeeYears

[1] [2] [3] [4][A]StockLevelAdvancement 0.317 -0.021 -8.937[B]CareerStageAdvancementpriorto2014 0.143 -0.025 -7.586[C]CareerStageAdvancementpost2014 0.231 -0.033 -3.534

TABLE6AStockLevelandCareerStageAdvancementDifferencesforMenandWomen

Basedonaprobitanalysisofanadvancementmeasureontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevelinRow[A]orCareerStageinRows[B]and[C].


70

284




NumberofWoman-Years


Women



NumberofExpected

Advancements,Women


WomenT-Statistic

[1] [2] [3] [4] [5] [6] [7] [8]59 9,513 964 0.526 0.548 1,006 -42 -4.7160 16,844 1,503 0.440 0.444 1,516 -13 -1.1161 24,806 1,754 0.337 0.362 1,886 -132 -8.8962 30,422 1,410 0.240 0.273 1,606 -196 -15.0763 25,346 876 0.220 0.248 990 -114 -12.0464 18,378 362 0.158 0.164 376 -14 -2.62

TotalShortfall,StockLevels59-64: -497

Note:


[7]=[3]-[6] ShortfallofAdvancements,Women

[4]=[3]/[2] AdvancementRate,Women[5]:ResultfromProbitModel ExpectedAdvancementRate,Women[6]=[5]x[2] NumberofExpectedAdvancements,Women

Column: Reports:

TABLE7AStockLevelAdvancementShortfall

AStockLeveladvancementisdefinedasachancefromalowerStockLevelintoahigherStockLevel,comparingStockLevelonSeptember1ofyeartwithStockLevelonSemptember1ofyeart-1.

BasedonaprobitanalysisoftheStockLeveladvancementindicatorvariableontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevel.ThismodelisestimatedformenonlyandtheprobabilityofStockLevelAdvancementispredictedforbothmenandwomen.


71

285



PercentDifferenceinTotal

CompensationT-statistic AdjustedR2

EmployeeYears

Damages

Model1 -7.2% -26.53 0.009Model2 -6.5% -27.36 0.345Model3 -6.6% -32.14 0.508Model4 -5.4% -27.71 0.559Model5 -2.6% -23.02 0.764

Thetotalin-classcompensationis:

Model1:GenderistheonlyexplanatoryvariableModel2:Addscontrolsfor"compensationyear",employeeage(anditssquare),employeetenureatMicrosoft(anditssquare),stateinwhichtheemployeeworks,cityinwhichtheemployeeworks,andPayScaleTypeModel3:Addscontrolsforemployees'performanceratingsModel4:AddscontrolsforDisciplineModel5:AddscontrolsforStandardTitle

TABLE8ADamagesAnalysis


72

286



IexcludeMCareerStages1-3fromthisanalysis.

FIGURE1A

Note:Theheightofeachbarrepresentstheproportionofindividualsineachcareerstage.IexcludeMCareerStages1-3fromthisanalysis.

Note:ThePredicted,NoDiscriminationdistributioniscalculatedbasedonanorderedprobitanalysisthatincludeseachemployee’sage,tenure,location,performanceratings,andDiscipline,aswellasthecompensationyearandanindicatorforfemale.Thecoefficientsofthisanalysisareusedtopredictthefemaledistributionassumingeachfemalewasmale(thefemaleindicatorsettozero)butotherwisehadherobservedcharacteristics.

0.1

.2.3

.4

1 2 3 4 5 6

Combined IC/L/M4-M9 (2011-2016)Career Stage by Gender - True Distribution (Percent)

Female Male

Female Coefficient: -.28 T-Stat: -21.86

0.1

.2.3

.4

1 2 3 4 5 6

Combined IC/L/M4-M9 (2011-2016)Career Stage: True Female Distrbution vs Model 4 (Counterfactual)

True Female Predicted, No Discrimination

73

287



FIGURE2A

Note:TheheightofeachbarrepresentstheproportionofindividualsineachCareerStage.


0.1

.2.3

.4

1 2 3 4 5 6

Combined IC/L/M, 2014-2016Career Stage by Gender - True Distribution (Percent)

Female Male


0.1

.2.3

.4

1 2 3 4 5 6

Combined IC/L/M, 2014-2016Career Stage: True Female Distrbution vs Model 4 (Counterfactual)


74

288



FIGURE3A

Note:TheheightofeachbarrepresentstheproportionofindividualsineachStockLevel.


0.0

5.1

.15

.2.2

5

59 60 61 62 63 64 65 66 67

Stock Level by Gender - True Distribution (Percent)

Female Male


0.0

5.1

.15

.2.2

5

59 60 61 62 63 64 65 66 67

Stock Level: True Female Distribution vs Model 4 (Counterfactual)


75

289


APPENDIX1BTimePeriod:January1,2010–August31,2016

76

290




59 3,310 19.5% 13,635 80.5%60 5,158 21.1% 19,339 78.9%61 7,455 20.9% 28,225 79.1%62 8,048 19.8% 32,634 80.2%63 6,100 16.3% 31,312 83.7%64 3,821 13.7% 24,020 86.3%65 1,799 11.4% 13,953 88.6%66 921 10.0% 8,300 90.0%67 363 6.7% 5,046 93.3%68 151 7.2% 1,938 92.8%69 65 6.3% 970 93.7%70 41 7.2% 532 92.8%80 11 6.7% 152 93.3%81 0 0.0% 23 100.0%82 0 0.0% 8 100.0%83 0 0.0% 4 100.0%0 325 25.3% 962 74.7%

overall 37,568 17.2% 181,053 82.8%


TABLE1BGenderCountsbyStockLevel


In-ClassStockLevel



77

291



Year Difference T-StatisticonDifference

PercentDifference T-StatisticonPercentDifference

EmployeeYears

[1] [2] [3] [4] [5]2011 -18.53 -7.7% -20.532012 -20.08 -6.9% -20.592013 -20.83 -6.9% -20.142014 -19.70 -6.8% -19.712015 -19.90 -7.1% -19.902016 -19.14 -7.3% -18.89Overall -26.75 -7.2% -46.88


TABLE2BDifferencebetweenWomen'sandMen'sTotalCompensationbyYear




78

292



LogPoint

Difference

T-statisticonLog

PointDifferenceP-Value

Percent

DifferenceAdjustedR

2Employee

Years

Model1 -0.075 -25.49 0.00 -7.2% 0.009

Model2 -0.068 -26.39 0.00 -6.5% 0.344

Model3 -0.069 -30.99 0.00 -6.6% 0.507

Model4 -0.056 -26.93 0.00 -5.5% 0.557

Model5 -0.026 -22.66 0.00 -2.6% 0.761

Model1:GenderistheonlyexplanatoryvariableModel2:Addscontrolsfor"compensationyear",employeeage(anditssquare),employeetenure

atMicrosoft(anditssquare),stateinwhichtheemployeeworks,cityinwhichtheemployee

works,andPayScaleType



TABLE3B



79

293



CoefficientonFemale


ContributionRanking -0.018 -0.98CommitmentRating -0.035 -1.84PerformanceRating 0.006 0.46RewardOutcome 0.005 0.38



TABLE4BAnalysisofGenderDifferenceinPeformanceMetrics


80

294




59 52.6% 52.7%

60 44.0% 44.4%

61 33.7% 37.3%

62 24.0% 29.7%

63 22.0% 25.3%

64 15.8% 17.5%

TABLE5B

StockLevelAdvancement

Astockleveladvancementisdefinedasa

changefromalowerstocklevelintoa

higherstocklevel,comparingstocklevel

onSeptember1ofyeartwithstocklevelonSeptember1ofyeart-1.

DroppingAllEmployeeYearsinwhichan

EmployeewasaManagerofManagers

Womenvs.Men

81

295



AverageAdvancement

Rate



[1] [2] [3] [4][A]StockLevelAdvancement 0.317 -0.021 -8.897[B]CareerStageAdvancementpriorto2014 0.143 -0.025 -7.586[C]CareerStageAdvancementpost2014 0.231 -0.033 -3.519

TABLE6BStockLevelandCareerStageAdvancementDifferencesforMenandWomen

DroppingallEmployeeYearsinwhichanEmployeewasaManagerofManagers

Basedonaprobitanalysisofanadvancementmeasureontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevelinRow[A]orCareerStageinRows[B]and[C].

82

296




NumberofWoman-Years


Women



NumberofExpected

Advancements,Women


WomenT-Statistic

[1] [2] [3] [4] [5] [6] [7] [8]59 9,516 965 0.526 0.551 1,012 -47 -3.0860 16,847 1,503 0.440 0.448 1,531 -28 -1.1061 24,814 1,756 0.337 0.366 1,905 -149 -5.2362 30,434 1,411 0.240 0.275 1,617 -206 -10.4163 25,359 878 0.220 0.250 999 -121 -7.7364 18,381 362 0.158 0.165 378 -16 -2.80

-539

Note:



BasedonaprobitanalysisoftheStockLeveladvancementindicatorvariableontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevel.ThismodelisestimatedformenonlyandtheprobabilityofStockLeveladvancementispredictedforbothmenandwomen.



TABLE7BStockLevelAdvancementShortfall



83

297





EmployeeYears

Damages




TABLE8BDamagesAnalysis


84

298




FIGURE1B



0.1

.2.3

.4

1 2 3 4 5 6


Female Male


0.1

.2.3

.4

1 2 3 4 5 6



85

299



FIGURE2B



0.1

.2.3

.4

1 2 3 4 5 6


Female Male


0.1

.2.3

.4

1 2 3 4 5 6



86

300



FIGURE3B



0.0

5.1

.15

.2.2

5

59 60 61 62 63 64 65 66 67


Female Male


0.0

5.1

.15

.2.2

5

59 60 61 62 63 64 65 66 67



87

301


APPENDIX1CTimePeriod:September1,2012–August31,2016

88

302




59 2,330 19.9% 9,385 80.1%60 3,140 20.8% 11,931 79.2%61 4,663 20.7% 17,884 79.3%62 5,480 20.1% 21,782 79.9%63 4,269 16.5% 21,669 83.5%64 2,806 14.0% 17,227 86.0%65 1,293 11.5% 9,984 88.5%66 714 10.8% 5,903 89.2%67 259 6.7% 3,628 93.3%68 112 7.7% 1,349 92.3%69 44 6.1% 677 93.9%70 29 7.5% 358 92.5%80 10 7.0% 132 93.0%81 0 0.0% 22 100.0%82 0 0.0% 8 100.0%83 0 0.0% 4 100.0%0 146 23.2% 483 76.8%

overall 25,295 17.1% 122,426 82.9%


In-ClassStockLevel


TABLE1CGenderCountsbyStockLevel



89

303



Year DifferenceT-StatisticonDifference PercentDifference

T-StatisticonPercentDifference EmployeeYears

[1] [2] [3] [4] [5]2013 -20.83 -6.9% -20.142014 -19.70 -6.8% -19.712015 -19.90 -7.1% -19.902016 -19.14 -7.3% -18.89Overall -24.77 -7.1% -38.91


TABLE2CDifferencebetweenWomen'sandMen'sTotalCompensationbyYear




90

304



LogPoint

Difference

T-statisticon

LogPoint

Difference

P-Value

Percent

DifferenceAdjustedR

2

Employee

Years

Model1 -0.074 -23.26 0.00 -7.1% 0.009

Model2 -0.064 -23.81 0.00 -6.2% 0.333

Model3 -0.065 -27.82 0.00 -6.3% 0.495

Model4 -0.056 -25.11 0.00 -5.4% 0.537

Model5 -0.027 -20.19 0.00 -2.7% 0.739

Model1:GenderistheonlyexplanatoryvariableModel2:Addscontrolsfor"compensationyear",employeeage(anditssquare),employee

tenureatMicrosoft(anditssquare),stateinwhichtheemployeeworks,cityinwhichthe

employeeworks,andPayScaleType



TABLE3C



91

305



CoefficientonFemale


PerformanceRating -0.006 -0.46RewardOutcome 0.005 0.38



TABLE4CAnalysisofGenderDifferenceinPeformanceMetrics


92

306




59 59.0% 56.4%

60 48.6% 49.2%

61 36.8% 40.0%

62 25.8% 32.4%

63 23.4% 26.7%

64 18.0% 20.1%

TABLE5C

StockLevelAdvancement,

Astockleveladvancementisdefinedasa

changefromalowerstocklevelintoa

higherstocklevel,comparingstocklevel

onSeptember1ofyeartwithstocklevelonSeptember1ofyeart-1.

DroppingAllEmployeeYearsinwhichan

EmployeewasaManagerofManagers

Womenvs.Men

93

307



AverageAdvancement

Rate



[1] [2] [3] [4][A]StockLevelAdvancement 0.339 -0.022 -6.829[B]CareerStageAdvancementpost2014 0.231 -0.033 -3.519

TABLE6CStockLevelandCareerStageAdvancementDifferencesforMenandWomen


Basedonaprobitanalysisofanadvancementmeasureontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevelinRow[A]orCareerStageinRow[B].

94

308




NumberofWoman-Years


Women



NumberofExpected

Advancements,Women


WomenT-Statistic

[1] [2] [3] [4] [5] [6] [7] [8]59 5,578 643 0.590 0.607 662 -19 -3.1860 9,188 878 0.486 0.499 902 -24 -2.7361 13,766 1,066 0.368 0.393 1,138 -72 -6.8862 18,133 913 0.258 0.300 1,059 -146 -12.9463 15,938 591 0.234 0.263 665 -74 -9.3464 12,084 283 0.180 0.187 295 -12 -2.53

-347

Note:



BasedonaprobitanalysisoftheStockLeveladvancementindicatorvariableontenure,tenuresquared,age,agesquared,year,performancemetrics,location,Discipline,andpriorStockLevel.ThismodelisestimatedformenonlyandtheprobabilityofStockLeveladvancementispredictedforbothmenandwomen.



TABLE7CStockLevelAdvancementShortfall



95

309





EmployeeYears Damages



Thetotalin-classcompensationreportedherediffersfromthefigurereportedinTable8B,becausethiscalculationdropscertainin-classcompensationearnedbetweenMay14,2012andAugust31,2012.


TABLE8CDamagesAnalysis


96

310




FIGURE1C



0.1

.2.3

.4

1 2 3 4 5 6


Female Male

Female Coefficient: -.3 T-Stat: -21

0.1

.2.3

.4

1 2 3 4 5 6



97

311



FIGURE2C



0.1

.2.3

.4

1 2 3 4 5 6


Female Male


0.1

.2.3

.4

1 2 3 4 5 6



98

312



FIGURE3C



0.0

5.1

.15

.2.2

5

59 60 61 62 63 64 65 66 67


Female Male


0.0

5.1

.15

.2.2

5

59 60 61 62 63 64 65 66 67



99

313


APPENDIX2

100

314



1. aimstoretainhertalent2. aimstoretainhistalent3. amazonoffer4. aswellasretaininpmc5. attritionrisk6. azurenotificationhubisexpanding7. azuresupportneeds8. azuresupportrequiresengineers9. businessconttinuestoneed

10. businessneedspeopleoperating11. businessneedspeopleoperatingatthisskill12. businessrequirementtohave13. businesswillcontinuetorely14. clearreplacement15. clusterserviceneedsseniordevelopers16. commercespacehasastrongneed17. cost/psthasaneed18. counteroffer19. criticalneedresource20. dmxandiceneed21. donotcurrentlyhaveanyseniorsdets22. doesnotcurrentlyhaveanyseniorsdets23. dspteamhastheneed24. dspteamhastheneedforseniorsoftwareengineers25. ensureretention26. escalationcenter27. facilitateexpansioninnewavenues28. fillvacancies29. fillvacancy30. growthhasincreasedtheneed31. hasaclearneed32. hasaneed33. hastheneedforseniorsoftwareengineers34. hasn'tbeentakencareofinhertimeatmicrosoft35. hasn'tbeentakencareofinhistimeatmicrosoft36. icoulduse37. ineedasenior38. importantwemakethesepartnerteamssuccessful39. increasedtheneedfor40. irreplaceble41. iscriticalthatweretain42. isdefinitelyinneedof43. itisimperativethatthereisaleader44. itisimperativethatthereisaleaderthat45. it'sveryimportanttokeepherintheteam46. jobopening47. keytokeepingbingadsprivatelabinfastructurerunning48. lotofupcomingwork49. mostwantedskill

StringsfromBN_Dictionary.RthatdonotmatchanyPromotionJustificationCommentsin"MRTSample1000.csv"(produced1/31/2018)

101

315



50. necessaryforthebusinesstohave51. needal6152. needal6253. needal6354. needal6155. needal6256. needal6357. needamoreseniorperson58. needamoreseniorpersontotakeonabiggerrole59. needapm60. needaseniorleveldatascientist61. needforbothbroadanddeepunderstanding62. needtohaveastrongsenioric63. needtoretaingoodtalent64. neededforlong-termsuccess65. needsal6266. needsal6367. needsal6268. needsal6369. needsamoreseniorperson70. needsaseniorleveldatascientist71. needsmoreseniorpms72. needsprincipalengineers73. needsswesatlevel6274. newbranch75. ongoingeffort76. ongoingneedforsuccess77. ongoingneedforthesuccess78. ourdemandforengineers79. ourservicestocontinuetogrow80. outsideopportunities81. promotionaimstoretain82. promotionhelpsusretain83. promotionwillhelpretain84. reactiveretention85. requireaseniorlevelpm86. requiredforcontinuedprogress87. requiresahighlevel88. requiresaseniorlevelpm89. requiresdesigners90. requirespeopleattheseniorlevel91. requiringhighlevelof92. retainastellar93. retainingcorequalitytalent94. retentionofgreatemployees95. retentionrisk96. roleneedperspective97. skippedforpromo98. teamissmallandrequiresahighlevel99. teamsizeissmall100. telemetryteamneeds

102

316



101. thebusinessisdemandingevermoreofthisskill102. thelevelofcomplexityinthatspacerequiresaseniorlevelpm103. theneedtohaveastrongsenioricbandgrowswithit104. theneedswiththesmbteam105. thenewplatformneeds106. theorgneedsseniorindividuals107. theproductsteamneeds108. theteamalsorequires109. theteamhasaneed110. thetextteamneeds111. thereisincreasedscopeforastrong112. thereisincreasedscopeforastrongl62113. thereistheneed114. thispromotionjustificationwillallowteamtocontinue115. thispromotionsupports116. thisrolerequires117. tigerteamneeds118. tnrhasgrowninsizeandcomplexity119. toretaingreattalent120. under-pay121. weareingreatneedofleaders122. wedon'twanttolosethem123. wehavearealneed124. weneedal62125. weneedalevel62126. weneedaseniorleveldatascientist127. weneeddesigners128. weneedherleadership129. weneedhigherlevel130. weneedleaders131. weneedseniorpeople132. weneedskills133. weneedstrongleadership134. weneedstronger135. weneedswesat63136. weneedtechnicalengineers137. weneedthesenior138. weneedtogrow139. wewanttomakesuretoretain140. wewillcontinuetorely141. wewouldliketoretain142. we'reinneed143. whowewanttoretain144. willhelpusretain145. willprovidecapacityrelief146. willprovidecapacityrelief147. withanincreasedsalarylevelwecancompensate148. wouldfreeupsomeofthefolks

Note:Thestring"willprovidecapacityrelief"appearstwiceintheBN_Dictionary.Rfile.

103

317


APPENDIX3

104

318


Documents Relied Upon

The following are documents relied upon in addition to the documents listed in my October 27, 2017 report. Expert Reports Expert Report of Ali Saad, Ph.D., in the matter of Moussouris et al., v. Microsoft Corporation, January 5, 2018. Errata to the Expert Report of Ali Saad, Ph.D., in the matter of Moussouris et al., v. Microsoft Corporation, February 5, 2018. Report of Henry S. Farber, in the Matter of Katherina Moussouris, Holly Muenchow, Dana Piermarini et al., v. Microsoft Corporation, October 27, 2017. Corrected Expert Report of Henry Farber, Ph.D., in the Matter of Moussouris et al., v. Microsoft Corporation, December 5, 2017. Documents Produced by Experts “MRT Sample 1000.csv” BN_Dictionary Case Documents Deposition of Ali Saad, Ph.D., January 30, 2018. MSFT_MOUSSOURIS_00010979.xlsx MSFT_MOUSSOURIS_00308243 Academic Literature ABA Section of Antitrust Law, Econometrics (2005) at p. 15. Duan, Naihua, “Smearing Estimate: A Nonparametric Retransformation Model.” Journal of the American Statistical Association., Volume 78, No. 383 (September 1983), pp. 605-610. Haan, C., E. Reardon, and A. Saad, “Employment Discrimination Litigation,” chapter in Litigation Services Handbook, ed., by Roman Weil, et al., 2012. Piette, Michael and White, Paul, “Approaches for Dealing with Small Sample Sizes in Employment Discrimination Litigation,” Journal of Forensic Economists, 12(1), 1999, pp. 43-56 Publicly Available Data Francis Galton Height Data https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/T0HSJ1

105

319


Data CELA_Moussouris_Population_ASG AEO_Highly Confidential.xlsx CELA_Moussouris_Population_C+E AEO_Highly Confidential.xlsx CELA_Moussouris_Population_COO AEO_Highly Confidential.xlsx CELA_Moussouris_Population_S&M AEO_Highly Confidential.xlsx CELA_Moussouris_Population_T&R AEO_Highly Confidential.xlsx CELA_Moussouris_Population_WDG AEO_Highly Confidential.xlsx

106

320


EXPERT REBUTTAL REPORT IN THE MATTER OF Moussouris v....

Documents

Transcript of EXPERT REBUTTAL REPORT IN THE MATTER OF Moussouris v....