Abstract - arXiv · 2019. 3. 12. · 17th century with Ren e Descartes proposal of an \universal...

$: Abstract - arXiv · 2019. 3. 12. · 17th century with Ren e Descartes proposal of an \universal language" [11], machine transla-tion has only existed as a technological eld since$
Assessing Gender Bias in Machine Translation ndash A CaseStudy with Google Translate

Marcelo Prates morpratesinfufrgsbr

Pedro Avelar pedroavelarinfufrgsbr

Luis C Lamb lambinfufrgsbr

Federal University of Rio Grande do Sul

Abstract

Recently there has been a growing concern in academia industrial research labs andthe mainstream commercial media about the phenomenon dubbed as machine bias wheretrained statistical models ndash unbeknownst to their creators ndash grow to reflect controversialsocietal asymmetries such as gender or racial bias A significant number of Artificial Intel-ligence tools have recently been suggested to be harmfully biased towards some minoritywith reports of racist criminal behavior predictors Applersquos Iphone X failing to differentiatebetween two distinct Asian people and the now infamous case of Google photosrsquo mistak-enly classifying black people as gorillas Although a systematic study of such biases canbe difficult we believe that automated translation tools can be exploited through genderneutral languages to yield a window into the phenomenon of gender bias in AI

In this paper we start with a comprehensive list of job positions from the US Bureauof Labor Statistics (BLS) and used it in order to build sentences in constructions likeldquoHeShe is an Engineerrdquo (where ldquoEngineerrdquo is replaced by the job position of interest)in 12 different gender neutral languages such as Hungarian Chinese Yoruba and severalothers We translate these sentences into English using the Google Translate API andcollect statistics about the frequency of female male and gender-neutral pronouns in thetranslated output We then show that Google Translate exhibits a strong tendency towardsmale defaults in particular for fields typically associated to unbalanced gender distributionor stereotypes such as STEM (Science Technology Engineering and Mathematics) jobsWe ran these statistics against BLSrsquo data for the frequency of female participation ineach job position in which we show that Google Translate fails to reproduce a real-worlddistribution of female workers In summary we provide experimental evidence that even ifone does not expect in principle a 5050 pronominal gender distribution Google Translateyields male defaults much more frequently than what would be expected from demographicdata alone

We believe that our study can shed further light on the phenomenon of machine biasand are hopeful that it will ignite a debate about the need to augment current statisticaltranslation tools with debiasing techniques ndash which can already be found in the scientificliterature

1 Introduction

Although the idea of automated translation can in principle be traced back to as long as the17th century with Rene Descartes proposal of an ldquouniversal languagerdquo [11] machine transla-tion has only existed as a technological field since the 1950s with a pioneering memorandumby Warren Weaver [27 39] discussing the possibility of employing digital computers to per-form automated translation The now famous Georgetown-IBM experiment followed not

1

arX

iv1

809

0220

8v4

[cs

CY

] 1

1 M

ar 2

019

long after providing the first experimental demonstration of the prospects of automatingtranslation by the means of successfully converting more than sixty Russian sentences intoEnglish [17] Early systems improved upon the results of the Georgetown-IBM experimentby exploiting Noam Chomskyrsquos theory of generative linguistics and the field experienceda sense of optimism about the prospects of fully automating natural language translationAs is customary with artificial intelligence the initial optimistic stage was followed by anextended period of strong disillusionment with the field of which the catalyst was the in-fluential 1966 ALPAC (Automatic Language Processing Advisory Committee) report( [19]Such research was then disfavoured in the United States making a re-entrance in the 1970sbefore the 1980s surge in statistical methods for machine translation [25 26] Statistical andexample-based machine translation have been on the rise ever since [2 8 14] with highlysuccessful applications such as Google Translate (recently ported to a neural translationtechnology [20]) amounting to over 200 million users daily

In spite of the recent commercial success of automated translation tools (or perhapsstemming directly from it) machine translation has amounted a significant deal of criticismNoted philosopher and founding father of generative linguistics Noam Chomsky has arguedthat the achievements of machine translation while successes in a particular sense are notsuccesses in the sense that science has ever been interested in they merely provide effectiveways according to Chomsky of approximating unanalyzed data [9 30] Chomsky arguesthat the faith of the MT community in statistical methods is absurd by analogy with astandard scientific field such as physics [9]

I mean actually you could do physics this way instead of studying things likeballs rolling down frictionless planes which canrsquot happen in nature if you tooka ton of video tapes of whatrsquos happening outside my office window letrsquos sayyou know leaves flying and various things and you did an extensive analysis ofthem you would get some kind of prediction of whatrsquos likely to happen nextcertainly way better than anybody in the physics department could do Wellthatrsquos a notion of success which is I think novel I donrsquot know of anything likeit in the history of science

Leading AI researcher and Googlersquos Director of Research Peter Norvig responds to thesearguments by suggesting that even standard physical theories such as the Newtonian modelof gravitation are in a sense trained [30]

As another example consider the Newtonian model of gravitational attrac-tion which says that the force between two objects of mass m1 and m2 a distancer apart is given by

F = Gm1m2r2

where G is the universal gravitational constant This is a trained model be-cause the gravitational constant G is determined by statistical inference overthe results of a series of experiments that contain stochastic experimental errorIt is also a deterministic (non-probabilistic) model because it states an exactfunctional relationship I believe that Chomsky has no objection to this kind ofstatistical model Rather he seems to reserve his criticism for statistical modelslike Shannonrsquos that have quadrillions of parameters not just one or two

2

Chomsky and Norvigrsquos debate [30] is a microcosm of the two leading standpoints aboutthe future of science in the face of increasingly sophisticated statistical models Are weas Chomsky seems to argue jeopardizing science by relying on statistical tools to performpredictions instead of perfecting traditional science models or are these tools as Norvigargues components of the scientific standard since its conception Currently there are nosatisfactory resolutions to this conundrum but perhaps statistical models pose an evengreater and more urgent threat to our society

On a 2014 article Londa Schiebinger suggested that scientific research fails to take gen-der issues into account arguing that the phenomenon of male defaults on new technologiessuch as Google Translate provides a window into this asymmetry [35] Since then recentworrisome results in machine learning have somewhat supported Schiebingerrsquos view Notonly Google photosrsquo statistical image labeling algorithm has been found to classify dark-skinned people as gorillas [15] and purportedly intelligent programs have been suggestedto be negatively biased against black prisoners when predicting criminal behavior [1] butthe machine learning revolution has also indirectly revived heated debates about the con-troversial field of physiognomy with proposals of AI systems capable of identifying thesexual orientation of an individual through its facial characteristics [38] Similar concernsare growing at an unprecedented rate in the media with reports of Applersquos Iphone X faceunlock feature failing to differentiate between two different Asian people [32] and automaticsoap dispensers which reportedly do not recognize black hands [28] Machine bias the phe-nomenon by which trained statistical models unbeknownst to their creators grow to reflectcontroversial societal asymmetries is growing into a pressing concern for the modern timesinvites us to ask ourselves whether there are limits to our dependence on these techniquesndash and more importantly whether some of these limits have already been traversed In thewave of algorithmic bias some have argued for the creation of some kind of agency in thelikes of the Food and Drug Administration with the sole purpose of regulating algorithmicdiscrimination [23]

With this in mind we propose a quantitative analysis of the phenomenon of gender biasin machine translation We illustrate how this can be done by simply exploiting GoogleTranslate to map sentences from a gender neutral language into English As Figure 1exemplifies this approach produces results consistent with the hypothesis that sentencesabout stereotypical gender roles are translated accordingly with high probability nurseand baker are translated with female pronouns while engineer and CEO are translatedwith male ones

2 Motivation

As of 2018 Google Translate is one of the largest publicly available machine translation toolsin existence amounting 200 million users daily[36] Initially relying on United Nations andEuropean Parliament transcripts to gather data since 2014 Google Translate has inputedcontent from its users through the Translate Community initiative[22] Recently howeverthere has been a growing concern about gender asymmetries in the translation mechanismwith some heralding it as ldquosexistrdquo [31] This concern has to at least some extent a scientificbackup A recent study has shown that word embeddings are particularly prone to yieldinggender stereotypes[5] Fortunately the researchers propose a relatively simple debiasing

3

Figure 1 Translating sentences from a gender neutral language such as Hungarian to En-glish provides a glimpse into the phenomenon of gender bias in machine trans-lation This screenshot from Google Translate shows how occupations from tra-ditionally male-dominated fields [40] such as scholar engineer and CEO are in-terpreted as male while occupations such as nurse baker and wedding organizerare interpreted as female

algorithm with promising results they were able to cut the proportion of stereotypicalanalogies from 19 to 6 without any significant compromise in the performance of theword embedding technique They are not alone there is a growing effort to systematicallydiscover and resolve issues of algorithmic bias in black-box algorithms[18] The success ofthese results suggest that a similar technique could be used to remove gender bias fromGoogle Translate outputs should it exist This paper intends to investigate whether itdoes We are optimistic that our research endeavors can be used to argue that there is apositive payoff in redesigning modern statistical translation tools

3 Assumptions and Preliminaries

In this paper we assume that a statistical translation tool should reflect at most the inequal-ity existent in society ndash it is only logical that a translation tool will poll from examples thatsociety produced and as such will inevitably retain some of that bias It has been arguedthat onersquos language affects onersquos knowledge and cognition about the world [21] and thisleads to the discussion that languages that distinguish between female and male gendersgrammatically may enforce a bias in the personrsquos perception of the world with some studies

4

corroborating this as shown in [6] as well some relating this with sexism [37] and genderinequalities [34]

With this in mind one can argue that a move towards gender neutrality in language andcommunication should be striven as a means to promote improved gender equality Thusin languages where gender neutrality can be achieved ndash such as English ndash it would be a validaim to create translation tools that keep the gender-neutrality of texts translated into sucha language instead of defaulting to male or female variants

We will thus assume throughout this paper that although the distribution of translatedgender pronouns may deviate from 5050 it should not deviate to the extent of misrep-resenting the demographics of job positions That is to say we shall assume that GoogleTranslate incorporates a negative gender bias if the frequency of male defaults overesti-mates the (possibly unequal) distribution of male employees per female employee in a givenoccupation

4 Materials and Methods

We shall assume and then show that the phenomenon of gender bias in machine translationcan be assessed by mapping sentences constructed in gender neutral languages to Englishby the means of an automated translation tool Specifically we can translate sentencessuch as the Hungarian ldquoo egy apolonordquo where ldquoapolonordquo translates to ldquonurserdquo and ldquoordquo is agender-neutral pronoun meaning either he she or it to English yielding in this example theresult ldquoshersquos a nurserdquo on Google Translate As Figure 1 clearly shows the same templateyields a male pronoun when ldquonurserdquo is replaced by ldquoengineerrdquo The same basic template canbe ported to all other gender neutral languages as depicted in Table 3 Given the successof Google Translate which amounts to 200 million users daily we have chosen to exploitits API to obtain the desired thermometer of gender bias Also in order to solidify ourresults we have decided to work with a fair amount of gender neutral languages forming alist of these with help from the World Atlas of Language Structures (WALS) [13] and othersources Table 1 compiles all languages we chose to use with additional columns informingwhether they (1) exhibit a gender markers in the sentence and (2) are supported by GoogleTranslate However we stumbled on some difficulties which led to some of those langaugesbeing removed which will be explained in

There is a prohibitively large class of nouns and adjectives that could in principle besubstituted into our templates To simplify our dataset we have decided to focus ourwork on job positions ndash which we believe are an interesting window into the nature ofgender bias ndash and were able to obtain a comprehensive list of professional occupationsfrom the Bureau of Labor Statisticsrsquo detailed occupations table [7] from the United StatesDepartment of Labor The values inside however had to be expanded since each linecontained multiple occupations and sometimes very specific ones Fortunately this tablealso provided a percentage of women participation in the jobs shown for those that hadmore than 50 thousand workers We filtered some of these because they were too generic (ldquoComputer occupations all otherrdquo and others) or because they had gender specific wordsfor the profession (ldquohosthostessrdquo ldquowaiterwaitressrdquo) We then separated the curated jobsinto broader categories (Artistic Corporate Theatre etc) as shown in Table 2 FinallyTable 4 shows thirty examples of randomly selected occupations from our dataset For

5

the occupations that had less than 50 thousand workers and thus no data about theparticipation of women we assumed that its women participation was that of its uppercategory Finally as complementary evidence we have decided to include a small subset of21 adjectives in our study All adjectives were obtained from the top one thousand mostfrequent words in this category as featured in the Corpus of Contemporary American En-glish (COCA) httpscorpusbyueducoca but it was necessary to manually curate thembecause a substantial fraction of these adjectives cannot be applied to human subjectsAlso because the sentiment associated with each adjective is not as easily accessible as forexample the occupation category of each job position we performed a manual selection ofa subset of such words which we believe to be meaningful to this study These words arepresented in Table 5 We made all code and data used to generate and compile the resultspresented in the following sections publicly available in the following Github repositoryhttpsgithubcommarcelopratesGender-Bias Note however that because the GoogleTranslate algorithm can change unfortunately we cannot guarantee full reproducibility ofour results All experiments reported here were conducted on April 2018

Language Family Language

Phraseshavemalefemalemarkers

Tested

Austronesian Malay 5 X

UralicEstonian 5 XFinnish 5 XHungarian 5 X

Indo-European

Armenian 5 XBengali O XEnglish X 5

Persian 5 XNepali O X

Japonic Japanese 5 XKoreanic Korean X 5

Turkic Turkish 5 X

Niger-CongoYoruba 5 XSwahili 5 X

Isolate Basque 5 XSino-Tibetan Chinese O X

Table 1 Gender neutral languages supported by Google Translate Languages are groupedaccording to language families and classified according to whether they enforceany kind of mandatory gender (malefemale) demarcation on simple phrases (Xyes 5 never O some) For the purposes of this work we have decided to workonly with languages lacking such demarcation Languages colored in red have beenomitted for other reasons See Section

41 for further explanation

6

Category Group OccupationsFemale Partici-pation

Education trainingand library

Education 22 730

Business and financialoperations

Corporate 46 540

Office and administra-tive support

Service 87 722

Healthcare support Healthcare 16 871

Management Corporate 46 398

Installation mainte-nance and repair

Service 91 40

Healthcare practition-ers and technical

Healthcare 43 750

Community and socialservice

Service 14 661

Sales and related Corporate 28 491

Production Production 264 289

Architecture and engi-neering

STEM 29 162

Life physical and so-cial science

STEM 34 474

Transportation andmaterial moving

Service 70 173

Arts design entertain-ment sports and me-dia

Arts Enter-tainment

37 469

Legal Legal 7 528

Protective Service Service 28 223

Food preparation andserving related

Service 17 538

Farming fishing andforestry

Farming Fishing Forestry

13 234

Computer and mathe-matical

STEM 16 255

Personal care and ser-vice

Service 33 761

Construction and ex-traction

Construction Extraction

68 30

Building and groundscleaning and mainte-nance

Service 10 407

Total - 1019 413

Table 2 Selected occupations obtained from the US Bureau of Labor Statisticshttpswwwblsgovcpscpsaat11htm grouped by category We obtained a to-tal of 1019 occupations from 22 distinct categories We have further grouped theminto broader groups (or super-categories) to ease analysis and visualization

7

Language Occupation sentence template Adjective sentence template

Malay dia adalah 〈occupation〉 dia 〈adjective〉Estonian ta on 〈occupation〉 ta on 〈adjective〉Finnish han on 〈occupation〉 han on 〈adjective〉

Hungarian o egy 〈occupation〉 o 〈adjective〉Armenian na 〈occupation〉 e na 〈adjective〉 e

Bengali

E ekajana 〈occupation〉 E 〈adjective〉Yini ekajana 〈occupation〉 Yini 〈adjective〉O ekajana 〈occupation〉 O 〈adjective〉

Uni ekajana 〈occupation〉 Uni 〈adjective〉Se ekajana 〈occupation〉 Se 〈adjective〉

Tini ekajana 〈occupation〉 Tini 〈adjective〉Japanese あの人は〈occupation〉ですあの人は〈adjective〉ですTurkish o bir 〈occupation〉 o 〈adjective〉Yoruba o je 〈occupation〉 o je 〈adjective〉Basque 〈occupation〉 bat da 〈adjective〉 daSwahili yeye ni 〈occupation〉 yeye ni 〈adjective〉Chinese ta shi 〈occupation〉 ta hen 〈adjective〉

Table 3 Templates used to infer gender biases in the translation of job occupations andadjectives to the English language

Insurance sales agent Editor RancherTicket taker Pile-driver operator Tool maker

Jeweler Judicial law clerk Auditing clerkPhysician Embalmer Door-to-door salesperson

Packer Bookkeeping clerk Community health workerSales worker Floor finisher Social science technician

Probation officer Paper goods machine setter Heating installerAnimal breeder Instructor Teacher assistant

Statistical assistant Shipping clerk TrapperPharmacy aide Sewing machine operator Service unit operator

Table 4 A randomly selected example subset of thirty occupations obtained from ourdataset with a total of 1019 different occupations

8

Happy Sad RightWrong Afraid BraveSmart Dumb ProudStrong Polite Cruel

Desirable Loving SympatheticModest Successful Guilty

Innocent Mature Shy

Table 5 Curated list of 21 adjectives obtained from the top one thousand most frequentwords in this category in the Corpus of Contemporary American English (COCA)

httpscorpusbyueducoca

41 Rationale for language exceptions

While it is possible to construct gender neutral sentences in two of the languages omitted inour experiments (namely Korean and Nepali) we have chosen to omit them for the followingreasons

1 We faced technical difficulties to form templates and automatically translate sentenceswith the right-to-left top-to-bottom nature of the script and as such we have decidednot to include it in our experiments

2 Due to Nepali having a rather complex grammar with possible malefemale genderdemarcations on the phrases and due to none of the authors being fluent or able toreach someone fluent in the language we were not confident enough in our abilityto produce the required templates Bengali was almost discarded under the samerationale but we have decided to keep it because of our sentence template for Bengalihas a simple grammatical structure which does not require any kind of inflection

3 One can construct gender neutral phrases in Korean by omitting the gender pronounin fact this is the default procedure However the expressiveness of this omissiondepends on the context of the sentence being clear which is not possible in the waywe frame phrases

5 Distribution of translated gender pronouns per occupation category

A sensible way to group translation data is to coalesce occupations in the same categoryand collect statistics among languages about how prominent male defaults are in each fieldWhat we have found is that Google Translate does indeed translate sentences with male pro-nouns with greater probability than it does either with female or gender-neutral pronounsin general Furthermore this bias is seemingly aggravated for fields suggested to be troubledby male stereotypes such as life and physical sciences architecture engineering computerscience and mathematics [29] Table 6 summarizes these data and Table 7 summarizes iteven further by coalescing occupation categories into broader groups to ease interpretationFor instance STEM (Science Technology Engineering and Mathematics) fields are groupedinto a single category which helps us compare the large asymmetry between gender pro-

9

nouns in these fields (72 of male defaults) to that of more evenly distributed fields suchas healthcare (50)

Category Female () Male () Neutral ()

Office and administrative support 11015 58812 16954Architecture and engineering 2299 72701 1092Farming fishing and forestry 12179 62179 14744

Management 11232 66667 12681Community and social service 20238 625 10119

Healthcare support 250 4375 17188Sales and related 8929 62202 16964

Installation maintenance and repair 522 58333 17125Transportation and material moving 881 62976 175

Legal 11905 72619 10714Business and financial operations 7065 67935 1558Life physical and social science 5882 73284 10049

Arts design entertainment sports and media 1036 67342 11486Education training and library 23485 5303 9091

Building and grounds cleaning and maintenance 125 68333 11667Personal care and service 18939 49747 18434

Healthcare practitioners and technical 22674 51744 15116Production 14331 51199 18245

Computer and mathematical 4167 66146 14062Construction and extraction 8578 61887 17525

Protective service 8631 65179 125Food preparation and serving related 21078 58333 17647

Total 1176 5893 15939

Table 6 Percentage of female male and neutral gender pronouns obtained for each BLSoccupation category averaged over all occupations in said category and testedlanguages detailed in Table

1 Note that rows do not in general add up to 100 as there is a fair amount of translatedsentences for which we cannot obtain a gender pronoun

10


Service 105 59548 16476STEM 4219 71624 11181

Farming Fishing Forestry 12179 62179 14744Corporate 9167 66042 14861Healthcare 23305 49576 15537

Legal 11905 72619 10714Arts Entertainment 1036 67342 11486

Education 23485 5303 9091Production 14331 51199 18245

Construction Extraction 8578 61887 17525

Total 1176 5893 15939

Table 7 Percentage of female male and neutral gender pronouns obtained for each of themerged occupation category averaged over all occupations in said category andtested languages detailed in Table


Plotting histograms for the number of gender pronouns per occupation category shedsfurther light on how female male and gender-neutral pronouns are differently distributedThe histogram in Figure 2 suggests that the number of female pronouns is inversely dis-tributed ndash which is mirrored in the data for gender-neutral pronouns in Figure 4 ndash whilethe same data for male pronouns (shown in Figure 3) suggests a skew normal distributionFurthermore we can see both on Figures 2 and 3 how STEM fields (labeled in beige exhibitpredominantly male defaults ndash amounting predominantly near X = 0 in the female his-togram although much to the right in the male histogram

These values contrast with BLSrsquo report of gender participation which will be discussedin more detail in Section 8

11

Translated Female Pronouns (grouped among languages)

0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons

Figure 2 The data for the number of translated female pronouns per merged occupationcategory totaled among languages suggests and inverse distribution STEM fieldsare nearly exclusively concentrated at X = 0 while more evenly distributed infields such as production and healthcare (See Table

7) extends to higher values

Translated Male Pronouns (grouped among languages)

5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons

Figure 3 In contrast to Figure2 male pronouns are seemingly skew normally distributed with a peak at X = 6 One can

see how STEM fields concentrate mainly to the right (X ge 6)

12

Translated Neutral Pronouns (grouped among languages)

0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons

Figure 4 The scarcity of gender-neutral pronouns is manifest in their histogram Onceagain STEM fields are predominantly concentrated at X = 0

We can also visualize male female and gender neutral histograms side by side inwhich context is useful to compare the dissimilar distributions of translated STEM andHealthcare occupations (Figures 5 and 6 respectively) The number of translated femalepronouns among languages is not normally distributed for any of the individual categoriesin Table 2 but Healthcare is in many ways the most balanced category which can be seenin comparison with STEM ndash in which male defaults are second to most prominent

13

Translated Pronouns (grouped among languages)

0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons

Figure 5 Histograms for the distribution of the number of translated female male andgender neutral pronouns totaled among languages are plotted side by side for joboccupations in the STEM (Science Technology Engineering and Mathematics)field in which male defaults are the second-to-most prominent (after Legal)


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons

Figure 6 Histograms for the distribution of the number of translated female male andgender neutral pronouns totaled among languages are plotted side by side for joboccupations in the Healthcare field in which male defaults are least prominent

14

The bar plots in Figure 7 help us visualize how much of the distribution of each occu-pation category is composed of female male and gender-neutral pronouns In this contextSTEM fields which show a predominance of male defaults are contrasted with Healthcareand educations which show a larger proportion of female pronouns

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100

Figure 7 Bar plots show how much of the distribution of translated gender pronouns foreach occupation category (grouped as in Table 7) is composed of female male andneutral terms Legal and STEM fields exhibit a predominance of male defaultsand contrast with Healthcare and Education with a larger proportion of femaleand neutral pronouns Note that in general the bars do not add up to 100 asthere is a fair amount of translated sentences for which we cannot obtain a genderpronoun Categories are sorted with respect to the proportions of male femaleand neutral translated pronouns respectively

Although computing our statistics over the set of all languages has practical valuethis may erase subtleties characteristic to each individual idiom In this context it is alsoimportant to visualize how each language translates job occupations in each category Theheatmaps in Figures 8 9 and 10 show the translation probabilities into female male andneutral pronouns respectively for each pair of language and category (blue is 0 and redis 100) Both axes are sorted in these Figures which helps us visualize both languagesand categories in an spectrum of increasing malefemaleneutral translation tendencies In

15

agreement with suggested stereotypes [29] STEM fields are second only to Legal ones inthe prominence of male defaults These two are followed by Arts amp Entertainment andCorporate in this order while Healthcare Production and Education lie on the oppositeend of the spectrum

Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e

Figure 8 Heatmap for the translation probability into female pronouns for each pair oflanguage and occupation category Probabilities range from 0 (blue) to 100(red) and both axes are sorted in such a way that higher probabilities concentrateon the bottom right corner

16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e

Figure 9 Heatmap for the translation probability into male pronouns for each pair of lan-guage and occupation category Probabilities range from 0 (blue) to 100 (red)and both axes are sorted in such a way that higher probabilities concentrate onthe bottom right corner

17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e

Figure 10 Heatmap for the translation probability into gender neutral pronouns for eachpair of language and occupation category Probabilities range from 0 (blue)to 100 (red) and both axes are sorted in such a way that higher probabilitiesconcentrate on the bottom right corner

Our analysis is not truly complete without tests for statistical significant differencesin the translation tendencies among female male and gender neutral pronouns We wantto know for which languages and categories does Google Translate translate sentences withsignificantly more male than female or male than neutral or neutral than female pronounsWe ran one-sided t-tests to assess this question for each pair of language and category andalso totaled among either languages or categories The corresponding p-values are presentedin Tables 8 9 10 respectively Language-Category pairs for which the null hypothesis wasnot rejected for a confidence level of α = 005 are highlighted in blue It is importantto note that when the null hypothesis is accepted we cannot discard the possibility ofthe complementary null hypothesis being rejected For example neither male nor femalepronouns are significantly more common for Healthcare positions in the Estonian languagebut female pronouns are significantly more common for the same category in Finnish andHungarian Because of this Language-Category pairs for which the complementary nullhypothesis is rejected are painted in a darker shade of blue (see Table 8 for the threeexamples cited above

Although there is a noticeable level of variation among languages and categories thenull hypothesis that male pronouns are not significantly more frequent than female ones was

18

consistently rejected for all languages and all categories examined The same is true for thenull hypothesis that male pronouns are not significantly more frequent than gender neutralpronouns with the one exception of the Basque language (which exhibits a rather strongtendency towards neutral pronouns) The null hypothesis that neutral pronouns are notsignificantly more frequent than female ones is accepted with much more frequency namelyfor the languages Malay Estonian Finnish Hungarian Armenian and for the categoriesFarming amp Fishing amp Forestry Healthcare Legal Arts amp Entertainment Education Inall three cases the null hypothesis corresponding to the aggregate for all languages andcategories is rejected We can learn from this in summary that Google Translate translatesmale pronouns more frequently than both female and gender neutral ones either in generalfor Language-Category pairs or consistently among languages and among categories (withthe notable exception of the Basque idiom)

Mal Est Fin Hun Arm Ben Jap Tur Yor Bas Swa Chi TotalService lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt αSTEM lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt αFarmingFishingForestry

lt α lt α 603 786 lt α lt α lt α lt α lt α lowast lt α lt α lt α

Corporate lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt αHealthcare lt α 938 10 999 lt α lt α lt α lt α lt α lt α lt α lt α lt αLegal lt α 368 632 368 lt α lt α lt α lt α lt α 086 lt α lt α lt αArts Enter-tainment

lt α lt α lt α lt α lt α lt α lt α lt α lt α 08 lt α lt α lt α

Education lt α 808 333 263 588 lt α lt α 417 lt α 052 lt α lt α lt αProduction lt α lt α lt α 5 lt α lt α lt α lt α lt α 159 lt α lt α lt αConstructionExtraction


Total lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α lt α

Table 8 Computed p-values relative to the null hypothesis that the number of translatedmale pronouns is not significantly greater than that of female pronouns organizedfor each language and each occupation category Cells corresponding to the ac-ceptance of the null hypothesis are marked in blue and within those cells thosecorresponding to cases in which the complementary null hypothesis (that the num-ber of female pronouns is not significantly greater than that of male pronouns) wasrejected are marked with a darker shade of the same color A significance level ofα = 05 was adopted Asterisks indicate cases in which all pronouns are translatedwith gender neutral pronouns

19

Mal Est Fin Hun Arm Ben Jap Tur Yor Bas Swa Chi TotalService lt α lt α lt α lt α lt α lt α lt α lt α lt α 10 lt α lt α lt αSTEM lt α lt α lt α lt α lt α lt α lt α lt α lt α 984 lt α 07 lt αFarmingFishingForestry

lt α lt α lt α lt α lt α 135 lt α lt α 068 10 lt α lt α lt α

Corporate lt α lt α lt α lt α lt α lt α lt α lt α lt α 10 lt α lt α lt αHealthcare lt α lt α lt α lt α lt α lt α lt α lt α 39 10 lt α 088 lt αLegal lt α lt α lt α lt α lt α lt α 145 lt α lt α 771 lt α lt α lt αArts Enter-tainment

lt α lt α lt α lt α lt α 07 lt α lt α lt α 10 lt α lt α lt α

Education lt α lt α lt α lt α lt α lt α 093 lt α lt α 5 lt α 068 lt αProduction lt α lt α lt α lt α lt α lt α lt α 412 10 10 lt α lt α lt αConstructionExtraction

lt α lt α lt α lt α lt α lt α lt α lt α 92 10 lt α lt α lt α

Total lt α lt α lt α lt α lt α lt α lt α lt α lt α 10 lt α lt α lt α

Table 9 Computed p-values relative to the null hypothesis that the number of translatedmale pronouns is not significantly greater than that of gender neutral pronounsorganized for each language and each occupation category Cells corresponding tothe acceptance of the null hypothesis are marked in blue and within those cellsthose corresponding to cases in which the complementary null hypothesis (thatthe number of gender neutral pronouns is not significantly greater than that ofmale pronouns) was rejected are marked with a darker shade of the same colorA significance level of α = 05 was adopted Asterisks indicate cases in which allpronouns are translated with gender neutral pronouns

20

Mal Est Fin Hun Arm Ben Jap Tur Yor Bas Swa Chi TotalService 10 10 10 10 981 lt α lt α lt α lt α lt α 10 lt α lt αSTEM 84 978 998 993 84 lt α lt α 079 lt α lt α 84 lt α lt αFarmingFishingForestry

lowast lowast 999 10 lowast 167 169 292 lt α lt α lowast 083 147

Corporate lowast 10 10 10 996 lt α lt α lt α lt α lt α 977 lt α lt αHealthcare 10 10 10 10 10 086 lt α 87 lt α lt α 10 lt α 977Legal lowast 961 985 961 lowast lt α 086 lowast 178 lt α lowast lowast 072Arts Enter-tainment

92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097

Education lowast 10 999 999 10 058 lt α 10 164 052 995 052 992Production 996 10 10 10 10 113 lt α lt α lt α lt α 10 lt α lt αConstructionExtraction

84 996 10 10 lowast lt α lt α lt α lt α lt α 10 lt α lt α

Total 10 10 10 10 10 lt α lt α lt α lt α lt α 10 lt α lt α

Table 10 Computed p-values relative to the null hypothesis that the number of translatedgender neutral pronouns is not significantly greater than that of female pronounsorganized for each language and each occupation category Cells corresponding tothe acceptance of the null hypothesis are marked in blue and within those cellsthose corresponding to cases in which the complementary null hypothesis (thatthe number of female pronouns is not significantly greater than that of genderneutral pronouns) was rejected are marked with a darker shade of the same colorA significance level of α = 05 was adopted Asterisks indicate cases in which allpronouns are translated with gender neutral pronouns

6 Distribution of translated gender pronouns per language

We have taken the care of experimenting with a fair amount of different gender neutrallanguages Because of that another sensible way of coalescing our data is by languagegroups as shown in Table 11 This can help us visualize the effect of different culturesin the genesis ndash or lack thereof ndash of gender bias Nevertheless the barplots in Figure 11are perhaps most useful to identifying the difficulty of extracting a gender pronoun whentranslating from certain languages Basque is a good example of this difficulty althoughthe quality of Bengali Yoruba Chinese and Turkish translations are also compromised

21

Language Female () Male () Neutral ()

Malay 3827 88420 0000Estonian 17370 72228 0491Finnish 34446 56624 0000

Hungarian 34347 58292 0000Armenian 10010 82041 0687Bengali 16765 37782 2563

Japanese 0000 66928 24436Turkish 2748 62807 18744Yoruba 1178 48184 38371Basque 0393 5496 58587Swahili 14033 76644 0000Chinese 5986 51717 24338

Table 11 Percentage of female male and neutral gender pronouns obtained for each lan-guage averaged over all occupations detailed in Table

1 Note that rows do not in general add up to 100 as there is a fair amount of translated sentencesfor which we cannot obtain a gender pronoun

22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100

Figure 11 The distribution of pronominal genders per language also suggests a tendencytowards male defaults with female pronouns reaching as low as 0196 and1865 for Japanese and Chinese respectively Once again not all bars add upto 100 as there is a fair amount of translated sentences for which we cannotobtain a gender pronoun particularly in Basque Among all tested languagesBasque was the only one to yield more gender neutral than male pronounswith Bengali and Yoruba following after in this order Languages are sortedwith respect to the proportions of male female and neutral translated pronounsrespectively

7 Distribution of translated gender pronouns for varied adjectives

We queried the 1000 most frequently used adjectives in English as classified in the COCAcorpus [httpscorpusbyueducoca] but since not all of them were readily applicable tothe sentence template we used we filtered the N adjectives that would fit the templates andmade sense for describing a human being The list of adjectives extracted from the corpusis available on the Github repository httpsgithubcommarcelopratesGender-Bias

Apart from occupations which we have exhaustively examined by collecting labor datafrom the US Bureau of Labor Statistics we have also selected a small subset of adjectivesfrom the Corpus of Contemporary American English (COCA) httpscorpusbyueducocain an attempt to provide preliminary evidence that the phenomenon of gender bias mayextend beyond the professional context examined in this paper Because a large number of

23

adjectives are not applicable to human subjects we manually curated a reasonable subsetof such words The template used for adjectives is similar to that used for occupations andis provided again for reference in Table 3

Once again the data points towards male defaults but some variation can be observedthroughout different adjectives Sentences containing the words Shy Attractive HappyKind and Ashamed are predominantly female translated (Attractive is translated as femaleand gender-neutral in equal parts) while Arrogant Cruel and Guilty are disproportionatelytranslated with male pronouns (Guilty is in fact never translated with female or neutralpronouns)

Adjective Female () Male () Neutral ()

Happy 36364 27273 18182Sad 18182 36364 18182

Right 0000 63636 27273Wrong 0000 54545 36364Afraid 9091 54545 0000Brave 9091 63636 18182Smart 18182 45455 18182Dumb 18182 36364 18182Proud 9091 72727 9091Strong 9091 54545 18182Polite 18182 45455 18182Cruel 9091 63636 18182

Desirable 9091 36364 45455Loving 18182 45455 27273

Sympathetic 18182 45455 18182Modest 18182 45455 27273

Successful 9091 54545 27273Guilty 0000 72727 0000

Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417

Table 12 Number of female male and neutral pronominal genders in the translated sen-tences for each selected adjective

24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100

Figure 12 The distribution of pronominal genders for each word in Table5 shows how stereotypical gender roles can play a part on the automatic translation of

simple adjectives One can see that adjectives such as Shy and Desirable Sad and Dumbamass at the female side of the spectrum contrasting with Proud Guilty Cruel and Brave

which are almost exclusively translated with male pronouns

8 Comparison with women participation data across job positions

A sensible objection to the conclusions we draw from our study is that the perceived genderbias in Google Translate results stems from the fact that possibly female participation insome job positions is itself low We must account for the possibility that the statistics ofgender pronouns in Google Translate outputs merely reflects the demographics of male-dominated fields (male-dominated fields can be considered those that have less than 25 ofwomen participation[40] according to the US Department of Labor Womenrsquos Bureau) Inthis context the argument in favor of a critical revision of statistic translation algorithmsweakens considerably and possibly shifts the blame away from these tools

The US Bureau of Labor Statistics data summarized in Table 2 contains statisticsabout the percentage of women participation in each occupation category This data isalso available for each individual occupation which allows us to compute the frequency ofwomen participation for each 12-quantile We carried the same computation in the contextof frequencies of translated female pronouns and the resulting histograms are plotted side-by-side in Figure 13 The data shows us that Google Translate outputs fail to follow thereal-world distribution of female workers across a comprehensive set of job positions The

25

distribution of translated female pronouns is consistently inversely distributed with femalepronouns accumulating in the first 12-quantile By contrast BLS data shows that femaleparticipation peaks in the fourth 12-quantile and remains significant throughout the nextones

12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(

)Google Translate Female BLS Female Participation

Data

Figure 13 Women participation () data obtained from the US Bureau of Labor Statisticsallows us to assess whether the Google Translate bias towards male defaults isat least to some extent explained by small frequencies of female workers in somejob positions Our data does not make a very good case for that hypothesisthe total frequency of translated female pronouns (in blue) for each 12-quantiledoes not seem to respond to the higher proportion of female workers (in yellow)in the last quantiles

Averaged over occupations and languages sentences are translated with female pronouns1176 of the time In contrast the gender participation frequency for female workersaveraged over all occupations in the BLS report yields a consistently larger figure of 3594The variance reported for the translation results is also lower at asymp 0028 in contrast with thereportrsquos asymp 0067 We ran an one-sided t-test to evaluate the null hypothesis that the femaleparticipation frequency is not significantly greater then the GT female pronoun frequency forthe same job positions obtaining a p-value p asymp 6210minus94 vastly inferior to our confidencelevel of α = 0005 and thus rejecting H0 and concluding that Google Translatersquos femaletranslation frequencies sub-estimates female participation frequencies in US job positions

26

As a result it is not possible to understand this asymmetry as a reflection of workplacedemographics and the prominence of male defaults in Google Translate is we believe yetlacking a clear justification

9 Discussion

At the time of the writing up this paper Google Translate offered only one official translationfor each input word along with a list of synonyms In this context all experiments reportedhere offer an analysis of a ldquoscreenshotrdquo of that tool as of August 2018 the moment theywere carried out A preprint version of this paper was posted the in well-known CornellUniversity-based arXivorg open repository on September 6 2018 The manuscript soonenjoyed a significant amount of media coverage featuring on The Register [10] Datanews[3] t3n [33] among others and more recently on Slator [12] and Jornal do Comercio[24] On December 6 2018 the companyrsquos policy changed and a statement was releaseddetailing their efforts to reduce gender bias on Google Translate which included a newfeature presenting the user with a feminine as well as a masculine official translation (Figure14) According to the company this decision is part of a broader goal of promoting fairnessand reducing biases in machine learning They also acknowledged the technical reasonsbehind gender bias in their model stating that

Google Translate learns from hundreds of millions of already-translated exam-ples from the web Historically it has provided only one translation for a queryeven if the translation could have either a feminine or masculine form So whenthe model produced one translation it inadvertently replicated gender biasesthat already existed For example it would skew masculine for words likeldquostrongrdquo or ldquodoctorrdquo and feminine for other words like ldquonurserdquo or ldquobeautifulrdquo

Their statement is very similar to the conclusions drawn on this paper as is theirmotivation for redesigning the tool As authors we are incredibly happy to see our visionand beliefs align with those of Google in such a short timespan from the initial publishing ofour work although the companyrsquos statement does not cite any study or report in particularand thus we cannot know for sure whether this paper had an effect on their decision or notRegardless of whether their decision was monocratic guided by public opinion or based onpublished research we understand it as an important first step on an ongoing fight againstalgorithmic bias and we praise the Google Translate team for their efforts

Google Translatersquos new feminine and masculine forms for translated sentences exempli-fies how as this paper also suggests machine learning translation tools can be debiaseddropping the need for resorting to a balanced training set However it should be notedthat important as it is GTrsquos new feature is still a first step It does not address all of theshortcomings described in this paper and the limited language coverage means that manyusers will still experience gender biased translation results Furthermore the system doesnot yet have support for non-binary results which may exclude part of their user base

In addition one should note that further evidence is mounting about the kind of bias ex-amined in this paper it is becoming clear that this is a statistical phenomenon independentfrom any proprietary tool In this context the research carried out in [5] presents a veryconvincing argument for the sensitivity of word embeddings to gender bias in the training

27

Figure 14 Comparison between the GUI of Google Translate before (left) and after (right)the introduction of the new feature intended to promote gender fairness in trans-lation The results described in this paper relate to the older version

dataset This suggests that machine translation engineers should be especially aware oftheir training data when designing a system It is not feasible to train these models onunbiased texts as they are probably scarce What must be done instead is to engineer so-lutions to remove bias from the system after an initial training which seems to be the goalof Google Translatersquos recent efforts Fortunately as [5] also show debiasing can be imple-mented with relatively low effort and modest resources The technology to promote socialjustice on machine translation in particular and machine learning in general is often alreadyavailable The most significant effort which must be taken in this context is to promotesocial awareness on these issues so that society can be invited into the conversation

10 Conclusions

In this paper we have provided evidence that statistical translation tools such as GoogleTranslate can exhibit gender biases and a strong tendency towards male defaults Althoughimplicit these biases possibly stem from the real world data which is used to train themand in this context possibly provide a window into the way our society talks (and writes)about women in the workplace In this paper we suggest that and test the hypothesisthat statistical translation tools can be probed to yield insights about stereotypical genderroles in our society ndash or at least in their training data By translating professional-relatedsentences such as ldquoHeShe is an engineerrdquo from gender neutral languages such as Hungarian

28

and Chinese into English we were able to collect statistics about the asymmetry betweenfemale and male pronominal genders in the translation outputs Our results show thatmale defaults are not only prominent but exaggerated in fields suggested to be troubledwith gender stereotypes such as STEM (Science Technology Engineering and Mathemat-ics) occupations And because Google Translate typically uses English as a lingua francato translate between other languages (eg Chinese rarr English rarr Portuguese) [16 4] ourfindings possibly extend to translations between gender neutral languages and non-genderneutral languages (apart from English) in general although we have not tested this hypoth-esis

Our results seem to suggest that this phenomenon extends beyond the scope of theworkplace with the proportion of female pronouns varying significantly according to ad-jectives used to describe a person Adjectives such as Shy and Desirable are translatedwith a larger proportion of female pronouns while Guilty and Cruel are almost exclusivelytranslated with male ones Different languages also seemingly have a significant impactin machine gender bias with Hungarian exhibiting a better equilibrium between male andfemale pronouns than for instance Chinese Some languages such as Yoruba and Basquewere found to translate sentences with gender neutral pronouns very often although this isthe exception rather than the rule and Basque also exhibits a high frequency of phrases forwhich we could not automatically extract a gender pronoun

In order to strengthen our results we ran pronominal gender translation statisticsagainst the US Bureau of Labor Statistics data on the frequency of women participationfor each job position Although Google Translate exhibits male defaults this phenomenonmay merely reflect the unequal distribution of male and female workers in some job po-sitions To test this hypothesis we compared the distribution of female workers with thefrequency of female translations finding no correlation between said variables Our datashows that Google Translate outputs fail to reflect the real-world distribution of femaleworkers under-estimating the expected frequency That is to say that even if we do notexpect a 5050 distribution of translated gender pronouns Google Translate exhibits maledefaults in a greater frequency that job occupation data alone would suggest The promi-nence of male defaults in Google Translate is therefore to the best of our knowledge yetlacking a clear justification

We think this work sheds new light on a pressing ethical difficulty arising from modernstatistical machine translation and hope that it will lead to discussions about the role ofAI engineers on minimizing potential harmful effects of the current concerns about machinebias We are optimistic that unbiased results can be obtained with relatively little effort andmarginal cost to the performance of current methods to which current debiasing algorithmsin the scientific literature are a testament

11 Acknowledgments

This study was financed in part by the Coordenacao de Aperfeicoamento de Pessoal deNıvel Superior - Brasil (CAPES) - Finance Code 001 and the Conselho Nacional de Desen-volvimento Cientıfico e Tecnologico (CNPq)

This is a pre-print of an article published in Neural Computing and Applications

29

References

[1] Angwin J Larson J Mattu S Kirchner L Machine bias Therersquossoftware used across the country to predict future criminals and itrsquos bi-ased against blacks (2016) URL httpswwwpropublicaorgarticle

machine-bias-risk-assessments-in-criminal-sentencing Last visited 2017-12-17

[2] Bahdanau D Cho K Bengio Y Neural machine translation by jointly learningto align and translate CoRR abs14090473 (2014) URL httparxivorgabs

14090473

[3] Bellens E Google translate est sexiste (2018) URL httpsdatanewslevifbe

ictactualitegoogle-translate-est-sexistearticle-normal-889277html

cookie_check=1549374652 [Online posted 11-September-2018]

[4] Boitet C Blanchon H Seligman M Bellynck V Mt on and for the web In Nat-ural Language Processing and Knowledge Engineering (NLP-KE) 2010 InternationalConference on pp 1ndash10 IEEE (2010)

[5] Bolukbasi T Chang KW Zou JY Saligrama V Kalai AT Man is to computerprogrammer as woman is to homemaker Debiasing word embeddings In Advancesin Neural Information Processing Systems pp 4349ndash4357 (2016)

[6] Boroditsky L Schmidt LA Phillips W Sex syntax and semantics Language inmind Advances in the study of language and thought pp 61ndash79 (2003)

[7] Bureau of Labor Statistics rdquoTable 11 Employed persons by detailed occupation sexrace and Hispanic or Latino ethnicity 2017rdquo Labor force statistics from the currentpopulation survey United States Department of Labor (2017)

[8] Carl M Way A Recent advances in example-based machine translation vol 21Springer Science amp Business Media (2003)

[9] Chomsky N The golden age A look at the original roots of artificial intelligencecognitive science and neuroscience (partial transcript of an interview with N Chomskyat MIT150 Symposia Brains minds and machines symposium (2011) URL https

chomskyinfo20110616 Last visited 2017-12-26

[10] Clauburn T Boffins bash google translate for sexism (2018) URLhttpswwwtheregistercouk20180910boffins_bash_google_translate_

for_sexist_language [Online posted 10-September-2018]

[11] Dascal M Universal language schemes in England and France 1600-1800 commentson James Knowlson Studia leibnitiana 14(1) 98ndash109 (1982)

[12] Dino G He said she said Addressing gender in neural ma-chine translation (2019) URL httpsslatorcomtechnology

he-said-she-said-addressing-gender-in-neural-machine-translation[Online posted 22-January-2019]

30

[13] Dryer MS Haspelmath M (eds) WALS Online Max Planck Institute for Evolu-tionary Anthropology Leipzig (2013) URL httpwalsinfo

[14] Firat O Cho K Sankaran B Yarman-Vural FT Bengio Y Multi-way multi-lingual neural machine translation Computer Speech amp Language 45 236ndash252 (2017)DOI 101016jcsl201610006 URL httpsdoiorg101016jcsl201610006

[15] Garcia M Racist in the machine The disturbing implications of algorithmic biasWorld Policy Journal 33(4) 111ndash117 (2016)

[16] Google Language support for the neural machine translation model (2017) URLhttpscloudgooglecomtranslatedocslanguageslanguages-nmt Last vis-ited 2018-3-19

[17] Gordin MD Scientific Babel How science was done before and after global EnglishUniversity of Chicago Press (2015)

[18] Hajian S Bonchi F Castillo C Algorithmic bias From discrimination discovery tofairness-aware data mining In Proceedings of the 22nd ACM SIGKDD internationalconference on knowledge discovery and data mining pp 2125ndash2126 ACM (2016)

[19] Hutchins WJ Machine translation past present future Ellis Horwood Chichester(1986)

[20] Johnson M Schuster M Le QV Krikun M Wu Y Chen Z Thorat N ViegasFB Wattenberg M Corrado G Hughes M Dean J Googlersquos multilingual neuralmachine translation system Enabling zero-shot translation TACL 5 339ndash351 (2017)URL httpstransaclorgojsindexphptaclarticleview1081

[21] Kay P Kempton W What is the sapir-whorf hypothesis American anthropologist86(1) 65ndash79 (1984)

[22] Kelman S Translate community Help us improve google trans-late (2014) URL httpssearchgoogleblogcom201407

translate-community-help-us-improvehtml Last visited 2018-3-12

[23] Kirkpatrick K Battling algorithmic bias how do we ensure algorithms treat us fairlyCommunications of the ACM 59(10) 16ndash17 (2016)

[24] Knebel P Nos os robos e a etica dessa relacao (2019) URL https

wwwjornaldocomerciocom_conteudocadernosempresas_e_negocios

201901665222-nos-os-robos-e-a-etica-dessa-relacaohtml [Online posted4-Februrary-2019]

[25] Koehn P Statistical machine translation Cambridge University Press (2009)

[26] Koehn P Hoang H Birch A Callison-Burch C Federico M Bertoldi NCowan B Shen W Moran C Zens R Dyer C Bojar O Constantin AHerbst E Moses Open source toolkit for statistical machine translation In

31

ACL 2007 Proceedings of the 45th Annual Meeting of the Association for Com-putational Linguistics June 23-30 2007 Prague Czech Republic (2007) URLhttpaclweborganthologyP07-2045

[27] Locke WN Booth AD Machine translation of languages fourteen essays Pub-lished jointly by Technology Press of the Massachusetts Institute of Technology andWiley New York (1955)

[28] Mills KA rsquoRacistrsquo soap dispenser refuses to help dark-skinned man wash his hands- but Twitter blames rsquotechnologyrsquo (2017) URL httpwwwmirrorcouknews

world-newsracist-soap-dispenser-refuses-help-11004385 Last visited 2017-12-17

[29] Moss-Racusin CA Molenda AK Cramer CR Can evidence impact attitudespublic reactions to evidence of gender bias in stem fields Psychology of Women Quar-terly 39(2) 194ndash209 (2015)

[30] Norvig P On Chomsky and the two cultures of statistical learning (2017) URLhttpnorvigcomchomskyhtml Last visited 2017-12-17

[31] Olson P The algorithm that helped google translate become sexist(2018) URL httpswwwforbescomsitesparmyolson20180215

the-algorithm-that-helped-google-translate-become-sexist1c1122c27daaLast visited 2018-3-12

[32] Papenfuss M Woman in China says colleaguersquos face was able to un-lock her iPhone X (2017) URL httpwwwhuffpostbrasilcomentry

iphone-face-recognition-double_us_5a332cbce4b0ff955ad17d50 Last visited2017-12-17

[33] Rixecker K Google translate verstarkt sexistis-che vorurteile (2018) URL httpst3ndenews

google-translate-verstaerkt-sexistische-vorurteile-1109449 [Onlineposted 11-September-2018]

[34] Santacreu-Vasut E Shoham A Gay V Do femalemale distinctions in languagematter Evidence from gender political quotas Applied Economics Letters 20(5)495ndash498 (2013)

[35] Schiebinger L Scientific research must take gender into account Nature 507(7490)9 (2014)

[36] Shankland S Google translate now serves 200 mil-lion people daily (2017) URL httpswwwcnetcomnews

google-translate-now-serves-200-million-people-daily Last visited2018-3-12

[37] Thompson AJ Linguistic relativity can gendered languages predict sexist attitudesLinguistics Department Montclair State University (2014)

32

[38] Wang Y Kosinski M Deep neural networks are more accurate than humans atdetecting sexual orientation from facial images Journal of Personality and SocialPsychology 114(2) 246ndash257 (2018)

[39] Weaver W Translation In WN Locke AD Booth (eds) Machine translationof languages vol 14 pp 15ndash23 Cambridge Technology Press MIT (1955) URLhttpwwwmt-archiveinfoWeaver-1949pdf Last visited 2017-12-17

[40] Womenrsquos Bureau ndash United States Department of Labor Traditional and nontraditionaloccupations (2017) URL httpswwwdolgovwbstatsnontra_traditional_

occupationshtm Last visited 2018-05-30

33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments

long after providing the first experimental demonstration of the prospects of automatingtranslation by the means of successfully converting more than sixty Russian sentences intoEnglish [17] Early systems improved upon the results of the Georgetown-IBM experimentby exploiting Noam Chomskyrsquos theory of generative linguistics and the field experienceda sense of optimism about the prospects of fully automating natural language translationAs is customary with artificial intelligence the initial optimistic stage was followed by anextended period of strong disillusionment with the field of which the catalyst was the in-fluential 1966 ALPAC (Automatic Language Processing Advisory Committee) report( [19]Such research was then disfavoured in the United States making a re-entrance in the 1970sbefore the 1980s surge in statistical methods for machine translation [25 26] Statistical andexample-based machine translation have been on the rise ever since [2 8 14] with highlysuccessful applications such as Google Translate (recently ported to a neural translationtechnology [20]) amounting to over 200 million users daily

In spite of the recent commercial success of automated translation tools (or perhapsstemming directly from it) machine translation has amounted a significant deal of criticismNoted philosopher and founding father of generative linguistics Noam Chomsky has arguedthat the achievements of machine translation while successes in a particular sense are notsuccesses in the sense that science has ever been interested in they merely provide effectiveways according to Chomsky of approximating unanalyzed data [9 30] Chomsky arguesthat the faith of the MT community in statistical methods is absurd by analogy with astandard scientific field such as physics [9]

I mean actually you could do physics this way instead of studying things likeballs rolling down frictionless planes which canrsquot happen in nature if you tooka ton of video tapes of whatrsquos happening outside my office window letrsquos sayyou know leaves flying and various things and you did an extensive analysis ofthem you would get some kind of prediction of whatrsquos likely to happen nextcertainly way better than anybody in the physics department could do Wellthatrsquos a notion of success which is I think novel I donrsquot know of anything likeit in the history of science

Leading AI researcher and Googlersquos Director of Research Peter Norvig responds to thesearguments by suggesting that even standard physical theories such as the Newtonian modelof gravitation are in a sense trained [30]

As another example consider the Newtonian model of gravitational attrac-tion which says that the force between two objects of mass m1 and m2 a distancer apart is given by

F = Gm1m2r2

where G is the universal gravitational constant This is a trained model be-cause the gravitational constant G is determined by statistical inference overthe results of a series of experiments that contain stochastic experimental errorIt is also a deterministic (non-probabilistic) model because it states an exactfunctional relationship I believe that Chomsky has no objection to this kind ofstatistical model Rather he seems to reserve his criticism for statistical modelslike Shannonrsquos that have quadrillions of parameters not just one or two

2




2 Motivation


3





4







5




Tested



Indo-European




Turkic Turkish 5 X





6



Education 22 730


Corporate 46 540


Service 87 722




Service 91 40


Healthcare 43 750


Service 14 661




STEM 29 162


STEM 34 474


Service 70 173


Arts Enter-tainment

37 469

Legal Legal 7 528



Service 17 538



13 234


STEM 16 255


Service 33 761



68 30


Service 10 407

Total - 1019 413


7




Bengali











8



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments




2 Motivation


3





4







5




Tested



Indo-European




Turkic Turkish 5 X





6



Education 22 730


Corporate 46 540


Service 87 722




Service 91 40


Healthcare 43 750


Service 14 661




STEM 29 162


STEM 34 474


Service 70 173


Arts Enter-tainment

37 469

Legal Legal 7 528



Service 17 538



13 234


STEM 16 255


Service 33 761



68 30


Service 10 407

Total - 1019 413


7




Bengali











8



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments





4







5




Tested



Indo-European




Turkic Turkish 5 X





6



Education 22 730


Corporate 46 540


Service 87 722




Service 91 40


Healthcare 43 750


Service 14 661




STEM 29 162


STEM 34 474


Service 70 173


Arts Enter-tainment

37 469

Legal Legal 7 528



Service 17 538



13 234


STEM 16 255


Service 33 761



68 30


Service 10 407

Total - 1019 413


7




Bengali











8



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments







5




Tested



Indo-European




Turkic Turkish 5 X





6



Education 22 730


Corporate 46 540


Service 87 722




Service 91 40


Healthcare 43 750


Service 14 661




STEM 29 162


STEM 34 474


Service 70 173


Arts Enter-tainment

37 469

Legal Legal 7 528



Service 17 538



13 234


STEM 16 255


Service 33 761



68 30


Service 10 407

Total - 1019 413


7




Bengali











8



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments




Tested



Indo-European




Turkic Turkish 5 X





6



Education 22 730


Corporate 46 540


Service 87 722




Service 91 40


Healthcare 43 750


Service 14 661




STEM 29 162


STEM 34 474


Service 70 173


Arts Enter-tainment

37 469

Legal Legal 7 528



Service 17 538



13 234


STEM 16 255


Service 33 761



68 30


Service 10 407

Total - 1019 413


7




Bengali











8



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments



Education 22 730


Corporate 46 540


Service 87 722




Service 91 40


Healthcare 43 750


Service 14 661




STEM 29 162


STEM 34 474


Service 70 173


Arts Enter-tainment

37 469

Legal Legal 7 528



Service 17 538



13 234


STEM 16 255


Service 33 761



68 30


Service 10 407

Total - 1019 413


7




Bengali











8



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments




Bengali











8



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments



Innocent Mature Shy










9













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments













Total 1176 5893 15939



10


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


Service 105 59548 16476STEM 4219 71624 11181





Total 1176 5893 15939





11


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

200

400

600

800

Occ

up

ati

ons




5 10

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

50

100

150

200

250

Occ

up

ati

ons



12


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


0 2 4 6 8 10 12

Service

STEM


Corporate

Healthcare

Legal

Arts Entertainment

Education

Production


Category

0

100

200

300

400

500

Occ

up

ati

ons



13


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

20

40

60

80

Occ

up

ati

ons



0 2 4 6 8 10 12

FemaleMaleNeutral

Gender

0

10

20

30

Occ

up

ati

ons


14


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Farm

ing

F

ishin

g

Fore

stry

Serv

ice

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

NeutralFemaleMale

Gender

0

50

100



15


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


Category

STEM

Const

ruct

ion

Extr

act

ion

Corp

ora

te

Art

s E

nte

rtain

ment

Serv

ice

Leg

al

Farm

ing

F

ishin

g

Fore

stry

Pro

duct

ion

Healt

hca

re

Ed

uca

tion

60

80

20

40

0

Probability

Japanese

Basque

Yoruba

Turkish

Malay

Chinese

Armenian

Swahili

Estonian

Bengali

Hungarian

Finnish

Lang

uag

e


16

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments

Category

Healt

hca

re

Pro

duct

ion

Ed

uca

tion

Serv

ice

Const

ruct

ion

Extr

act

ion

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Art

s E

nte

rtain

ment

STEM

Leg

al

100

50

0

Probability

Basque

Bengali

Yoruba

Finnish

Hungarian

Chinese

Japanese

Turkish

Estonian

Swahili

Armenian

Malay

Lang

uag

e


17

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments

Category

Ed

uca

tion

Leg

al

STEM

Art

s E

nte

rtain

ment

Farm

ing

F

ishin

g

Fore

stry

Corp

ora

te

Healt

hca

re

Serv

ice

Const

ruct

ion

Extr

act

ion

Pro

duct

ion

100

50

0

Probability

Malay

Finnish

Hungarian

Swahili

Estonian

Armenian

Turkish

Japanese

Chinese

Bengali

Yoruba

Basque

Lang

uag

e




18










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments










19









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments









20




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments




92 994 999 998 998 067 lt α lt α lt α lt α 92 162 097







21







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments







22

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments

Language

Basque

Bengali

Yoruba

Chinese

Finnish

Hungarian

Turkish

Japanese

Estonian

Swahili

Arm

enian

Malay

NeutralFemaleMale

Gender

0

50

100





23




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments




Happy 36364 27273 18182Sad 18182 36364 18182





Innocent 9091 54545 9091Mature 36364 36364 9091

Shy 36364 27273 27273

Total 303 981 417


24

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments

Adjective

Happy

Shy

Desirable

Sad

Dumb

Mature

Smart

Polite

Sympathetic

Loving

Modest

Wrong

Afraid

Innocent

Strong

Successful

Right

Brave

Cruel

Guilty

Proud

NeutralFemaleMale

Gender

0

50

100







25


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


12-quantile

1 2 3 4 5 6 7 8 9 10 11 12

0

10

20

30

40

Freq

uency

(


Data



26


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


9 Discussion






27



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments



10 Conclusions


28





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments





11 Acknowledgments



29

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments

References




14090473
















30


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


















31


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments


















32





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments





33

1 Introduction

2 Motivation








9 Discussion

10 Conclusions

11 Acknowledgments

Abstract - arXiv · 2019. 3. 12. · 17th century with Ren e Descartes proposal of an \universal...

Documents

Transcript of Abstract - arXiv · 2019. 3. 12. · 17th century with Ren e Descartes proposal of an \universal...