News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf

8/18/2019 News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf

1/36

1

NEWS FEED RATER GUIDELINES

Introduction 2

An Overview of the Rating Process 3

Things to do BEFORE rating each story ......................................................................................................... 3

Things to do WHILE rating each story ........................................................................................................... 3

Rating Explanations 5

Overall Rating: How much do you want to see this in your News Feed? ..................................................... 5

3 Considerations for Overall Rating: Person / Page, Content, and Impact ................................................... 6

Person / Page and Content ........................................................................................................................... 7

Person / Page ............................................................................................................................................ 7

Content ...................................................................................................................................................... 7

Impact ......................................................................................................................................................... 12

Type of Impact ........................................................................................................................................ 12

Amount of Impact ................................................................................................................................... 13

How to use Person / Page, Content, and Impact together ......................................................................... 15 Extra factors related to story ratings .......................................................................................................... 16

Repetition ................................................................................................................................................ 16

Reading Articles, Watching Videos, and Viewing Photo Albums ............................................................ 16

Overview of Ratings .................................................................................................................................... 17

Instructions for Comments

–

First Section 18

Describing Person/Page .............................................................................................................................. 18

Describing Content...................................................................................................................................... 19

Describing Impact (Type and Amount) ....................................................................................................... 19

Describing 2’s and 1’s.................................................................................................................................. 20

Additional Questions 21

Content type: Friend/Family, Public, or Both? ........................................................................................... 21

How much does this story make you feel entertained? ............................................................................. 24

How much does this story help you feel connected to family/friends? ..................................................... 25

How much does this story help you feel informed about the world? ........................................................ 26

Instructions for Comments – Second Section 28

Submitting Your Responses: “Submit”, “Skip”, and “Submit (no share)” 30

When to “Submit” ....................................................................................................................................... 30

When to “Skip” ........................................................................................................................................... 30

When to “Submit (no share)” ..................................................................................................................... 30

Instructions for “Reorder” and “ akeoff” Tasks

31

Where and When to access these tasks ..................................................................................................... 31

Reorder ....................................................................................................................................................... 31

Bakeoff ........................................................................................................................................................ 31

Printable Reference Materials 33


2/36

2

Introduction

Overall Project Goals

Understand how people feel about the stories they see in their News Feed. Understand which stories people want to see in their News Feed, and why.

Why are we using this rating system?

By using this rating system, we’ll be able to understand which stories you think are the very best and

which you think are the very worst. Second, we’ll understand the difference between just good enough

to be in feed and just bad enough to not be there.

An Honest Approach

Your feedback should be as honest as possible. There’s no judgment about what you like to see and

don’t like to see in your News Feed! The more honest the feedback, the more helpful it is.

About this Document

While this document is long, it contains the information necessary for you to complete your ratings.

There are many different types of stories you may see on Facebook, and we’ve created guidelines to

help you give feedback about the stories you see in a thoughtful, systematic way.


3/36

3

An Overview of the Rating Process

Things to do BEFORE rating each story

View the story as you normally would if you saw it in your News Feed to get a naturalistic first

reaction.

Like, comment, share, click, or do anything with the post that you normally would do in everyday

life.

o

It is very important to do this BEFORE giving the post a rating.

o

DO NOT do any of these actions unless you would normally do so on your own time.

o

You can interact with the post from the rating tool.

Things to do WHILE rating each story

Use this rating guide and Printable Reference Materials to determine what numerical ratings to

give a story.

Provide open-ended feedback in the comment boxes to help explain why you rated a story the

way you did.

Complete the additional questions for each story and be sure to reference this rating guide for

these questions as well.

STEP 1

Interact as you

normally would

STEP 2

Complete your

ratings


4/36

4

What to do AFTER you are done with rating all stories for the day

After you have finished your 1 batch of 60 stories, complete the “Bakeoff” and “Reorder” tasks,

explained later on in this document.


5/36

5

Rating Explanations

Overall Rating: How much do you want to see this in your News Feed?

Definitely do

NOT want to see

in Feed

Do not want to

see in Feed

Don’t mind

seeing in Feed

but wouldn’t

mind missing

either

Want to see in

Feed

Definitely want

to see in Feed

1 2 3 4 5

The main factor in considering how you rate a story is ultimately whether or not you want to see this

story in your News Feed. For some stories, it may be very clear whether you want to see them in your

Feed (“5”) or do NOT want to see them in your Feed (“1”). For other stories, you may feel less strongly

about them but still want to see them (“4”) or NOT see them (“2”). And, for some stories, you may feel

ambivalent about whether or not you would want to see them (“3”). In other words, a “3” means it

would be okay if the story were in your Feed, but it would also be okay if the story were not in your

Feed.

5’s should be reserved for the best stories, and 1’s should be reserved for the worst stories. This will

help us to best differentiate between stories that are good vs. stories that are the best (and stories that

are bad vs. stories that are the worst). Therefore, it is likely that you will be choosing 5’s and 1’s less

often than you will be choosing 2’s, 3’s and 4’s.

When rating a post, in addition to considering the main part of the post, you may also take into account

additional information like how many likes and comments a post has, who has liked or commented, orthe nature of the comments. In other words, consider everything you see on the screen in the rating

tool as you normally would consider these aspects of a post in everyday life. If you don’t look at these

things normally, you don’t have to look at them when rating.

Also, we realize that you may have seen some of these posts before in everyday life (e.g., on Facebook

but not while doing your work for Appen) and some you may be seeing for the first time when you rate

them. For posts you’ve seen before, please try to recall your feelings and reactions from the first time

that you saw this and channel this into your ratings. For posts you have not seen before, please be sure

to take the time to make your first impression before you go through the rating process (“Things to do

BEFORE rating each story”).


6/36

6

3 Considerations for Overall Rating: Person / Page, Content, and Impact

In order to arrive at your final score, and to help you mentally work toward your overall rating, you

should take 3 aspects of the story into account:

Person / Page: How much do you care about this person or Page?

Content: How interested are you in seeing this specific content?

Impact: How much of an impact does seeing this have on you?

You must choose a numerical rating for each of these 3 aspects of a story, which will then feed into your

overall rating.

Details about how to approach these questions are discussed in the next section.


7/36

7

Person / Page and Content

Person / Page

Someone youreally don’t care

about

Someone youdon’t care about

Someone yousomewhat care

about

Someone youcare about

Someone youreally care about

1 2 3 4 5

Content

Content you are

really notinterested in

seeing

Content you are

not interested in

seeing

Content you are

somewhatinterested in

seeing

Content you are

interested in

seeing

Content you are

really interested

in seeing

1 2 3 4 5

For Person / Page and Content, there is no right or wrong way to feel. Considering these elements

separately allows you to give high or low overall ratings based on how much you care about the person

(or Page) posting OR based on how much you are interested in seeing the content. There may be posts

from people you care about that contain content you don’t care about. Or, there may be posts from

people you don’t care about but that contain content you do care about. However, there may be manytimes for which your feelings about the person / Page and content match up. That’s fine too!

For Page, you can think about “how much you care about the Page” in a couple different ways. First of

all, Pages can be businesses or brands, news organizations, celebrities, public figures, or any other non-

friend on Facebook. So, you may care about a Page more if you tend to like most of the Page’s content,

or you may like it less if you only like some of the Page’s content. You might also consider how much you

like the Page, how much you trust the Page, or anything else that helps you make an assessment about

how much you care about the Page.

In addition, sometimes there may be multiple people or a person and a Page featured in one story. Here

are examples of some of these, and what you should do in each case:


8/36

8

The person posting is someone you don’t know.

o If you see a story where a friend is tagged or featured in a post by someone else on

Facebook that you don’t know, please give the Person rating based on the person you

do know in the post.

o

Example:

If you know Dan (who

was tagged in a photo)

but not Mike, your

Person rating should be

about Dan.


9/36

9

You know both the person who posted as well as the person who was tagged in a post.

o If you see a story where a friend is tagged or featured in a post by another one of your

friends, your “person” rating should be about the friend who is tagged or featured in the

story .

o

The fact that a story might feature an interaction between two of your friends should be

considered in the “content” rating.

o Example:

If you know Ben (who

was tagged in a photo)AND Steve, your Person

rating should be about

Ben, who was tagged.


10/36

10

You see a post that displays multiple stories at the same time.

o If you see an “aggregate” post (a post where there are two or more people / Pages that

have done something, you should provide your Person rating based on the person in the

story who has the most impact on you.

o

Example: If two of your friends shared an article and you see ONE story about those two

shares, you would probably rate Person based on the friend who is more meaningful to

you in this context.

o Example: If you see a story about multiple people posting to your friend’s timeline for

his/her birthday, you should rate Person based on the person who is receiving those

Choose between

Cameron and Lindsay

for the Person rating by

choosing the friend who

is more meaningful to

you in this context.


11/36

11

posts (the person whose birthday it is), since that is the main reason why you are seeing

that “aggregate story.” (see example below).

o For the Content rating, you should take into account the fact that you are seeing

multiple stories in one and report on how you feel about that content as a whole.

Additional scenarios

There are two or more people you know tagged in a post.

o

For this type of story, you should rate the person who is more meaningful to you in this

context.

Someone posted something in a group.

o For this type of story, you should rate the person who posted in the group, not the

group itself. If this post was interesting to you or had impact on you because it was in a

particular group, you should take that into account in your Content and Impact ratings

and discuss the role of the group in your comments.

The Person rating

for this story

should be based

on Annie and not

based on the

people posting on

her timeline.


12/36

12

Impact

In addition to Person and Content, you must consider: How much impact does this story have on you?

Impact must be considered for every story, and should be derived directly from the Person / Page

and/or Content rating.

For impact, you will provide 2 pieces of feedback: Type of Impact and Amount of Impact.

Type of Impact

You use this information in discussions with people, online or offline.

Examples include:

Talking about what you saw or read with the person who posted.

Talking about what you saw or read with someone else you know.

Sharing the post with others, either through Facebook or via another method.

You gain information that you want to have.

Examples include:

Seeing information about a friend or family member, which can help you stay connected.

Learning about a news story, product, person, event, or something else that is interesting,

informative, useful, valuable, or relevant to you.

Bookmarking the information or saving it in some way.

You have a (desired) emotional reaction:

Examples include:

Feeling delighted, joyful, grateful, proud, excited, or happy.

Laughing or being highly entertained.

Feeling empathy for someone who had a bad day.

Emotions can be negative as well, as long as it is an emotion that you want to feel. For example,

you could feel sad about a friend’s parent passing away but be glad that you saw that sad story,

or feel anger at information in a news article but be glad that you know what happened.

If you feel bad or sad or upset but you still want to see the post in your News Feed because you

are ok with feeling this way, then you should report that you had a desired emotional reaction.

In other words, this type of impact captures the fact that you had an emotional reaction that

was wanted or acceptable in some way . If you feel bad or sad or upset and do not want to feel

this way, then you should not choose “desired emotional reaction” as impact type and choose a

“1” for Amount of Impact, which in this case would be negative. (See next section for more

details on Amount of Impact.)


13/36

13

N/A

You should choose “N/A” when there is no impact (2) or a negative impact (1) on you. For instance, if

you have an undesired emotional reaction, you should choose “N/A” for impact type. If you have an

undesired emotional reaction, you should discuss this in your comments.

If you select “N/A” for Type of Impact, it does not make sense to then choose “Has a little impact onyou” (3), “Has an impact on you” (4), or “Has a significant impact on you” (5) for Amount of Impact.

Remember that if there is any impact, you must identify the type. If there is no impact, then Type of

Impact should be “N/A.”

Amount of Impact

Really does not

have an impact

on you / has a

negative impact

Does not have

an impact onyou

Has a little

impact on you

Has an impact

on you

Has a significant

impact on you

1 2 3 4 5

The Amount of Impact score directly relates to the Type of Impact score in that it reflects how much of

an impact a story had on you.

For instance if you had a desired emotional reaction but it was small or at a low level, the

amount of impact would be 3 (e.g., a chuckle). If you had a desired emotional reaction that was

stronger, amount of impact would be 4 (e.g., a laugh). If you had a really big emotional reaction,

amount of impact would be 5 (e.g., laughed hard for a few minutes).

Or, imagine you picked “You use this information in discussions with people, online or offline.” If

you thought you might use the information in discussions, amount of impact would be 3. If you

thought you probably would use the information in discussions, amount of impact would be 4. If

you were very certain you would use the information in discussions, amount of impact would be

5.

Or, imagine you picked “You gain information that you want to have.” If you thought the

information was a little useful/helpful/etc., amount of impact would be 3. If you thought the

information was useful/helpful/etc., amount of impact would be 4. If you thought theinformation was very useful/helpful/etc., amount of impact would be 5.

Stories should be rated a “5” if they had a significant impact on you in one of the ways described in the

Type of Impact section. The less impact on you, the lower the impact score.

What does a “1” impact mean? (“Really does not have an impact on you / has a negative impact”). A

negative impact includes anything that detracts from your experience on Facebook or makes your


14/36

14

experience worse. If you use words like “annoying” or “offensive”, it should be a 1 on impact. So, the

difference between a 2 and a 1 on impact is that a 2 means “no impact” but a 1 is either a more extreme

version of no impact, or impact that is negative.

Note: only one form of impact needs to be met in order for there to be impact. Not all of these criteria

must be met, just impact in one of the ways mentioned above.

Multiple Types of Impact

In some cases, a story may be associated with multiple types of impact. The Type of Impact question in

the rating tool allows you to choose multiple types of impact:

You may choose multiple types of impact if they apply, but please think carefully about whether morethan one type of impact applies, and refer to your rater guide for the distinctions between impact types.

In addition, if there are multiple types of impact you should rate Amount of Impact based on the highest

level of impact that exists. Here are some examples:

Example:

If “gain information” is a 3 on Amount of Impact and “emotional reaction” is a 4 on Amount of

Impact, the Amount of Impact score = 4.

Example:

If “gain information” is a 5 on Amount of Impact and “emotional reaction” is a 3 on Amount ofImpact, the Amount of Impact score = 5.

Example:

If “gain information” is a 4 on Amount of Impact and “emotional reaction” is a 4 on Amount of

Impact, the Amount of Impact score = 4.


15/36

15

Note: You should NOT average or add the Amount of Impact scores together when there are multiple

Types of Impact. Simply choose your Amount of Impact score based on the highest amount of impact

that exists.

How to use Person / Page, Content, and Impact together

The flow for the ratings should be:

1)

Consider Person / Page and Content.

o In many cases, these ratings will be aligned but in some cases they might differ.

2)

Consider Impact.

o You should be able to draw a clear line between caring about the person / Page

and/or being interested in the content and how much impact it has on you.

o

Depending on the level of impact, however, your Overall rating may go up or down.

o Remember, you will need to record both the type (3 choices or N/A) and amount of

impact (1 through 5 score).

3)

Consider the Person / Page, Content, and Impact scores in relation to the Overall score.

o Verify that these scores align with how much you want to see the story (Overall score).

o If they don’t align, reconsider Impact, which should have the most direct relationship to

the Overall score. (In other words, Person / Page or Content should translate directly to

Impact, which should translate directly to how much you want to see a story).


16/36

16

Extra factors related to story ratings

Repetition

We know that some people post more often than others, and that seeing multiple posts from a person

or Page recently may make you less likely to want to see similar content from that person or Page in the

future. You may take this into consideration. However, if multiple different sources post, for example, asimilar article, please DO NOT consider this when rating for each day.

For example, imagine that multiple friends or Pages are posting articles about a recent news event (e.g.,

Malaysia Airlines missing plane). Please rate each of these stories as if you had not seen other similar

stories.

When you begin a new day of ratings, you are allowed to consider repetition of stories that you saw

from the previous day. In other words, if you see multiple stories on a specific topic that you also saw

stories on the day before, you may consider the repetitiveness of the content when completing your

ratings. If repetition plays a role in your assessment of the story, please describe this in your comments.

Reading Articles, Watching Videos, and Viewing Photo Albums

Articles

When you see a post with an article in it that you would typically read , please go ahead and read

or skim the article (for up to 5 minutes) before making your final rating.

You should take into account how much you are interested in the actual content, and how much

impact it has on you, before you make your final rating about whether or not you want to see it

in your News Feed.

Videos

The same approach (from reading articles) applies to watching videos.

When you see a post with a video in it that you would typically watch, please go ahead and

watch the video (for up to 5 minutes).

Photo Albums

The same approach (from reading articles) applies to photo albums.

If you see a post with a photo album in it that you would typically view, please go ahead and

view some of the photos in the album before making your final rating.

Note: While it is important that you look at the articles, videos, and photos you’re interested in before

completing your ratings, remember that you also must complete your ratings within the time

provided. Please take this into account as you look at content throughout the day.


17/36

17

Overview of Ratings

Score 1 2 3 4 5

Person / Page

Someone you

really don’t careabout

Someone you

don’t care about

Someone you

somewhat careabout

Someone you care

about

Someone you

really care about

Content Content you are

really not

interested to see

Content you are

not interested in

seeing

Content you are

somewhat

interested in

seeing

Content that you

are interested in

seeing

Content you are

really interested in

seeing

Impact

Really does not

have an impact on

you / has a

negative impact

Does not have an

impact on you

Has a little impact

on you

Has an impact on

you

Has a significant

impact on you

Overall Rating Definitely do NOT

want to see in

Feed.

Do not want to see

in Feed.

Don’t mind seeing

in Feed but

wouldn’t mind

missing either.

Want to see in

Feed.

Definitely want to

see in Feed.


18/36

18

Instructions for Comments – First Section

In addition to providing a numerical rating, please indicate why you chose the rating you did. Please

include details that help to further describe the choices you made about rating the Person / Page,Content, and Impact. For instance, you might give feedback about why the person is important (or

unimportant) to you, or why the topic was something you really did (or did not) want to know about.

You could have many other reasons for why you would or would not want to see a story in your News

Feed. Please use the open-ended Reason box to provide additional insights as to why you chose the

rating you did.

We recommend that you describe your feedback in your own words, but we ask that you also

supplement your explanation with words from the guidelines to make clear how you’ve decided on your

ratings. Here’s an example:

“This is a post from a high school friend of mine that I still keep in touch with and care about. She

posted an article about knitting and I like knitting. After looking at the article I learned

something new about knitting, which was cool. So the content was interesting to me and I

gained information that I wanted to have. I want to see this story in my Feed.”

Your explanations should correspond to your numerical ratings. Also, please be sure to discuss Person /

Page, Content, and Impact in your open-ended explanations. All three must be addressed. This will both

help your team leads to best understand your thought processes, as well as assist you in thoroughly

capturing how you feel about a story.

Add your personal touch to each comment

Then tie back into the guidelines

Others should be able to determine what your numerical scores were based on the quality ofyour comments

You must explain why you chose the rating you chose for Person / Page, Content, and Impact. Using

words like “because” and “since” in your sentences is a helpful way to ensure you explain why you feel

how you feel.

Describing Person/Page

Here are some ways you might describe why you care or don’t care about the person or Page you are

rating:

“ I d on’t know anything about this page and never visit it so this is a page I really do not care

about .” (Person = 1)

“ This person is someone I don't care about because she's an acquaintance from high school that I

don't speak to.” (Person = 2)

“ This is from a local friendly acquaintance who I somewhat care about.” (Person = 3)

“I care about this Page because it’s a TV show I like.” (Person = 4)


19/36

19

“This is someone I really care for since she is my sister who I am very close with.” (Person = 5)

Describing Content

Here are some ways you might describe why you are or are not interested in the specific piece ofcontent you are rating:

“T his article is about this diet that is actually really unhealthy which is something that I am really

not interested in seeing in my feed because I really hate it – it’s obnoxious.” (Content = 1)

“ The content is not that interesting to me because it ’s a recipe with olives in it and I don’t like

olives or really cook that much.” (Content = 2)

“ This content is somewhat interesting to me since I like this basketball team a little and I like to

see posts about the team.” (Content = 3)

“ This story features the life of a musician which is interesting to me because I used to be a

musician as well.” (Content = 4)

“ This is a photo of my young niece who lives far away. I am really interested in seeing thisbecause I don’t get to see her very often and I love to see her sweet face and that she is doing

well!” (Content = 5)

Describing Impact (Type and Amount)

Here are some ways you might describe why the story you are rating does or does not have an impact

on you:

“This political joke had a negative impact on me since I was offended by the joke and the

language it used.” (Type of Impact = N/A, Amount of Impact = 1) “This has no impact on me because I didn’t take this quiz and don’t plan to share it with others.”

(Type of Impact = N/A, Amount of Impact = 2)

“This post had a little impact on me since it made me feel slightly happy to see that her dog was

feeling better.” (Type of Impact = have a desired emotional reaction, Amount of Impact = 3)

“This post had an impact because I learned a good amount of information about this country.”

(Type of Impact = gain the information that you want to have, Amount of Impact = 4)

“This had significant impact on me because I’m definitely going to talk about these tips for

communicating tonight with my wife. (Type of Impact = use this information in discussions with

people, Amount of Impact = 5)


20/36

20

Describing 2’s and 1’s

In some cases, there will be people or Pages you don’t/really don’t care about, content that you are

not/really not interested in seeing, and posts that don’t/really don’t have an impact on you. These will

be rated with 2’s or 1’s for the corresponding questions. Though it may be difficult, you should still

explain why you don’t care about a person or Page, why you’re not interested in a particular piece of

content, or why something does not have an impact on you. Here are some examples:

Person/ Page: “Someone you don’t care about”

“This is an old coworker that I don’t care about because I never see them.”

“This is someone I met once at a party and I don’t care about them because they are not an

actual friend.”

“This is someone I went to school with and was never really close with - I don’ t care about him

because it doesn’t matter to me what happens in this person’s life.”

“I don’t care about this Page because it’s for a product I used to like but don’t like anymore.”

“I don’t care about this Page because it’s a celebrity that doesn’t interest me.”

Content: “ Content you are not interested in seeing”

“I don’t find this post interesting because I don’t care to see the art that this friend likes.”

“This post isn’t interesting to me because this story is about bowling and I think bowling is

bor ing.”

“This doesn’t interest me because it’s a blog post about a show I don’t watch and am not

interested in watching.”

Impact: “Does not have an impact on you”

“ This had no impact on me since I probably won't take part in the conversation.”

“ Since I am not planning on doing anything with this information, this info. is not useful to meand it does not have an impact on me.”

“It doesn’t impact me because I will not be playing this video game in the future and have no

emotional reaction to seeing this.”

** IMPORTANT **

Be sure to enter this information in the Comment box on the left hand side of your screen, not in the

comment box of the actual News Feed story, in the middle of your screen. Otherwise, your comments

describing your ratings will be shared on Facebook (yikes!)


21/36

21

Additional Questions

In addition to the main rating, we’d like you to answer some additional questions about each story that

you see.

** IMPORTANT **

This additional set of questions is NOT related to the previous set of questions. You should consider

these questions to be separate and should not worry about aligning your answers to these additional

questions with your answers to the first set of questions.

Content type: Friend/Family, Public, or Both?

For this question, we’d like you to categorize which type of content the story contains – content about a

friend or family member, “Public” content, or both? In general, content that is Public is anything posted

by a Page or other entity like a news organization or celebrity. On the other hand, Friend / Family

content is produced by people you know.

In general, if a friend shares something that is public content, you should rate this as “Public.” However,

if a friend shares something that is Public content but he/she also adds additional personal information

in the post that makes the overall story seem personal, you should rate that as “Both.” In other words,

you should make a distinction between articles (or other pieces of Public content) where the main

information you gain is from the article vs. articles where you both learn something from the article AND

about your friend who shared it. See below for examples.

Friend/Family Content

Stories about people you know personally

Photos, status updates, or videos created and posted by a friend or family member

Updates from groups you belong to

Public Content

Stories about public things or people you don’t know personally

Content from a Page (e.g., public figure, business, news organization)

News photos, articles, or videos

Celebrity content

Friend shares of “Public” content where the information you gain is about the content and

NOT about your friend

Both

Friend shares of “Public” content where the information you gain is about BOTH the content

AND your friend

Friend shares of “Public” content where:

o You learn more about your friend


22/36

22

o You learn how your friend feels about something

o You feel closer to your friend

o For you, the post is more about your friend than the public content that is being

shared

o Here is an example to help explain the when you would pick “Public” vs. when you

would pick “Both”:

For the two stories above, the one on the left is an article shared by a friend with not much additional

context or commentary. It likely does not provide much information about the friend – it says “nice

narrative.” The story on the right, however, might reveal to you that your friend is very passionate about

cycling, or that he/she is planning to vote in an upcoming election. Since there is information about a

friend alongside a post from a public Page that the friend is sharing, this could be considered “Both.”

Probably

“Both”

Probably

“Public”


23/36

23

In addition, here’s a decision tree you should use to help you decide whether a story is “Friends /

Family”, “Public”, or “Both.”


24/36

24

How much does this story make you feel entertained?

There are many types of stories you might see on Facebook. One aspect that we’d like you to consider

for each story is – how entertained did the story make you feel? You should use the following scale to

answer this:

Really notentertained

Notentertained

A littleentertained

Entertained Reallyentertained

1 2 3 4 5

For this question, we want you to only consider stories that are about entertainment-style content.

Examples of entertainment-style content are:

Movie trailers, music videos, TV show clips

Videos, articles, photos, and other media (e.g., memes, gifs) that make you laugh or amuse you

Things that are lighthearted in nature

Things that are jokes, are silly, or “just for fun”

Things that are NOT necessarily entertaining are:

A nice picture of your friend

An interesting article

A useful update about something, like an event

However, just because a story features entertainment-style content doesn’t mean that you feelentertained by the story. So, if you see something like a meme or a TV show clip that doesn’t make you

feel entertained, you should choose “Not entertained” (2).

Stories should be rated “Really not entertained” (1) if something gives you a negative feeling. For

instance, if a post was intended to be humorous but you felt it was in poor taste or offensive, you should

rate it “Really not entertained.”

You may find that you’ve already considered how entertained you were by a story when talking about

type of impact (desired emotional reaction). That’s ok! This is a more specific way for us to understand

which stories make you feel entertained.

** IMPORTANT **

“Entertaining” should be considered as a different dimension from “interesting.”


25/36

25

How much does this story help you feel connected to family/friends?

One of the things that people use Facebook for is to stay connected with friends and family. In addition

to telling us how much you care about the person related to the story, we’d like you to consider for each

story – how connected to family/friends did this story make you feel? You should use the following scale

to answer this:

Really not

connected

Not

connected

A little

connectedConnected

Really

connected

1 2 3 4 5

By feeling connected to family and friends, we mean a feeling more than just“I'm reminded that this

person is around” or “I like seeing these people on Facebook.”

You don’t have to actually connect to / communicate with someone for this question to get a high rating

– it’s more about how connected you feel to the person by seeing the information inthe story.However, if you do communicate with the person because of seeing the story and that makes you feel

connected, feel free to take that into account.

By feeling connected to family and friends, we mean that you feel closer to the person or people in the

story, whether by learning something about them, interacting with them, or enhancing your relationship

with them in some way.

Stories should be rated “Really not connected” (1) if something gives you a negative feeling. Forinstance, if the story was about family or friends but ended up making you feel alienated, disconnected,

irritated, or offended, you should rate it “Really not connected.”

This question is completely separate from the previous set of questions - don't worry if you give

something a 4 or a 5 for your Overall rating but do not rate it as a 4 or 5 for this question.

Note: This question is ONLY about feeling connected to family and friends. This question is NOT about

feeling connected to a Page, to a celebrity, to a news story, groups of people you don’t actually know,

or to any other entity. Accordingly, if the story is from a Page, it should most likely be rated “ Not

connected ” ( 2).


26/36

26

How much does this story help you feel informed about the world?

Again, there are many types of stories you might see on Facebook, and we’re interested in

understanding the different aspects of each story. Some stories in News Feed can help inform you, teach

you something, or keep you up-to-date about the world around you. In this case, “world” could mean as

close to you as something in your own neighborhood or as far from you as something going on in

another country.

For this question, we want you to consider – how informed about the world does this story make you

feel? You should use the following scale to answer this:

Really not

informed

Not informed A little

informedInformed

Really

informed

1 2 3 4 5

The main thing to remember for this question is: we are NOT asking about information about friends,

family, or people you know. (“ How much does this story help you feel connected to family/friends? ” is

the question that focuses on people you know.) In addition, “informed about the world” does NOT mean

the same thing as “interesting” or “entertaining”.

Here are some examples of things that might make you feel informed about the world:

A local, US, or World news story

A story about weather, sports, or celebrities

A human interest or feature story

An upcoming movie or product release

An event going on in your town

Something instructional (e.g., recipe, “how to”)

A friend discussing any of the above

Here are some examples of things that probably would NOT make you feel informed about the world (in

the way we’re asking about it here):

A post from a friend with a picture of his/her new baby

A post that a family member is not feeling well

A post about a friend’s relationship, birthday, or pet

A picture or video that is funny but doesn’t give you any information


27/36

27

As usual, if the story is something like an article or video, only read or watch the content if you would do

so in real life. If you don't feel informed just by seeing the post in your News Feed that's fine. Similarly, if

you don't feel informed even after reading or watching the content, that's fine too! It's all about how

informed you feel.

In addition, if you see something like a news story that provides information but you’re not personally

interested in it and don’t get any information that is valuable to you, then you should choose “Not

informed” (2). In other words, just because a story may provide information does not mean that you will

feel informed by it.

Stories should be rated “Really not informed” (1) if something gives you a negative feeling. For instance,

if the story is misleading, provides false information, makes you feel like you wasted your time, or

presented information that was confusing or irritating to you in some way, you should rate it “Really not

informed.”

A couple other things to keep in mind:

Remember the repetition rules. If you already have the information you see in a story, then it is

okay to say you don’t feel informed by it. However, if you got the information from a story you

rated the same day, please rate the story as if you did not already have the information.

You may have noticed that this question is similar to the type of impact “information you want

to have.” We recognize this. Please try to consider these questions separately. You do not need

to make sure that your impact rating and your “informed” rating match perfectly, just pick the

rating that reflects how much the story makes you feel informed about the world.

** IMPORTANT **

With all of these questions (“entertaining”, “connected”, “informed”), you should not feel compelled

to report that something makes you feel entertained, makes you feel connected to family/friends, or

makes you feel informed about the world if you do not feel that way. It is more important to be

honest in your responses. We expect that these questions will not apply to each story you rate. As

long as you are honest and thoughtful, and explain your choices in the comments (see next section),you are doing the right thing.


28/36

28

Instructions for Comments – Second Section

In addition to writing comments for the first set of ratings (Person / Page, Content, Impact, and Overall

Rating), you will provide comments to explain your responses on the 3 additional questions mentioned

previously (“ How much does this story… make you feel entertained, help you feel connected to

family/friends, help you feel informed about the world?” )

As with the first section’s comments, your comments should be in your own voice but should reference

the specific questions you answered. The comments should focus on answering the “why” of the

responses you chose. Here are a few examples:

“This post made me feel really entertained because I watched the video of the little

girl dancing and was amazed by how good she was, plus it was fun to watch. It didn’t

make me feel connected to the person who posted it because it has nothing to do with

her personally, and overall this doesn’t make me feel informed.”

(Entertaining = “Really entertained” , Connected = “Not connected” ,

Informed = “Not informed” )

“This post was more serious so it didn’t make me feel entertained, but it did help me

feel really connected to my cousin. She is moving across the country so me seeing a

picture of that was good because I now know about a big moment in her life. This post

is about my cousin, so it didn’t make me feel informed about the world.”

(Entertaining = “Not entertained”, Connected = “Really connected”,

Informed = “Not informed”)

“This post does nothing for me – I did not feel entertained because I don’t find this

topic amusing , I don’t feel connected to the person who is in this picture since he didn’treveal much about himself, and it didn’t make me feel informed.”

(Entertaining = “Not entertained”, Connected = “Not connected” ,

Informed = “Not informed” )

Similar to the comments in the first section, you need to explain why you’ve rated something a 1 (e.g.,

Really not entertained, Really not connected to family/friends, Really not informed about the world.

Here are some examples:

Entertained = 1 (Really not entertained)

o “This post was a meme that was supposed to be funny but I found it offensive to the

group of people it was about which gave me a bad feeling, so I was really not

entertained .”


29/36

29

Connected = 1 (Really not connected)

o “ This post was from my friend trying to look cool but he comes off as not genuine.

This is annoying to me and really does not make me feel connected to him.”

Informed = 1 (Really not informed)

o

“ This story is trying to spread information that is just wrong. I hate that people outthere will see something like this. This is the opposite of making me feel informed – I

really do not feel informed about the world by seeing this.

Please note, it is not okay for you to use the exact language in the examples above. Instead, you should

find your own words to explain why a specific post may not entertain you, help you feel connected to

friends and family, or matter to your life.


30/36

30

Submitting Your Responses: “Submit”, “Skip”, and “Submit (no share)”

When to “Submit”

After you have completed all of the ratings, comments, and questions for a story, you can click “Submit”.

Note: you cannot go back to a story after you have submitted it.

When to “Skip”

As part of ongoing checks, your comments may be reviewed for quality from time to time by Appen

team leads. No other raters will ever see your posts or your ratings.

There may be times where you do not feel comfortable providing ratings on a particular story. The

“skip” option should be used only when you are very uncomfortable providing a rating for a story at all.

“Skip” should NOT be used when:

You’re having a hard time deciding

If you find content offensive or bad (like a gruesome photo). In this case, you should go

ahead and rate that a 1.

When the “rate but don’t share” option will work (see next section)

You should also use “Skip” when you see an error. This includes times when a story does not render

properly in the tool or when you encounter some other type of technical error. We will provide

instructions on how you should report problems in a separate document.

Also, you may have noticed that when you post something to Facebook, it often shows up in your own

News Feed. We will try not to show you your own posts in the tool but if you do see a post that you

yourself made, please do not rate it but instead, skip it.

In addition, if you see a post that you think is a repeat, please DO NOT skip it. Sometimes a post may

look like a repeat but it is actually a new story (e.g., a post you’ve previously seen that has a new like or

comment on it). Again, please rate these stories – do not skip them.

When to “Submit (no share)”

There may be some instances, however, where you feel comfortable providing a rating on a post but do

not feel comfortable sharing it with your Appen team leads. In this case, we ask that you provide a rating

so that your feedback can be analyzed anonymously, in aggregate. No one will ever discuss the ratingswith you if you click “submit (no share)”.


31/36

31

Instructions for “Reorder” and “Bakeoff” Tasks

At the end of the day, and after a break from completing your last story rating you will do 2 additional

tasks called “Reorder” and “Bakeoff”, which are explained below.

Where and When to access these tasks

On the main rating page, there will be links to the Reorder and Bakeoff tasks.

** IMPORTANT **

You may only click on the Reorder and Bakeoff links at the end of the day, after you have completed

rating all of your stories. You are not allowed to click on these links until the end of your workday.

Reorder

At the end of each day of rating (after rating 1 batch of 60 stories), you will view a group of stories on

one screen. Your task is to reorder these stories in the following way: if you could only see one story,

which would it be? That story should be first. If you could only see two stories, which would they be?

Those should be the first and second story. In other words, we’d like you to reorder the stories in order

of how much you would want to see them, with the one you want to see most first.

To do this, you can click on the story and drag it into the right order. Here are some tips for making this

task easier:

To see more stories at once, zoom out in your browser

Drag stories to order them horizontally, with the best story on the far left of the screen and the

least best story at the far right of the screen

Bakeoff

For the bakeoff task, you will view a series of story pairs and will choose which story you would rather

see in your News Feed. In other words, you will “bake off” two stories against each other and pick the

one you would like to see more. Here’s what the tool looks like:


32/36

32

To do this, use the left and right arrows on your keyboard to toggle between the two stories. Then click

the “Save Selection” button or press enter to make your selection. If you click the right arrow, the story

on the right will be outlined in green. If you click the left arrow, the story on the left will be outlined in

green. The story outlined in green at the time you press “Save Selection” or Enter should be the story

you would rather see.


33/36

33

Printable Reference Materials

Work Flow Chart

One page Reference Guide for printing

Please print the one-page reference guide on the last page and keep this next to you while you do the

ratings. When not working on the ratings, please store this one-page reference guide in a safe place.


34/36

34


35/36

Rating 1 2 3 4 5

Person / Page Someone you really

don’t care about

Someone you don’t

care about

Someone you

somewhat care about

Someone you care

about

Someone you really

care about

Content

Content you are

really not interested

to see

Content you are not

interested in seeing

Content you are

somewhat interested

in seeing

Content that you are

interested in seeing

Content you are

really interested in

seeing

Impact

Really does not have

an impact on you /

has a negative impact

Does not have an

impact on you

Has a little impact on

youHas an impact on you

Has a significant

impact on you

Overall

Rating

Definitely do NOT

want to see in Feed

Do not want to see in

Feed

Don’t mind seeing in

Feed but wouldn’t

mind missing either

Want to see in FeedDefinitely want to see

in Feed

Rating 1 2 3 4 5

How much does this story make you feel

entertained?

Really not

entertainedNot entertained

A little

entertainedEntertained

Really

entertained

How much does this story help you feel

connected to friends/ family?

Really not

connectedNot connected

A little

connectedConnected

Really

connected

How much does this story help you feel

informed about the world?

Really not

informedNot informed A little informed Informed Really informed

Content Type

Friend/Family = Stories about people you know

personally that do not contain Public content

Public = Page posts, news articles, or other stories

that are about public topics, not about your

friends/family

Both = Stories where you gain information about a

friend AND the Public content shared

Type of Impact

You use this information

in discussions with people,

online or offline.

You gain information that

you want to have.

You have a (desired)

emotional reaction.

You talk about or share what you saw or read with

others in some way.

The information you learn is interesting, informative,

useful, valuable, or relevant in some way.

You feel delight, joy, excitement, laughter, empathy,

happiness or another desired emotion.


36/36

News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf

Documents

Transcript of News Feed Rater Guidelines - 2015.08.06 (Consultant Version).pdf