Confidence in Music Judgements and the Influence of Expert Reviews

Running head: CONFIDENCE AND EXPERT REVIEWS

Confidence in Music Judgments and the Influence of Expert Reviews

Confidence in Music Appraisal and Mitigating Anchoring from Critical Reviews

Maxwell Alley

Bates College

CONFIDENCE AND EXPERT REVIEWS

Abstract

This experiment looked at anchoring and adjustment in the context of evaluating music

after being exposed to reviews by music critics. College students were shown to hold on

to the anchors provided by critics’ reviews and ultimately influence their evaluation on a

piece of music. Individuals who reported high confidence in music appraisal were no

different in mitigating the strength of the anchor formed by the critical review than those

with low confidence in music appraisal. Confidence in music appraisal was not shown to

produce independent evaluations of music when faced with a critical review.

2


Confidence in Music Judgments and the Influence of Expert Reviews

Can we trust the critics? Evaluating critical reviews from experts across various

dimensions of art is an efficient way of filtering material to discover high quality content.

For the most part, people like to form their own opinions of art, but often consult expert

reviewers to gain a broader perspective on a piece of work. Opinions on pieces of work

can be altered by reading a review, and most commonly one’s views tend to anchor to the

reviewer’s bias. The process of anchoring and adjustment happens when people are given

a reference point (anchor), and when making judgments, end up relying on the anchor

when forming their individual opinion (Tversky & Kahneman,1992). Reviews from

critics are inherently subjective, and a strong adherence to these reviews can influence us

to emphasize the objective qualities of art. Though critics’ reviews are useful, it is

problematic if society’s opinions on art are too heavily based on the potentially biased

perspectives of a few expert reviewers.

The media’s presentation of expert sources can influence an audience's evaluation

of expert opinions. The influence of expert opinions has been shown to alter public

opinion toward social issues. Different portrayals of expert sources were shown to

facilitate acceptance of expert opinions on Health Maintenance Organizations (HMOs)

(Chan,1999). In these circumstances, expert sources affected public opinion by inducing

audience members to scrutinize the content of issue arguments. Those who agreed with

issue arguments of expert sources were more likely to shift their attitudes toward experts’

issue positions. If expert opinions can change people’s views on social issues it is worth

considering how they can affect opinions on music.

3


Reviews containing numerical ratings are particularly effecting in influencing one’s

objective opinion on a piece of work. Providing a numerical rating accompanied with a

review serves as a specific anchor on which people base their judgments. People make

estimates by starting from an initial value that is adjusted to form one’s own estimate.

Oftentimes people adjust insufficiently and allow their estimates to be significantly

influenced by the anchor (Tversky & Kahneman, 1974). In an experiment where

participants were asked to evaluate the pricing of a product, anchoring and adjustment

influenced their views on the value of a product by providing them with a price reference

of the (maximum, minimum, or no price reference) (Elaad, Sayag, Exer, 2010).

Participants held on to the anchor of the price and gave higher value to the products in the

high price condition. Critics’ music reviews with numerical values may serve as an

anchor for participants to provide their own ratings and opinions on the piece.

Factors such as confidence in one’s judgments and elimination of ambiguity can

allow one to evaluate a critic’s review more objectively. According to processing fluency

theory, increased exposure to a similar stimulus lead to more confident judgments on that

stimulus, which allowed the target to process it more easily and pleasantly (Reber,

Schwarz & Winkielman, 2004). Increased exposure to a piece of art, specifically music,

can facilitate fluency and more confident judgments of the work, which we can theorize

will allow people to create evaluative judgments, with less influence of expert opinions.

All art can be evaluated objectively and subjectively. I theorize that confident

evaluations of art tend to be more subjective, while ambiguity in one’s judgments will

lead people to adhere to critics’ reviews and take a more objective perspective on the

piece of work. It has been shown that when judges receive more information, their

4


confidence increases more than their accuracy, which produces substantial confidence-

accuracy discrepancies (Tsai, 2008). One’s confidence in their judgments in music or

another art form may lead to less accurate evaluations on the objective qualities of the art,

however because appreciation of art is such a personal endeavor, it can be argued that

more emphasis should be placed on the subjective qualities of art. I believe that the

influence of experts’ reviews will influence people’s objective evaluations of art, but

confidence in ones judgments, although shown to decrease accuracy, will mitigate the

influence of the critics and allow people to make more subjective judgments.

My study intends to show how confidence in one’s judgments concerning music

affects the level of influence that expert opinions have on people’s personal evaluations

on music. By providing participants with a fake reviews of songs from supposed “music

critics” from a credible music reviewing website and then allowing them to listen to the

song and evaluate it themselves, we hypothesized that their confidence in music appraisal

would predict the amount of influence that the fake reviews from critics had on their

evaluations of the pieces themselves. If expert opinions can influence people’s views on

social issues, (Chan,1999), their opinions on music, a much less personal concept, may be

significantly swayed by critics’ reviews. Additionally, since process fluency with a piece

of art increases confidence in judgments (Reber et al., 2004), and confidence in one’s

judgments leads to less accurate judgments of objective qualities allowing subjectivity to

flourish (Tsai, 2008), I hypothesize that those who are confident in their ability to

evaluate music will be less influenced by expert reviews and more inclined to make their

judgments independently and subjectively.

5


Method

Participants

A total of 88 participants were a part of this study, 47 of them male and 40 of them

female. One participant recorder their gender as other. All participants were students at

Bates College between the ages of (18-22). The participants were selected using an

internet survey, and all students who responded with legitimate data were included in the

study. The survey was distributed through various email list-serves at Bates College.

Students who reported that they had heard the songs used in our study before were

excluded.

Materials and Procedure

The experiment consisted of one survey, which participants completed in one

sitting. This was a 2x2 mixed design, with the first independent variable being confidence

in music appraisal in the selected independent group. The second independent variable

was the type of anchor; high (positive review) or low (negative review). Participants were

randomly assigned to either condition for the first song, while an opposite anchor

accompanied the second song. The dependent variable was the participant’s review

rating. The survey was constructed using Qualtrics survey software. Participants were

first asked to rate their confidence in their ability to appraise music on a scale of 1-10

with one being “Not confident at all” and ten being “Extremely confident.” After that,

they were instructed to read a fake review for a song, listen to that song, and give the

song a numerical rating out of 10 based on their opinion of its quality. They were asked

to repeat these steps for a second song. Each participant saw a strongly positive review

6


for one of the songs, and a strongly negative review for the other song both with a brief

paragraph and a number rating out of 10 (see Appendix). All reviews were presented with

a fake reviewer’s name to give the illusion of validity. The songs used were “Pillow

Talk” by Wild Child and “Indiana” by Cymbals Eat Guitars. The order of the two songs

and which one the participant saw a positive/ negative review for was selected using

random assignment, with four possible conditions (Negative review for “Pillow Talk”

then positive review for “Indiana”; positive review for “Pillow Talk” then negative

review for “Indiana”; negative review for “Indiana” then positive review for “Pillow

Talk”; positive review for “Indiana” then negative review for “Pillow Talk”). After

reading reviews, listening to and rating both songs, participants were asked general

demographic information regarding their gender, age and race. Participants were also

asked what type of music they generally listened to. After these demographic questions,

the survey was complete.

Results

In the analysis I ran a 2x2 mixed groups ANOVA with the independent group

variable of confidence level (top 75% quartile vs. bottom 25% quartile) in music

appraisal and the repeated measure variable as the type of review (positive vs. negative).

There was a significant main effect of type of review F(1,25) = 7.22 p = .013.

People given positive reviews (M = 6.74, SD = .395), rated music significantly higher

than those who were given negative reviews (M = 5.42, SD = .355) indicating that they

were affected by the anchor.

There was no significant main effect for the confidence on music appraisal F(1,25)

= .617 p = .440. Participants in the high quartile of confidence (M = 6.30, SD = .452), had

7


reported ratings that did not significantly differ from the low quartile of confidence (M =

5.85, SD = .346).

There was no significant interaction between confidence and the type of review

F(1,25) = .028 p = .868.

Discussion

Results indicated that the type of review the participants were given produced a

significant effect on their independent evaluations of the piece of music. This reproduces

Tversky and Kahneman’s (1992) findings on anchoring and adjustment. The participants

held on to the anchor (the numerical critical review) and allowed it to significantly

influence their independent view on the song. Participants shown a negative review of a

song rated that song lower than they would if it was accompanied by a positive review.

This was true collapsing across both songs.

I hypothesized that high confidence in one’s ability to evaluate music would allow

participants to make more independent judgments on music and therefore stray away

from the anchor more than those with lower confidence in music. My experiment failed

to show that confidence in one’s ability to evaluate music had a significant effect on how

distant one’s evaluations of music were away from the critics’ reviews.

Although I did not find that confidence had a significant effect on how

independently one evaluates music, the reproduction of the anchoring and adjustment

heuristic in the context of music appraisal is a very significant finding. This provides

concrete evidence that peoples’ “independent” opinions of a piece of music are not as

independent as they think, and their ideas are shaped by the critics’ reviews. Participants

who were told that a piece of music was rated as a 9 out of 10 rated it closer to a 9 (6.74)

8


then when they were told the piece was a 3 out of 10 (5.42). These results replicate

similar findings of anchoring and adjustment found in product pricing (Elaad, Sayag,

Exer, 2010). These numerical portions of the reviews provide an anchor, while the

qualitative content of the review provides context to support these numerical claims.

Music reviewing websites ought to be aware of these findings, because every time

they produce a subjective review, especially accompanied with a numerical rating, they

change people’s opinions of the piece of music. Music reviewing websites such as

Pitchfork provide short reviews accompanied by numerical rating in a similar manner to

my study. The results conclude that reviews in these formats can completely alter one’s

perception on a piece of music. Reading these reviews does not allow one to

independently evaluate the music without the influence from the anchor of the critics’

review. This data should urge us to scrutinize the critics and not take everything they say

at face value. Perhaps if critics avoid numerical ratings in their reviews, people will

assess the qualitative content of the review without significant influence.

When I tried to find a variable to eliminate these biases, I thought confidence in

music appraisal would produce more independent judgments; however the effects were

negligible. My study ran a high-low quartile analysis and made two separate groups for

participants with significantly low self-reported music appraisal confidence (scores below

4) and a group with significantly high self-reported music appraisal confidence (scores

above 8). The problem with this is that 48 of the 75 participants reported their confidence

in music appraisal as either a 6,7 or 8. These participants where not in the high or low

quartile and were not included in the analysis leaving 17 participants in the low quartile

and 10 participants in the high quartile as the sole analysis groups.

9


It would greatly benefit the study to have a greater distribution of confidence in

appraisal for music. Perhaps I should have asked the participants to report if their

confidence to evaluate music was simply either high or low. Additionally, I could have

found a more intuitive way to evaluate confidence in music appraisal, rather than a

simple self-reporting measure where participants tended to choose similar scores. I could

have constructed a questionnaire to gauge confidence in music appraisal, collapsing

across many aspects of confidence instead of self-reporting. For example, I could have

asked participants to report any prior knowledge of music theory, how many years

they’ve played an instrument, or how often they listen to music weekly. Combining those

variables could help create a confidence in music appraisal measure that could have

substituted for a self-reported music confidence measure.

Many participants might have difficulty reporting their confidence in their ability

to evaluate music as very low score due to the social desirability bias (Persson & Solevid,

2013). We often like to think that we are confident in our judgments. Because many

people listen to music frequently, we like to believe that we have a substantial basis for

our confidence in judgments on music. Because of this bias, the experiment contained a

significantly low frequency of participants reporting low confidence in music appraisal.

In this experiment I found evidence and replicated Tversky and Kahneman’s

(1992) anchoring and adjustment heuristic in the context of critics’ music reviews.

Confidence in music appraisal did not provide evidence for mitigating these effects,

possibly due to participants’ difficulty in gauging their confidence in music appraisal on a

1-10 scale. Perhaps there is another mediating variable that will allow people to make

more independent evaluations of music and loosen their grip on the critical reviewers’

10


anchors. Until that variable is uncovered we must be conscious in avoiding influence

from critics’ reviews when forming our own opinions.

11


References

Chan, S. S. (1999, September). Media use of expert sources and Its effects on public

opinion. Dissertation Abstracts International Section A, 60, 0577.

Elaad, E., Sayag, N., & Ezer, A. (2010). Effects of anchoring and adjustment in the

evaluation of product pricing. Psychological Reports, 107 (1), 58-60. doi: 10.2466

/ 01.PR0.107.4.58-60

Persson, M., & Solevid, M. (2014). Measuring political participation social desirability

bias-Testing in a web-survey experiment. International Journal Of Public Opinion

Research, 26 (1), 98-112. doi: 10.1093 / ijpor / edt002

Reber, R., Schwarz, N., & Winkielman, P. (2004). Processing Fluency and Aesthetic

Pleasure: Is Beauty in the Perceiver's Processing Experience ?. Personality And

Social Psychology Review, 8 (4), 364-382. doi: 10.1207 / s15327957pspr0804_3

Tsai, C. I. Klayman, J., & Hastie, R. (2008). Effects of amount of information on

judgment accuracy and confidence. Organizational Behavior and Human Decision

Processes, 107 (2), 97-105. doi: 10.1016 / j.obhdp.2008.01.005

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases.

Science, 185 (4157), 1124-1131. doi: 10.1126 / science.185.4157.1124

Tversky, A., Kahneman, D., Gentner, D., Collins, A., Fischhoff, B. Hoch, SJ, &

Loewenstein, GF (1992). Aspects and inferential judgment under uncertainty. In

TO Nelson (Ed.), Metacognition: Core readings (pp 377-436.). Needham Heights,

MA, US: Allyn & Bacon.

12


Appendix

Reviews used in the Study:

By the end of Cymbals Eat Guitars’ overstuffed and overcomplicated album Why There Are Mountains, you might find yourself nursing a migraine. While the band certainly sets their sights high, they ultimately fall short in a heap of disharmony that is only made worse by the whiney, shrill voice of their lead singer. The lack of form feels tiresome rather than innovative and rather than ponder what psychedelic pseudo-epiphanies these guys might be having, you’ll be more likely to find that you just don’t care.3/10 Bruce Eakin, Rolling Stone

Cymbals Eat Guitars achieves a disjointed masterpiece with this album; the guitar reverb and jangling piano create an intense atmosphere rich with sorrow and ecstasy alike. Lead Singer Joseph D'agostino’s voice switches between a scream and a whisper at a dizzying pace as the band reaches more crescendos than you might think possible within a 3-minute song. This record may overreach itself at times, but the sheer power of the orchestration accompanied by clever and unique hooks certainly make up for it.9/10 Bruce Eakin, Rolling Stone

“Pillow Talk” perpetuates the stale melodrama that that is constant throughout indie rock and really brings nothing new to the table. Sonically, the track is boring and repeats the same chord progressions and hopes that its “cuteness” will resonate with listeners. Kelsey Wilson (lead female vocals) sounds scratchy and simply out of practice on her harmonies. With simple melodies and lazy chord progressions, Wild Child inhabits the banal pseudo-pop realm that has plagued indie artists for decades. 3/10 Sam Friedhoff, New York Times

"Pillow Talk" boasts very solid production values and a maturity that one wouldn't expect from a first record. The album is incredibly versatile and works well as a record to sit down and listen to or as that mellow, acoustic record you have as background music while working or getting together with friends. Utilizing a baritone ukulele as the bulk of a fairly sparse orchestration, Wild Child achieves something that is both unique and entirely familiar. Their polished sound and elegantly simple melodies help to push indie pop in the right direction. 9/10 Sam Friedhoff, New York Times

Rich Content Editor...

13

CONFIDENCE AND EXPERT REVIEWS 14

Confidence in Music Judgements and the Influence of Expert Reviews

Science

Transcript of Confidence in Music Judgements and the Influence of Expert Reviews