CragCrunch: Insight Data Science Project

54
CragCrun.ch by Amy Skerry

Transcript of CragCrunch: Insight Data Science Project

CragCrun.chby Amy Skerry

Yosemite Valley: 1,276 Total Routes

Yosemite Valley: 1,276 Total Routes

Yosemite Valley: 1,276 Total Routes

My Memory is a bit sketchy on some of the pitches....feel free to offer corrections.... !!P1 11b slabby, bolt protected face crux starting off of a platform. !P2 11b Thin, left arching crack undercling/layback. Lots of fixed gear !P3 10a Slightly flaring thin corner, !P4 Big Traverse. !P5 10a Cool 10a arete/bulge move following by incredible 5.8 climbing, on atypically Yosemite sculpted jugs which turn out to be the trademark of the climb. Pay attention here, easy to get on the wrong crack. !Link Pitches 6&7 with a 70m rope(68m)....but save some gear for the anchor! !P6+P7 11c Long thin stemming corner with a oddly placed anchor at the crux. This will be the route crux for most. This anchor is not the end, keep going past the crux to the belay ledge above. With a 70m, you will have little rope to spare. !P8 11c A fantastic short but strenous 30' (?) overhanging hand and finger crack;

My Memory is a bit sketchy on some of the pitches....feel free to offer corrections.... !!P1 11b slabby, bolt protected face crux starting off of a platform. !P2 11b Thin, left arching crack undercling/layback. Lots of fixed gear !P3 10a Slightly flaring thin corner, !P4 Big Traverse. !P5 10a Cool 10a arete/bulge move following by incredible 5.8 climbing, on atypically Yosemite sculpted jugs which turn out to be the trademark of the climb. Pay attention here, easy to get on the wrong crack. !Link Pitches 6&7 with a 70m rope(68m)....but save some gear for the anchor! !P6+P7 11c Long thin stemming corner with a oddly placed anchor at the crux. This will be the route crux for most. This anchor is not the end, keep going past the crux to the belay ledge above. With a 70m, you will have little rope to spare. !P8 11c A fantastic short but strenous 30' (?) overhanging hand and finger crack;

My Memory is a bit sketchy on some of the pitches....feel free to offer corrections.... !!P1 11b slabby, bolt protected face crux starting off of a platform. !P2 11b Thin, left arching crack undercling/layback. Lots of fixed gear !P3 10a Slightly flaring thin corner, !P4 Big Traverse. !P5 10a Cool 10a arete/bulge move following by incredible 5.8 climbing, on atypically Yosemite sculpted jugs which turn out to be the trademark of the climb. Pay attention here, easy to get on the wrong crack. !Link Pitches 6&7 with a 70m rope(68m)....but save some gear for the anchor! !P6+P7 11c Long thin stemming corner with a oddly placed anchor at the crux. This will be the route crux for most. This anchor is not the end, keep going past the crux to the belay ledge above. With a 70m, you will have little rope to spare. !P8 11c A fantastic short but strenous 30' (?) overhanging hand and finger crack;

cragcrun.ch

1) climb attributes (e.g. # of pitches) & unstructured text

2) individual user ratings 3) overall popularity

Model features:

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations

scraped >100,000 climbs, ~5000 unique users (w/ ratings,

grades, comments) from MountainProject.com

Climb Similarities

• Random Forest Classifier • Feature selection for final

algorithm based on tf-idf and RF feature importances

Product

1) climb attributes (e.g. # of pitches) & unstructured text

2) individual user ratings 3) overall popularity

Model features:

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations

scraped >100,000 climbs, ~5000 unique users (w/ ratings,

grades, comments) from MountainProject.com

Climb Similarities

• Random Forest Classifier • Feature selection for final

algorithm based on tf-idf and RF feature importances

Product

1) climb attributes (e.g. # of pitches) & unstructured text

2) individual user ratings 3) overall popularity

Model features:

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations

scraped >100,000 climbs, ~5000 unique users (w/ ratings,

grades, comments) from MountainProject.com

Climb Similarities

• Random Forest Classifier • Feature selection for final

algorithm based on tf-idf and RF feature importances

Product

1) climb attributes (e.g. # of pitches) & unstructured text

2) individual user ratings 3) overall popularity

Model features:

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations

scraped >100,000 climbs, ~5000 unique users (w/ ratings,

grades, comments) from MountainProject.com

Climb Similarities

• Random Forest Classifier • Feature selection for final

algorithm based on tf-idf and RF feature importances

Product

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations Climb Similarities

Product

Personalized Recommendations

• flexibly identify candidates based on user-specified

constraints

• RF predicted ratings used to rank climbs based on

user’s personal preference history

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations Climb Similarities

Product

Validation

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations Climb Similarities

Product

Validation

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

true

posi

tive

rate

false positive rate

Data Acquisition + Storage

Exploration & Feature Processing

Predicting Individual User Ratings

User Recommendations Climb Similarities

Product

final model classifies individual user four-star ratings of different climbs

with 70.4% accuracy

reliably above chance and outperforms model based on

average popularity alone (56%)

Validation

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

true

posi

tive

rate

false positive rate

What features or dimensions of climbs determine individual differences in preferences?

What features or dimensions of climbs determine individual differences in preferences?

What features or dimensions of climbs determine individual differences in preferences?

• to better understand the model and diagnose diverse preferences

What features or dimensions of climbs determine individual differences in preferences?

• to better understand the model and diagnose diverse preferences• to integrate new users (with no rating history) by mapping them into a

reduced space of features

What features or dimensions of climbs determine individual differences in preferences?

• to better understand the model and diagnose diverse preferences• to integrate new users (with no rating history) by mapping them into a

reduced space of features

Brain & Cognitive Sciences MIT

Harvard University Ph.D spring 2015

Yale University, 2010

CragCrun.chby Amy Skerry

Limitations

Limitations

1. Didn't take into account variability in grading standards across region

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas• Ideally, a personalized model can go beyond recorded/consensus grades when

suggesting skill-appropriate climbs

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas• Ideally, a personalized model can go beyond recorded/consensus grades when

suggesting skill-appropriate climbs• Unfortunately, it is difficult to estimate the region/area-level differences in grading

standards from the data, because some areas are more stiff vs. soft in the grading, but some areas also contain easier vs. harder climbs. Thus, a simple approach like normalizing by average difficulty of the region would not suffice

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas• Ideally, a personalized model can go beyond recorded/consensus grades when

suggesting skill-appropriate climbs• Unfortunately, it is difficult to estimate the region/area-level differences in grading

standards from the data, because some areas are more stiff vs. soft in the grading, but some areas also contain easier vs. harder climbs. Thus, a simple approach like normalizing by average difficulty of the region would not suffice

2. Didn't take into account relative difficulty assessment as a signal

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas• Ideally, a personalized model can go beyond recorded/consensus grades when

suggesting skill-appropriate climbs• Unfortunately, it is difficult to estimate the region/area-level differences in grading

standards from the data, because some areas are more stiff vs. soft in the grading, but some areas also contain easier vs. harder climbs. Thus, a simple approach like normalizing by average difficulty of the region would not suffice

2. Didn't take into account relative difficulty assessment as a signal• A major factor in enjoyment of a climb is fit to the climber's skill set (i.e. if there is

a climb rated 5.12 that is all crimps and I find it easy, I'll probably enjoy that climb more than a climb that is rated 5.10d but very overhung and difficult for me

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas• Ideally, a personalized model can go beyond recorded/consensus grades when

suggesting skill-appropriate climbs• Unfortunately, it is difficult to estimate the region/area-level differences in grading

standards from the data, because some areas are more stiff vs. soft in the grading, but some areas also contain easier vs. harder climbs. Thus, a simple approach like normalizing by average difficulty of the region would not suffice

2. Didn't take into account relative difficulty assessment as a signal• A major factor in enjoyment of a climb is fit to the climber's skill set (i.e. if there is

a climb rated 5.12 that is all crimps and I find it easy, I'll probably enjoy that climb more than a climb that is rated 5.10d but very overhung and difficult for me

• In principle, we should be able to use the perceived difficulty of a climb to an individual user as another signal relating to the climber's skills and preferences

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas• Ideally, a personalized model can go beyond recorded/consensus grades when

suggesting skill-appropriate climbs• Unfortunately, it is difficult to estimate the region/area-level differences in grading

standards from the data, because some areas are more stiff vs. soft in the grading, but some areas also contain easier vs. harder climbs. Thus, a simple approach like normalizing by average difficulty of the region would not suffice

2. Didn't take into account relative difficulty assessment as a signal• A major factor in enjoyment of a climb is fit to the climber's skill set (i.e. if there is

a climb rated 5.12 that is all crimps and I find it easy, I'll probably enjoy that climb more than a climb that is rated 5.10d but very overhung and difficult for me

• In principle, we should be able to use the perceived difficulty of a climb to an individual user as another signal relating to the climber's skills and preferences

3. Popularity is weighted equally

Limitations

1. Didn't take into account variability in grading standards across region• The same grade can refer to very different difficulty levels across different areas• Ideally, a personalized model can go beyond recorded/consensus grades when

suggesting skill-appropriate climbs• Unfortunately, it is difficult to estimate the region/area-level differences in grading

standards from the data, because some areas are more stiff vs. soft in the grading, but some areas also contain easier vs. harder climbs. Thus, a simple approach like normalizing by average difficulty of the region would not suffice

2. Didn't take into account relative difficulty assessment as a signal• A major factor in enjoyment of a climb is fit to the climber's skill set (i.e. if there is

a climb rated 5.12 that is all crimps and I find it easy, I'll probably enjoy that climb more than a climb that is rated 5.10d but very overhung and difficult for me

• In principle, we should be able to use the perceived difficulty of a climb to an individual user as another signal relating to the climber's skills and preferences

3. Popularity is weighted equally• Overall popularity should be weighted in a graded fashion depending on the

amount of personalized information available for a given user

What features determine route preferences?bolt, climb, right, belay, route, ledge, corner, anchor, climbing, foot, face, tree, gear,

rope, continue, rock, wall, facing, follow, start, small, past, base, trail, summit, section, large, cam, easy, rappel, rap, gully, dihedral, straight, arete, class, short, reach, finger,

head, dike, camalot, pro, step, obvious, lead, block, approach, fixed, wide, easier, west, clip, ramp, descent, lower, gain, south, double, rack, bolts, loose, north, buttress,

east, near, second, way, finish, angle, scramble, walk, protected, ground, harder, directly, climber, single, nice, set, bit, standard, pull, possible, feet, horizontal,

protection, low, climbed, clean, big, end, long, line, passing, couple, edge, cliff, sustained, higher, variation, final, hold, hard, area, difficult, descend, road, lieback, seam, anchors, protect, upper, huge, bring, pitches, sloping, cams, slightly, leaning,

awkward, pocket, high, runout, cool, formation, downclimb, use, using, direct, grade, vertical, slings, quality, classic, fun, mantle, main, shallow, good, piece, ll, guide,

place, super, middle, chain, trad, run, located, solid, excellent, fa, center, nuts, easily, pretty, probably, starts, great, leading, positive, point, follows, moves, slanting,

medium, far, shared, just, draws, somewhat, lots, crag, lip, need, maybe, rail, half, hands, look, fall, sure

flakes, hand cracks, tricky moves, underclings, slabs, clipping, technical climbs,

bouldery moves, bulge, steep climbs, juggy holds, chimney climbs, crack climbs, tough

cruxes, roofs, flakes, traverses, multi-pitch climbs,

crimps

PredictedTr

ue

flakes, hand cracks, tricky moves, underclings, slabs, clipping, technical climbs,

bouldery moves, bulge, steep climbs, juggy holds, chimney climbs, crack climbs, tough

cruxes, roofs, flakes, traverses, multi-pitch climbs,

crimps

Algorithm

Algorithm

• Analysis decisions/hyper-parameters determined based on exploratory analysis of New Hampshire climbs, final model and validation based on

California data

Algorithm

• Analysis decisions/hyper-parameters determined based on exploratory analysis of New Hampshire climbs, final model and validation based on

California data

Algorithm

• Analysis decisions/hyper-parameters determined based on exploratory analysis of New Hampshire climbs, final model and validation based on

California data

• For each individual user, trained a single model based on feature vectors derived from 3 sources: climb attributes (mostly a priori term frequencies from unstructured text), overall popularity, and individual preferences from

other climbers (binarized).

Algorithm

• Analysis decisions/hyper-parameters determined based on exploratory analysis of New Hampshire climbs, final model and validation based on

California data

• For each individual user, trained a single model based on feature vectors derived from 3 sources: climb attributes (mostly a priori term frequencies from unstructured text), overall popularity, and individual preferences from

other climbers (binarized).

Algorithm

• Analysis decisions/hyper-parameters determined based on exploratory analysis of New Hampshire climbs, final model and validation based on

California data

• For each individual user, trained a single model based on feature vectors derived from 3 sources: climb attributes (mostly a priori term frequencies from unstructured text), overall popularity, and individual preferences from

other climbers (binarized).

• Random Forest Classifer (k=80) used to classify individual star ratings. Recommendations generated by applying pre-trained model to new candidates based on constraints (e.g. area) specified by the user

Algorithm

• Analysis decisions/hyper-parameters determined based on exploratory analysis of New Hampshire climbs, final model and validation based on

California data

• For each individual user, trained a single model based on feature vectors derived from 3 sources: climb attributes (mostly a priori term frequencies from unstructured text), overall popularity, and individual preferences from

other climbers (binarized).

• Random Forest Classifer (k=80) used to classify individual star ratings. Recommendations generated by applying pre-trained model to new candidates based on constraints (e.g. area) specified by the user

Algorithm

• Analysis decisions/hyper-parameters determined based on exploratory analysis of New Hampshire climbs, final model and validation based on

California data

• For each individual user, trained a single model based on feature vectors derived from 3 sources: climb attributes (mostly a priori term frequencies from unstructured text), overall popularity, and individual preferences from

other climbers (binarized).

• Random Forest Classifer (k=80) used to classify individual star ratings. Recommendations generated by applying pre-trained model to new candidates based on constraints (e.g. area) specified by the user

• Features constructed and validated from personalization models also used to construct affinity matrix of all climbs (cosine similarity on text and user-based

feature vectors)

Validation0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Accuracy

Validation

• Leave-one-out cross validation procedure (on separate model for each

user)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7Accuracy

Validation

• Leave-one-out cross validation procedure (on separate model for each

user)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7Accuracy

Validation

• Leave-one-out cross validation procedure (on separate model for each

user)

• Model comparison: for each model, compute separate accuracy score for each user assess & statistical

significance of model differences by testing

whether gain of model is reliable across users

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7Accuracy

Validation

• Leave-one-out cross validation procedure (on separate model for each

user)

• Model comparison: for each model, compute separate accuracy score for each user assess & statistical

significance of model differences by testing

whether gain of model is reliable across users

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7Accuracy

Validation

• Leave-one-out cross validation procedure (on separate model for each

user)

• Model comparison: for each model, compute separate accuracy score for each user assess & statistical

significance of model differences by testing

whether gain of model is reliable across users

• Model performance increases as a

function of the data available for the individual user

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7Accuracy