2nd Detection / Segmentation ChallengeSegmentation Leaderboard (I) COCO AP (over all IoU) COCO AP...

2ndDetection / Segmentation Challenge

Yin Cui, Tsung-Yi Lin, Matteo Ruggero Ronchi, Genevieve Patterson

ImageNet and COCO Visual Recognition Challenges WorkshopSunday, October 9th, ECCV 2016

Workshop Organizers

Yin CuiCornell Tech

Matteo Ruggero RonchiCaltech

Genevieve PattersonBrown University

Michael MaireSerge BelongieLubomir BourdevRoss GirshickJames HaysPietro PeronaLarry ZitnickPiotr Dollár

Workshop Advisors:Deva RamananPietro PeronaMichael MaireLubomir BourdevSerge BelongieMatteo Ruggero RonchiGenevieve PattersonYin Cui

Award Committee:

Tsung-Yi LinCornell Tech

Outline

1. Download MS COCO Train / Val set

Participate in challenge?

No 3. Download MS COCO Test-Dev

3. Download MS COCO Test-Full

2. Develop the algorithm

4. Upload to CodaLab (unlimited)

4. Upload to CodaLab (5 times max)

Outline

No 3. Download MS COCO Test-Dev

• 80 object categories • 200k images• 1.2M instances (350k people)• Every instance segmented

Available for download atmscoco.org

COCO Dataset

• 80 object categories • 200k images• 1.2M instances (350k people)• Every instance segmented

Available for download atmscoco.org

COCO Dataset

• 106k people with keypoints

Available for download atmscoco.org/external

COCO 3rd Party Datasets

Outline

3. Download MS COCO Test-Dev

Outline

Shout-out to previous algorithms!

Outline

Challenges at ECCV 2016

/ 5410

1. Download MS COCO Train / Val set 2. Develop the algorithm

Outline

/ 5410

Outline

/ 5411

MS COCO Test Sets

/ 5411

The 2015/2016 MS COCO Test set consists of ~80k test images.

MS COCO Test Sets

/ 5411

Test-dev (development) Debugging, Validation and Ablation Studies. Allows unlimited submission to the evaluation server.

MS COCO Test Sets

/ 5411

Test-standard (publications) Used to score entries for the Public Leaderboard.

MS COCO Test Sets

/ 5411

Test-challenge (competitions) Used to score workshop competition.

MS COCO Test Sets

/ 5411

Test-challenge (competitions) Used to score workshop competition.

Test-reserve (security) Used to estimate overfitting. Scores on this set are never released.

MS COCO Test Sets

/ 5412

Outline

/ 5412

Outline

/ 5413

Evaluation Server Usage

Submissions to all test sets

/ 5414

Evaluation Metrics

/ 5415

Evaluation Metrics

• AP is averaged over multiple IoU values between 0.5 and 0.95.

Challenges Score: AP• More comprehensive metric than

the traditional AP at a fixed IoU value (0.5 for PASCAL).

• AP is averaged over instance size:• small (A < 32 x 32)• medium (32x 32 < A < 96 x 96)• large (A > 96 x 96)

Evaluation Metrics

A<32x32

32x32<A<96x96

A>96x96Other Scores: Size AP

/ 5417

Evaluation Metrics

Other Scores: AR

• Measures the maximum recall over a fixed number of detections allowed in the image of 1, 10, 100.

• AR is averaged over small (A < 32 x 32), medium (32x 32 < A < 96 x 96) and large (A > 96 x 96) instances of objects.

/ 5418

Evaluation Ambiguity

Which one is better?

/ 5418

Ground-Truth BBox

/ 5418

Detection BBoxGround-Truth BBox

/ 5418

IoU = 0.5 IoU = 0.7 IoU = 0.95

Detection BBoxGround-Truth BBox

/ 5419

COCO Challenges Results

/ 5420

Bounding Boxes Leaderboard (I)

COCO AP (over all IoU)

/ 5420

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

R-CNN*

(VGG-16)

/ 5420

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

R-CNN*

(VGG-16)

(*) Performance on Test-Dev

/ 5420

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

R-CNN*

(VGG-16)

/ 5420

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

R-CNN*

(VGG-16)

(*) Performance on Test-Dev (**) 2015 Winner

/ 5420

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

R-CNN*

(VGG-16)

+22% absolute+110% relative

(**) 2015 Winner

/ 5420

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

R-CNN*

(VGG-16)

+22% absolute+110% relative

(**) 2015 Winner

+4.2% absolute+11.2% relative

/ 5421

Segmentation Leaderboard (I)

/ 5421

MSRAG-R

MSRAVC**

/ 5421

MSRAG-R

MSRAVC**

(**) 2015 Winner

/ 5421

MSRAG-R

MSRAVC**

(**) 2015 Winner

/ 5421

MSRAG-R

MSRAVC**

(**) 2015 Winner

/ 5421

MSRAG-R

MSRAVC**

(**) 2015 Winner

/ 5421

MSRAG-R

MSRAVC**

COCO AP for segmentation winner trails the one for bbox detection by ~4%:

• Last year the gap was ~10%• Localization is harder

for segmentation

(**) 2015 Winner

/ 5422

Bounding Boxes Leaderboard (II)

Object Localization is improving

/ 5422

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

AP_50 AP_75

/ 5422

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

AP_50 AP_75

/ 5422

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

AP_50 AP_75

objects correctly detected but not well localized17% AP

/ 5422

MSRAVC**

Trimps-S

ine La

Cmu-a2-v

ellucid Wall

AP_50 AP_75

17% AP 19% AP

/ 5423

Segmentation Leaderboard (II)

Mask localization can improve

/ 5423

MSRAG-R

MSRAVC**

AP_50 AP_75

/ 5423

MSRAG-R

MSRAVC**

AP_50 AP_75

/ 5423

MSRAG-R

MSRAVC**

AP_50 AP_75

20% AP

/ 5424

Bounding Boxes vs Segmentation

Segmentation provides great bounding boxes!

/ 5424

(*) 2015 Winner

MSRA (seg

MSRAVC (bbox

AP AP_75 AP_50

/ 5424

(*) 2015 Winner

MSRA (seg

MSRAVC (bbox

AP AP_75 AP_50

/ 5424

(*) 2015 Winner

MSRA (seg

MSRAVC (bbox

AP AP_75 AP_50

/ 5424

• Results in 2nd place in the bbox challenge!

(*) 2015 Winner

MSRA (seg

MSRAVC (bbox

AP AP_75 AP_50

/ 5424

• Gap is about constant at multiple IoU values.

(*) 2015 Winner

MSRA (seg

MSRAVC (bbox

AP AP_75 AP_50

/ 5424

• Gap is about constant at multiple IoU values.

• Participate in Segmentation Challenge!

(*) 2015 Winner

MSRA (seg

MSRAVC (bbox

AP AP_75 AP_50

/ 5425

Performance Breakdown (I)

COCO AP varies across supercategories and size

/ 5425

Performance across teams improved on all supercategories

• Average AP increase of ~10%.• Average Standard Deviation

decrease of ~1%.

/ 5426

Performance Breakdown (II)

Impact of size on performance

2015 2016

medium large

/ 5426

2015 2016

medium large

/ 5426

2015 2016

medium large

/ 5426

2015 2016

medium large

+118%!!

/ 5427

Correlation between methods

/ 5427

How similarly do algorithms perform?

/ 5427

MSRAVC*

Bounding Boxes

(*) 2015 Winner

/ 5427

MSRAVC*

Bounding Boxes

0 % 80%AP

(*) 2015 Winner

/ 5427

MSRAVC*

Bounding Boxes

0 % 80%AP

(*) 2015 Winner

/ 5427

MSRAVC*

Bounding Boxes

0 % 80%AP

R2 = 0.98

(*) 2015 Winner

/ 5427

MSRAVC*

Bounding Boxes

0 % 80%AP

R2 = 0.98

(*) 2015 Winner

Segmentation

/ 5427

MSRAVC*

Bounding Boxes

0 % 80%AP

R2 = 0.98

(*) 2015 Winner

Segmentation

/ 5427

MSRAVC*

Bounding Boxes

0 % 80%AP

R2 = 0.98

(*) 2015 Winner

Segmentation

/ 5427

MSRAVC*

Bounding Boxes

0 % 80%AP

R2 = 0.98 R2 = 0.97

(*) 2015 Winner

Segmentation

/ 5428

Bounding Box Detection Errors

/ 5428

How similarly do top algorithms perform?

/ 5428

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Super-category FP removed

Category FP removed

Background FP removed

All errors are removed

0 0.2 0.4 0.6 0.8 1recall

overall-all-all

[.456] C75[.623] C50[.686] Loc[.700] Sim[.723] Oth[.925] BG[1.00] FN

/ 5428

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Category FP removed

0 0.2 0.4 0.6 0.8 1recall

overall-all-all

G-RMI MSRAVC*

0 0.2 0.4 0.6 0.8 1recall

overall-all-all

(*) 2015 Winner

/ 5429

Bounding Box Detection Errors (I)

/ 5429

What type of errors are algorithms doing?

/ 5429

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Category FP removed

/ 5429

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Category FP removed

G-RMI MSRAVC*

(*) 2015 Winner

/ 5429

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Category FP removed

0 0.2 0.4 0.6 0.8 1recall

person-person-all

0 0.2 0.4 0.6 0.8 1recall

person-person-all

G-RMI MSRAVC*

(*) 2015 Winner

/ 5430

Bounding Box Detection Errors (II)

/ 5430

What type of errors are algorithms doing?

/ 5430

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Category FP removed

/ 5430

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Category FP removed

G-RMI MSRAVC*

(*) 2015 Winner

/ 5430

0 0.2 0.4 0.6 0.8 1recall

overall-all-small

0 0.2 0.4 0.6 0.8 1recall

overall-all-small

AP @ IoU = [0.5; 0.75]

AP @ IoU = 0.1

Category FP removed

G-RMI MSRAVC*

(*) 2015 Winner

/ 5431

Summary of Findings

/ 5431

Summary of Findings

2016 Detection Challenge Take-aways

/ 5431

Summary of Findings

• MSRAVC 2015 set a very high bar for performance.

/ 5431

Summary of Findings

• MSRAVC 2015 set a very high bar for performance.• G-RMI imroved COCO AP by 4% absolute, 11% relative.

/ 5431

Summary of Findings

• MSRAVC 2015 set a very high bar for performance.• G-RMI imroved COCO AP by 4% absolute, 11% relative.• MSRA 2016 segmentation algorithm is great on bboxes.

/ 5431

Summary of Findings

• MSRAVC 2015 set a very high bar for performance.• G-RMI imroved COCO AP by 4% absolute, 11% relative.• MSRA 2016 segmentation algorithm is great on bboxes.• Performance on all classes has improved across entries.

/ 5431

Summary of Findings

• MSRAVC 2015 set a very high bar for performance.• G-RMI imroved COCO AP by 4% absolute, 11% relative.• MSRA 2016 segmentation algorithm is great on bboxes.• Performance on all classes has improved across entries.• Localization improved greatly in both challenges.

/ 5431

Summary of Findings

• MSRAVC 2015 set a very high bar for performance.• G-RMI imroved COCO AP by 4% absolute, 11% relative.• MSRA 2016 segmentation algorithm is great on bboxes.• Performance on all classes has improved across entries.• Localization improved greatly in both challenges.• High relative improvement on small object instances.

/ 5431

Summary of Findings

• MSRAVC 2015 set a very high bar for performance.• G-RMI imroved COCO AP by 4% absolute, 11% relative.• MSRA 2016 segmentation algorithm is great on bboxes.• Performance on all classes has improved across entries.• Localization improved greatly in both challenges.• High relative improvement on small object instances.• False negatives are reduced, thus better recall of teams.

/ 5432

Challenges Ranking

G-RMI 1st 2nd

MSRA - 1st

Trimps-Soushen 2nd -

Imagine Lab 3rd -

UofA 5th -

1026 - 3rd

Challenges RankingTeam BBox Segmentation

G-RMI 1st 2nd

MSRA - 1st

Trimps-Soushen 2nd -

Imagine Lab 3rd -

UofA 5th -

1026 - 3rd

Challenges Ranking

Invited Speakers:• G-RMI / Object Detection / (2:30pm - 2:45pm)• MSRA / Segmentation / (2:45pm - 3:00pm)

Team BBox Segmentation

2nd Detection / Segmentation ChallengeSegmentation Leaderboard (I) COCO AP (over all IoU) COCO AP...

Documents

Transcript of 2nd Detection / Segmentation ChallengeSegmentation Leaderboard (I) COCO AP (over all IoU) COCO AP...

Artbag Studio | Crystal containers of Frozen Facts | Debra ... · coco CHANEL . ABOUT coco CHANEL PARIS . ABOUT coco CHANEL PARIS . coco . coco . co o . coco . CHAN FL cnco coca .

Getting started with the COCO chemical process simulator COCO ...

THE COCO - s3.ap-southeast-2.amazonaws.com

MS COCO image and annotations COCO-Text annotations

Coco-Ethanol Production of the Hilltop Coco Products Enterprises-updated

DetectoRS: Detecting Objects With Recursive Feature ... · DetectoRS into the state-of-art HTC [8] on the challenging COCO dataset [51]. On COCO test-dev, we report box AP for object

Video Object Segmentation based on Pixel-level Annotated ... · Introduction The objective of Video Object Segmentation is ... or MS-COCO for image captioning and object instance

Abstract - arXiv.org e-Print archive · segmentation, we achieve 49.1 mask AP and 57.3 box AP, an improvement of +0.6 mask AP and +1.5 box AP over the previous state-of-the-art. We

Coco paper

@ coco coco CO LO o COCO coco co 00 n < 00 00 Zii ......coco coco CO LO o COCO coco co 00 n < 00 00 Zii CuCO LO CO COCO COCO co CUCUCUCU cu O COCO COCO CO o o LO O) CD o CO CO CO oo

Dense Segmentation-aware Descriptors1fetrulls,sanfeliu,fmorenog@iri.upc.edu, 2iasonas.kokkinos@ecp.fr Abstract In this work we exploit segmentation to construct ap-pearance descriptors

Dear COCO Beauties, A COCO Gift - Coco Skin Laser Healthcocoskinlaserhealth.com.au/wp-content/uploads/2014/12/Christmas... · Dear COCO Beauties, ... We have been raving about PicoSure

coco cambodia

Learning Instance Occlusion for Panoptic Segmentation · with occlusion head, we observe signiﬁcant gains in the overall performance on the COCO and Cityscapes panoptic segmentation

Panoster: End-to-end Panoptic Segmentation of LiDAR Point Clouds · 2020. 10. 30. · Our method Panoster extends semantic segmentation ap-proaches to panoptic segmentation. In this

Coco Sugar

Barclays CoCo

Coco presentation

Coco Spread

Coco chanel