Subjective and Objective Quality ... - Pavan Chennagiri ·...
Transcript of Subjective and Objective Quality ... - Pavan Chennagiri ·...
Subjective and Objective Quality Assessment of StitchedImages for Virtual Reality
Pavan C M
M Tech (Research) CandidateAdvisor: Dr. Rajiv Soundararajan
Department of Electrical Communication EngineeringIndian Institute of Science, Bangalore
Thesis DefenseOctober 26, 2018
Outline of the Talk
1 IntroductionProblem DefinitionChallengesPrior Work
2 Thesis OverviewDatabase and Subjective Quality AssessmentAutomatic Quality Assessment Algorithm
3 Experiments and Results4 Conclusion and Future Work
Pavan Stitched Image QA October 26, 2018 1 / 42
Outline of the Talk
1 IntroductionProblem DefinitionChallengesPrior Work
2 Thesis OverviewDatabase and Subjective Quality AssessmentAutomatic Quality Assessment Algorithm
3 Experiments and Results4 Conclusion and Future Work
Pavan Stitched Image QA October 26, 2018 1 / 42
Introduction
Virtual Reality (VR) - immersive experience through wide field of viewimages/videosVR applications - motion pictures, cinematic VR, immersivestorytelling etc.Head mounted displays (HMD) - freedom to choose desired viewsWide field of view images - stitching multiple images with overlappingviews
Pavan Stitched Image QA October 26, 2018 2 / 42
Introduction
Stitching algorithm - multiple stagesEach stage - influence on quality of stitched image
VR popularity - necessity for quality controlRelevance - benchmark, tune parameters and compare variousstitching algorithms.
Pavan Stitched Image QA October 26, 2018 3 / 42
Problem Statement
StitchingAlgorithm
Quality?
Constituent Images Distorted Stitched Image
Pavan Stitched Image QA October 26, 2018 4 / 42
Challenges
Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature points
Color distortion - images with different exposure levelsBlur and geometric distortion - improper blending of multiple images
Ghosting Blur Color Distortion Geometric
Stitching induced distortions - specific to panoramic images
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Ghosting
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Blur
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature pointsColor distortion - images with different exposure levels
Blur and geometric distortion - improper blending of multiple images
Ghosting Blur Color Distortion Geometric
Stitching induced distortions - specific to panoramic images
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Color Distortion
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature pointsColor distortion - images with different exposure levelsGeometric distortion - improper blending of multiple images
Ghosting Blur Color Distortion Geometric
Stitching induced distortions - specific to panoramic images
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Geometric
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature pointsColor distortion - images with different exposure levelsGeometric distortion - improper blending of multiple images
Ghosting Blur Color Distortion Geometric
Stitching induced distortions - specific to stitched images
Pavan Stitched Image QA October 26, 2018 5 / 42
Challenges
Absence of reference stitched images - not full reference qualityassessment
Constituent images - reference information
UnknownStitchingAlgorithm
Quality?
Constituent Images Distorted Stitched Image
Problem Setup
Assumption - access to individual and stitched images, no knowledge ofstitching algorithm
Pavan Stitched Image QA October 26, 2018 6 / 42
Challenges
Absence of reference stitched images - not full reference qualityassessmentConstituent images - reference information
UnknownStitchingAlgorithm
Quality?
Constituent Images Distorted Stitched Image
Problem Setup
Assumptions - access to individual and stitched images, no knowledge ofstitching algorithm
Pavan Stitched Image QA October 26, 2018 6 / 42
Problem Setup - Assumption
Absence of reference stitched images - not full reference qualityassessmentConstituent images - reference information
StitchingAlgorithm
Quality?
Constituent Images Distorted Stitched Image
Problem Setup
Assumptions - access to individual and stitched images, no knowledge ofstitching algorithm
Pavan Stitched Image QA October 26, 2018 6 / 42
Prior Art - No Reference Quality Assessment (QA)
No Reference (NR) quality assessment - rich literature and widelystudiedNatural Scene Statistics (NSS) - DIIVINE[Moorthy2011], BRISQUE[Mittal2012], NIQE [Mittal2013]
StitchingAlgorithm
Blind QA
Constituent Images Distorted Stitched Image
Problem Setup
Existing QA - do not address types of distortions observed in stitchedimages
Pavan Stitched Image QA October 26, 2018 7 / 42
Prior Art - QA in Stitched Images
[WeiXu2010] - evaluates color similarity and structural similarityRestrictive model - uses pointwise comparison, can be inaccurate
[Qureshi2012] - computes color and structural similarity in overlappingregions
Extension of [WeiXu2010] - uses high pass content in overlappingregion for structural similarity
Above algorithms - not evaluated on subjective database
[WeiXu2010] method [Qureshi2012] method
Pavan Stitched Image QA October 26, 2018 8 / 42
Contributions
Stitched image quality assessment databaseStitched images captured across diverse scenesSubjective evaluation - perception of distortions
Objective quality assessmentNatural scene statistics model
Bivariate statistics - increased correlations due to distortions
Correlates well with human perception
Pavan Stitched Image QA October 26, 2018 9 / 42
Outline of the Talk
1 IntroductionProblem DefinitionChallengesPrior Work
2 Thesis OverviewDatabase and Subjective Quality AssessmentAutomatic Quality Assessment Algorithm
3 Experiments and Results4 Conclusion and Future Work
Pavan Stitched Image QA October 26, 2018 10 / 42
Dataset
Stitched image quality assessment databaseImages from 26 scenes - buildings, gardens, indoor and public places264 stitched images - fusing multiple views with overlapping regionsStatic scenes - no object motion
Stitched image qualityChoice of algorithm for each stageParameter options associated with each block
Major impairments - ghosting, blur, geometric and color
Pavan Stitched Image QA October 26, 2018 11 / 42
Dataset
Stitched image quality assessment databaseImages from 26 scenes - buildings, gardens, indoor and public places264 stitched images - fusing multiple views with overlapping regionsStatic scenes - no object motion
Stitched image qualityChoice of algorithm for each stageParameter options associated with each block
Major impairments - ghosting, blur, geometric and color
Pavan Stitched Image QA October 26, 2018 11 / 42
Dataset
Stitched image quality assessment databaseImages from 26 scenes - buildings, gardens, indoor and public places264 stitched images - fusing multiple views with overlapping regionsStatic scenes - no object motion
Stitched image qualityChoice of algorithm for each stageParameter options associated with each block
Major impairments - ghosting, blur, geometric and color
Pavan Stitched Image QA October 26, 2018 11 / 42
Feature Detection, Matching and Outlier Removal
Image alignment - detecting keypoints in overlapping regionsDetection and matching- SIFTOutlier removal - Random Sample Consensus
Figure: Keypoint Detection Figure: Keypoint Matching
Figure: Outlier removal
Pavan Stitched Image QA October 26, 2018 12 / 42
Dataset
Stitched image quality assessment databaseImages from 26 scenes - buildings, gardens, indoor and public places264 stitched images - fusing multiple views with overlapping regionsStatic scenes - no object motion
Stitched image qualityChoice of algorithm for each stageParameter options associated with each block
Major impairments - ghosting, blur, geometric and color
Pavan Stitched Image QA October 26, 2018 12 / 42
Homography and Image Warping
Warp - transformation on co-ordinates for aligning overlapping regionsHomography - generalized transform, Direct Linear Transform (DLT)Moving DLT [Zaragoza2013] (MDLT) - patch level homographyShape preserving warp (SPHP) [Chang2014] - constrained homography
Homography MDLT SPHP
Homography
Pavan Stitched Image QA October 26, 2018 13 / 42
Dataset
Stitched image quality assessment databaseImages from 26 scenes - buildings, gardens, indoor and public places264 stitched images - fusing multiple views with overlapping regionsStatic scenes - no object motion
Stitched image qualityChoice of algorithm for each stageParameter options associated with each block
Major impairments - ghosting, blur, geometric and color
Pavan Stitched Image QA October 26, 2018 13 / 42
Image Blending
Blending - fusing multiple images to form single composite imageSmooth transition with no visible seams
Feathering - weighted averagingMultiband - Laplacian pyramid based blendingPoisson - gradient domain, optimizing the cost function
No Blending Multiband
Feathering with exposurecompensation
Poisson
Pavan Stitched Image QA October 26, 2018 14 / 42
Dataset
Stitched image quality assessment databaseImages from 26 scenes - buildings, gardens, indoor and public places264 stitched images - fusing multiple views with overlapping regionsStatic scenes - no object motion
Stitched image qualityChoice of algorithm for each stageParameter options associated with each block
Major impairments - ghosting, blur, geometric and color
Pavan Stitched Image QA October 26, 2018 14 / 42
Subjective Study
Single stimulus continuous quality assessmentRating - viewing images on a VR head mounted device (HMD)Images rated by 35 subjects across 3 sessionsProcessing of scores - Mean Opinion Score (MOS) for each imageafter rejecting outliers
0 10 20 30 40 50 60 70 80 90 100
MOS
0
5
10
15
20
25
30
35
40
Num
ber
of Im
ages
Distribution of MOS
MOS = 21.546
MOS = 65.238
Pavan Stitched Image QA October 26, 2018 15 / 42
Subjective Study
MOS = 21.546
MOS = 65.238
Pavan Stitched Image QA October 26, 2018 15 / 42
Outline of the Talk
1 IntroductionProblem DefinitionChallengesPrior Work
2 Thesis OverviewDatabase and Subjective Quality AssessmentAutomatic Quality Assessment Algorithm
3 Experiments and Results4 Conclusion and Future Work
Pavan Stitched Image QA October 26, 2018 16 / 42
Stitched Image Quality Evaluator (SIQE) Framework
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
NaturalScene Statistics
Model
NaturalScene Statistics
Model
PatchWeighting
PatchWeighting
Difference Regression
StitchedImage
ConstituentImages
ModelParameters
ModelParameters
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 17 / 42
Origin of Distortions
I1 I2 Original Ghosting
I Original Geometry
H
I(x) = (1− α(x))I1(x) + α(x)I2(Hx), where α(x) ∈ (0, 1)
I1(x) 6= I2(Hx) - combination of ghosting and blurPresence of additional edgesIncreased spatial correlation
Geometric distortion - presence of extraneous edges
Pavan Stitched Image QA October 26, 2018 18 / 42
Multi-Orientation Decomposition
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 18 / 42
Multi-Orientation Decomposition
Structural artifacts from ghosting and geometric - orientationdependentSteerable pyramid decomposition - 6 orientations, 2 scales for eachN ×N patch
Ghosting s60◦
1 s150◦
1
Geometry s30◦
1 s90◦
1
Pavan Stitched Image QA October 26, 2018 19 / 42
Divisive Normalization
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
fs1−12
f c1−12
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 20 / 42
Divisive Normalization
Divisive Normalization - y = y/p for subband coefficient y, with
p =√Y TC−1
U Y/N where CU is the covariance of neighborhoodaround y, N - number of neighbors
Contrast maskingReduce statistical dependencies - decorrelation
Previously shown to capture blur in [Li2009], [Moorthy2011]Besides blur, captures edges introduced due to distortions
Normalization factor p - measure of local variancep - higher values near edges
Pavan Stitched Image QA October 26, 2018 21 / 42
Divisive Normalization - Modeling
Ghosting and geometric distortions - presence of additional edgesDistribution of distorted patch - higher peak value at zero as y = y/p
Model - Generalized Gaussian Distribution (GGD)Features - GGD shape parameters
Original Ghosting Original Geometry
-8 -6 -4 -2 0 2 4 6 8
Subband Coefficients
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0.04
0.045
Num
ber
of C
oeffi
cien
ts (
Nor
mal
ized
)
original patchghosted patch
P (d150◦
1 (x, y)
-15 -10 -5 0 5 10 15
Subband Coefficients
0
0.01
0.02
0.03
0.04
0.05
0.06
Num
ber
of C
oeffi
cien
ts (
Nor
mal
ized
)
original patchgeometric distorted patch
P (d30◦
1 (x, y))
Pavan Stitched Image QA October 26, 2018 22 / 42
Bivariate Model
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
fs13−36
f c13−36
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 23 / 42
Bivariate Model
Capturing increased spatial correlation in ghosting - bivariatedistributionBivariate statistics - adjacent subband coefficientsP (sθα(x, y), sθα(x+ 1, y)) (with no divisive normalization)Distribution of ghosted patch - higher peak value than undistorteddistribution
Original Patch Ghosted PatchPavan Stitched Image QA October 26, 2018 24 / 42
Bivariate Model - Conditional Distribution Interpretation
Conditional Statistics - P (sθα(x+ 1, y)/sθα(x, y) ∈ (−δ, δ))
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
Subband Coefficients
0
0.05
0.1
0.15
0.2
0.25
0.3
Num
ber
of C
oeffi
cien
ts (
Nor
mal
ized
)
Conditional Histogram (original patch)Conditional Histogram (ghosted patch)
Deviation between conditional distributions - higher than marginaldistributions
Pavan Stitched Image QA October 26, 2018 25 / 42
Bivariate Model - Conditional Distribution Interpretation
Conditional Statistics - P (sθα(x+ 1, y)/sθα(x, y) ∈ (−δ, δ))
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
Subband Coefficients
0
0.05
0.1
0.15
0.2
0.25
0.3
Num
ber
of C
oeffi
cien
ts (
Nor
mal
ized
)
Conditional Histogram (original patch)Conditional Histogram (ghosted patch)Marginal Histogram (original patch)Marginal Histogram (ghosted patch)
Deviation between conditional distributions - higher than marginaldistributions
Pavan Stitched Image QA October 26, 2018 25 / 42
Bivariate Model - GMM and BGGD
Previous approaches - Bivariate GGD - let sθα(x+ 1, y) = a,sθα(x, y) = b, z = [a, b]T
f(z) = K exp (−(zTM−1z)β)
Bivariate Gaussian (BVG) - BGGD with β = 1
Our method - Gaussian mixture model
f(a, b) =
M∑i=1
ωiN (0,Σi)
Components - zero mean, distribution modeled by ωi,ΣiParameter estimation - Expectation Maximization (EM) algorithmGMM - QA
Pavan Stitched Image QA October 26, 2018 26 / 42
Bivariate Model - Model Comparison
Model comparisons - GMM and BGGD
Pristine
-2 -1 0 1 2
Subband Coefficients (Left)
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
Sub
band
Coe
ffici
ents
(R
ight
)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
10-3
(KL - 0.1477)
-2 -1 0 1 2
Subband Coefficients (Left)
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
Sub
band
Coe
ffici
ents
(R
ight
)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
10-3
(KL - 0.2167)
Distorted
-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5 0
1
2
10-4
(KL - 0.1547)
-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5 0
1
2
10-4
(KL - 0.24)
Empirical GMM BGGD
Pavan Stitched Image QA October 26, 2018 27 / 42
Bivariate Model - Model Comparison
Model comparisons - GMM and BVG
Pristine
-6 -4 -2 0 2 4 6
Subband Coefficients (Left)
-6
-4
-2
0
2
4
6
Su
bb
an
d C
oe
ffic
ien
ts (
Rig
ht)
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
10-5
(KL - 0.1535)
-6 -4 -2 0 2 4 6
Subband Coefficients (Left)
-6
-4
-2
0
2
4
6
Su
bb
an
d C
oe
ffic
ien
ts (
Rig
ht)
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
10-5
(KL - 0.4635)
Distorted
-5 0 5
Subband Coefficients (Left)
-5
-4
-3
-2
-1
0
1
2
3
4
5
Su
bb
an
d C
oe
ffic
ien
ts (
Rig
ht)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
10-4
(KL - 0.1547)
-5 0 5
Subband Coefficients (Left)
-5
-4
-3
-2
-1
0
1
2
3
4
5
Su
bb
an
d C
oe
ffic
ien
ts (
Rig
ht)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
10-4
(KL - 0.3823)
Empirical GMM BVG
Pavan Stitched Image QA October 26, 2018 28 / 42
Bivariate Model - Features
For sθα(x+ 1, y) = a, sθα(x, y) = b
f(a, b) =
M∑i=1
ωiN (0,Σi)
Covariance C =∑M
i=1 ωiΣi, eigen values of C as featuresHorizontal - sθα(x+ 1, y) = a, sθα(x, y) = bVertical - sθα(x, y + 1) = a, sθα(x, y) = b
Pavan Stitched Image QA October 26, 2018 29 / 42
Patch Weighting
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 30 / 42
Patch Weighting
All patches equal contribution?
e = 0.957 e = 0.459 e = 0.354 e = 0.106
Ghosting and blur artifacts - not perceived in smooth regionsGray level co-occurrence matrix (GLCM) [Haralick1973]Energy values of GLCM, e ∈ [0, 1], with e = 1 for constant imagePatch weight w = 1− eTextured patches - equal weights through non-linearity
g(w) = 1− exp
(−(wσ
)2)Pavan Stitched Image QA October 26, 2018 31 / 42
Model Summary
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 32 / 42
Model Summary
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
fs1−12
f c1−12
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 32 / 42
Model Summary
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
fs13−36
f c13−36
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 32 / 42
Model Summary
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
ModelParameters
ModelParameters
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 32 / 42
Prediction
SteerablePyramids
DivisiveNormalization(Marginal)
SteerablePyramids
DivisiveNormalization(Marginal)
BivariateStatistics
BivariateStatistics
DistributionModelling
DistributionModelling
PatchWeighting
PatchWeighting
DifferenceSupport
Vector Regression
StitchedImage
ConstituentImages
ModelParameters
f c1−36
ModelParameters
fs1−36
PredictedScore
Features from Stitched Image
Features from Constituent Images
Pavan Stitched Image QA October 26, 2018 32 / 42
Outline of the Talk
1 IntroductionProblem DefinitionChallengesPrior Work
2 Thesis OverviewDatabase and Subjective Quality AssessmentAutomatic Quality Assessment Algorithm
3 Experiments and Results4 Conclusion and Future Work
Pavan Stitched Image QA October 26, 2018 33 / 42
Correlation with Human Judgments
Database - 80% training and 20% testing with non overlap of scenesPerformance metric - Spearman rank order correlation coefficient(SROCC) and Pearson’s linear correlation coefficient (LCC)Median performance value - 1000 random train-test combinationsPerformance comparison - NR QA metrics BRISQUE [Mittal2012],NIQE [Mittal2012], DIIVINE [Moorthy2011]
SROCC LCCBRISQUE (trained on our database) 0.6224 0.5914
NIQE 0.1524 0.1051DIIVINE (trained on our database) 0.5706 0.5897
SIQE 0.8318 0.8380
Table: Median correlation across 1000iterations
BRISQUE NIQE DIIVINE SIQE
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure: Box plot of SROCCdistributions over 1000 trials
Pavan Stitched Image QA October 26, 2018 34 / 42
Significance of each conceptual feature
Each conceptual feature - tested in isolationFeatures only from stitched image - drop in performance, importanceof constituent images
NR setting - higher performance than NR-IQA methods
Feature SROCC LCCMarginal statistics model (f1−12) 0.7951 0.7934
Bivariate model (f13−36) 0.6825 0.6972Features from stitched image (fs1−36) 0.6524 0.6816
(when constituent image features are omitted)SIQE (fs1−36 and f c1−36) 0.8318 0.8380
Pavan Stitched Image QA October 26, 2018 35 / 42
Comparison with FR-QA Algorithms
FR metric - [WeiXu2010] and [Qureshi2012]Dependent on Pointwise correspondencesPerformance evaluation - 238 images, images obtained fromcommercial stitching algorithms ignored
Feature SROCC LCCXu (PSNR) 0.1795 0.2341Xu (SSIM) 0.3383 0.4077Qureshi 0.3238 0.3627SIQE 0.7848 0.8032
Pavan Stitched Image QA October 26, 2018 36 / 42
Analysis of Color Distorted Images
Scores for images with color distortion - close to images with little orno distortionColor distortion - less annoying when viewed on a HMD?HMD - 90◦ field of view, instances of non-appearance of colordistortion
(a) MOS = 61.7624 (b) MOS = 61.6771
(c) MOS = 59.7492 (d) MOS = 59.954
Pavan Stitched Image QA October 26, 2018 37 / 42
Outline of the Talk
1 IntroductionProblem DefinitionChallengesPrior Work
2 Thesis OverviewDatabase and Subjective Quality AssessmentAutomatic Quality Assessment Algorithm
3 Experiments and Results4 Conclusion and Future Work
Pavan Stitched Image QA October 26, 2018 38 / 42
Conclusion
Subjective quality assessmentStitched image quality databaseDistortions - ghosting, blur, geometric and colorSubjective evaluation on VR
Objective quality assessmentIndependent of underlying stitching algorithmCaptures stitching induced distortionsHigh correlation with human judgments, outperforms existing qualitymeasures
Path aheadModel - color distortion characterizationMethods to account for geometric shape changes and their relevance institched image QA
Pavan Stitched Image QA October 26, 2018 39 / 42
References
W. Xu and J. Mulligan, "Performance evaluation of color correctionapproaches for automatic multi-view image and video stitching," inIEEE Computer Vision and Pattern Recognition, June 2010, pp.263-270.
H. S. Qureshi, M. M. Khan, R. Hafiz, Y. Cho, and J. Cha,"Quantitative quality assessment of stitched panoramic images," IETImage Processing, vol. 6, no. 9, pp. 1348-1358, December 2012.
A. K. Moorthy and A. C. Bovik, "Blind image quality assessment:From scene statistics to perceptual quality," IEEE Trans. ImageProcess., vol. 20, no. 12, Dec. 2011.
A. Mittal, A. K. Moorthy, and A. C. Bovik, "No-reference imagequality assessment in the spatial domain," IEEE Trans. Image Process.,vol. 21, no. 12, pp. 4695-4708, Dec. 2012.
Pavan Stitched Image QA October 26, 2018 40 / 42
Acknowledgments
Advisor - RajivFunding agency - Department of Science and TechnologyVolunteers of subjective studyLabmates - Sameer, Biju and others
Pavan Stitched Image QA October 26, 2018 41 / 42
Thank You!
Pavan Stitched Image QA October 26, 2018 42 / 42