Post on 01-Aug-2020
InteractivevisualizationofvolumetricdatawithWebGLinreal-time
MarioJ.Lorenzo
Problem
Renderingofvolumetricdatawiththefollowing…– Real-time– Realistic– Interactive
…supportedthroughaweb-browser
ProblemDomainBackground
MedicalMedicalimagingtechniquessuchasCTscan,MRI,andPETproducesetsofparallelslicesthatformvolumetricdataset.StorageandDistributionofmedicalimagesaremanagedbyaPictureArchivingandCommunicationSystems(PACS)
• Client–to-Serverarchitecture• Requiresspecializedsoftware
Congoteetal,2011
WeatherWeatherdataisalsovolumetric• Collectedbyusingradarthatscansavolume
surroundingit• Collectsvaluesinacircularorconicalscansurface• Dopplerradaralsosamplesphysicalvariables
(reflectivity,radialvelocityetc)
ProblemDomainMotivation
• ScientificfieldssuchasMedicalImagingandMeteorologyhavebeenpioneersin3Dvisualization
• VisualizationistheprimarytargetofanalysisbyRadiologistandMeteorologist– TheadventofCTandMRIscansprovidesslicesofimagesthatcanbeusedto
render3Dvolumes– 3Dvolumerenderingasabasisforvisualassessmentbyradiologistisstillnewand
notwidelyused.– Mayrequirere-trainingforhowradiologistsdiagnose
• Improvingfidelityanddimensionalityofthevisualizationcandirectlyimprovetheoverallaccuracyofthedecisionmaking.
BackgroundandPriorResearch
• VolumeRenderingseekstoprojecta3Dmodelinto2Dimage– Theabilitytointeractinreal-timegivestheillusionofa3Dimage
• PriortoWebGL,3Dvisualizationwasperformedthroughnativeapplicationsornon-standardplug-insinstalledinthewebbrowser– Requiresaccesstolocalhardware(i..eGPU)
• Mostlyrenderingpolygonmeshes– StraightforwardandsupportedbyGPUrasterrendering
Background:VolumeRending
LEVOY,M.1988.Displayofsurfacesfromvolumedata.IEEEComput.Graph.Appl.8,3,29–37.
Demonstratedthatvolume-renderingtechniquesproducedbetterimagesfromvolumetricdatathanusingconventionalgeometricprimitives.
MarcLevoy-consideredapioneerinvolumerendering;workedonvisualizingCTimaging
Levoy,1988
Hadwigeretal-VolumeRaycastingHADWIGER,M.,LJUNG,P.,SALAMA,C.R.,ANDROPINSKI,T.2009.Advancedilluminationtechniquesforgpu-basedvolumeraycasting.InACMSIGGRAPH2009Courses,ACM,1–166.• DescribesVolumeray-castingismostbasic,butmostflexiblevolumerendering
algorithm• Ray-castingasanumericalmethodforevaluatingvolumerenderingintegral
• Integralisbasedonintegratinglightinteractioneffectsalongaviewingrayontheopticalmodel
• Absorption–radiantenergyabsorbedoverdistance• AlphaBlending–combiningtranslucentforegroundcolorwithbackgroundcolor
Volumeissampledatregularintervalsbetweenstartingandendingpositions Hadwigeretal,2009
Emptyskippingandestablishingboundinggeometry
ProposedMethodCongote,J.,Segura,A.,Kabongo,L.,Moreno,A.,Posada,J.,&Ruiz,O.(2011,June).Interactivevisualizationofvolumetricdatawithwebgl inreal-time. InProceedingsofthe16thInternationalConferenceon3DWebTechnology(pp.137-146).ACM.
• ApproachreliesonLevoyandHadwigeretalworkonvolumerenderingalgorithms
• AdaptedVolumeRay-castingalgorithmsforWebGL
• Volumeinterpolationfrom2Dimageslices• Avoidsuseofserver-sideforremoterendering,enablingbetterscalability
forlarge-userenvironments(suchasinmedicalimaging)• Doesnotrequirepreprocessing
• Realismisaccomplishedthroughtheuseofblendingfunctions,transferfunctions,andPhongillumination
ComposingTexturefromImageslices• Constructa“texture”bytransformingindividualimageslicesintomosiac(matrix)ofimages• UsesinterpolationtofillinthegapsinbetweentheslicesIdentificationofRaycoordinates• Usesacubeastheboundinggeometryfortheimagebeingrendered• Rendersthecubeintosceneusingviewpoint• Firstrendersthecolorusingdepthfunctionchange
– 1stpasspresentsclosestregionofthecubetothecamera– 2ndpasspresentsfarregion
Raygeneration• Rayiscalculatedfromfrontregiontobackregion
– Colorinterpolationisused• Cartesianx,y,zcoordinatesdefinedwhereonedimensionisanimagesliceselectorandthe
othertwoisacoordinatewithinthatimage• RayisdividedbySsteps(numberofsamples)
– Computex,y,zforeachsampleinthevolume– Valueofpositionisinterpolatedfromthetexture
TransferFunction• Usevalueoftextureat(x,y,z)toidentifycolor• Usecolorincompositionfunction• Rayfinisheswhencompositionfunctionreaches
theendinthecube• Assignscolorvaluetoeachpixelalongtheray
MethodDetails
CTImageSlicesusedforVolumeRendering
Congoteetal,2011
PerformanceResults
DemoofCTScan3DRendering
Congote, J., Segura,A.,Kabongo, L.,Moreno,A.,Posada, J.,&Ruiz,O. (2011, June).Interactivevisualizationofvolumetricdatawithwebglinreal-time.InProceedingsofthe16thInternationalConferenceon3DWebTechnology(pp.137-146).ACM.http://demos.vicomtech.org/volren/
RelatedWork
Haehn,D.,Rannou,N.,Ahtam,B.,Grant,E.,&Pienaar,R.(2014).NeuroimaginginthebrowserusingtheXToolkit.FrontiersinNeuroinformatics,101.
Qiao,L.,Chen,X.,Zhang,Y.,Zhang,J.,Wu,Y.,Li,Y.,...&Qiu,M.(2017).AnHTML5-BasedPureWebsiteSolutionforRapidlyViewingandProcessingLarge-Scale3DMedicalVolumeReconstructiononMobileInternet.Internationaljournaloftelemedicineandapplications,2017.
• Combinesurfacerenderingcombinedwithremotevolumerendering
• WebGLbasedonclientsideandnativeprocessforvolumerendering
• Solutionfor3Dmedicalvolumereconstruction• Claimitis“purewebenvironment”,butreliesonremote
renderingcalled“master-slave”
• XTK–“TheXToolkit”• ClaimstobefirstWebGL/JavaScript-based
visualizationformedicalimaging• http://socr.umich.edu/HTML5/BrainViewer/
Review
• Latencyforloadingtheoriginalimages(whichcanbeverylargeinmedicalimaging)– Considercompressionoptimizations,videocompressionschemesmayhelp
eliminatetemporalredundancy
• Surfacerenderingfortexturescanbeveryhelpfultoradiologists• ExploreuseofThreeJSframeworktoabstractfromWebGL(suchasXTKand
ATM)• ReducedResolution
– MedicalImagesuselargesamplebitdepth(vscommon8bit)
• IssueswithreliabilityandconsistencyofJavascriptsetTimeout()functionhavinga10msoverheadpercall,causingperformanceissues
• Exposingadditionalcontrolsformanipulatinglighting,maybehelpful
• Ageneralconcernontheeffectsofinterpolationandothersimulationsthatcandistortormisleadanexpert
ReferencesReviewedPaper:Congote,J.,Segura,A.,Kabongo,L.,Moreno,A.,Posada,J.,&Ruiz,O.(2011,June).Interactivevisualizationofvolumetricdatawithwebglinreal-time.InProceedingsofthe16thInternationalConferenceon3DWebTechnology(pp.137-146).ACM.Congote,J.,Segura,A.,Kabongo,L.,Moreno,A.,Posada,J.,&Ruiz,O.–DemoofReal-TimeInteractiveVisualizationofVolumetricDatawithWebGL.http://demos.vicomtech.org/volren/.AdditionalpapersHADWIGER,M.,LJUNG,P.,SALAMA,C.R.,ANDROPINSKI,T.2009.Advancedilluminationtechniquesforgpu-basedvolumeraycasting.InACMSIGGRAPH2009Courses,ACM,1–166.Haehn,D.,Rannou,N.,Ahtam,B.,Grant,E.,&Pienaar,R.(2014).NeuroimaginginthebrowserusingtheXToolkit.FrontiersinNeuroinformatics,101.LEVOY,M.1988.Displayofsurfacesfromvolumedata.IEEEComput.Graph.Appl.8,3,29–37.Min,Q.,Wang,Z.,&Liu,N.(2018).AnEvaluationofHTML5andWebGLforMedicalImagingApplications.Journalofhealthcareengineering,2018.