InteractivevisualizationofvolumetricdatawithWebGLinreal-time

MarioJ.Lorenzo

Problem

Renderingofvolumetricdatawiththefollowing…–  Real-time–  Realistic–  Interactive

…supportedthroughaweb-browser

ProblemDomainBackground

MedicalMedicalimagingtechniquessuchasCTscan,MRI,andPETproducesetsofparallelslicesthatformvolumetricdataset.StorageandDistributionofmedicalimagesaremanagedbyaPictureArchivingandCommunicationSystems(PACS)

•  Client–to-Serverarchitecture•  Requiresspecializedsoftware

Congoteetal,2011

WeatherWeatherdataisalsovolumetric•  Collectedbyusingradarthatscansavolume

surroundingit•  Collectsvaluesinacircularorconicalscansurface•  Dopplerradaralsosamplesphysicalvariables

(reflectivity,radialvelocityetc)

ProblemDomainMotivation

•  ScientificfieldssuchasMedicalImagingandMeteorologyhavebeenpioneersin3Dvisualization

•  VisualizationistheprimarytargetofanalysisbyRadiologistandMeteorologist–  TheadventofCTandMRIscansprovidesslicesofimagesthatcanbeusedto

render3Dvolumes–  3Dvolumerenderingasabasisforvisualassessmentbyradiologistisstillnewand

notwidelyused.–  Mayrequirere-trainingforhowradiologistsdiagnose

•  Improvingfidelityanddimensionalityofthevisualizationcandirectlyimprovetheoverallaccuracyofthedecisionmaking.

BackgroundandPriorResearch

•  VolumeRenderingseekstoprojecta3Dmodelinto2Dimage–  Theabilitytointeractinreal-timegivestheillusionofa3Dimage

•  PriortoWebGL,3Dvisualizationwasperformedthroughnativeapplicationsornon-standardplug-insinstalledinthewebbrowser–  Requiresaccesstolocalhardware(i..eGPU)

•  Mostlyrenderingpolygonmeshes–  StraightforwardandsupportedbyGPUrasterrendering

Background:VolumeRending

LEVOY,M.1988.Displayofsurfacesfromvolumedata.IEEEComput.Graph.Appl.8,3,29–37.

Demonstratedthatvolume-renderingtechniquesproducedbetterimagesfromvolumetricdatathanusingconventionalgeometricprimitives.

MarcLevoy-consideredapioneerinvolumerendering;workedonvisualizingCTimaging

Levoy,1988

Hadwigeretal-VolumeRaycastingHADWIGER,M.,LJUNG,P.,SALAMA,C.R.,ANDROPINSKI,T.2009.Advancedilluminationtechniquesforgpu-basedvolumeraycasting.InACMSIGGRAPH2009Courses,ACM,1–166.•  DescribesVolumeray-castingismostbasic,butmostflexiblevolumerendering

algorithm•  Ray-castingasanumericalmethodforevaluatingvolumerenderingintegral

•  Integralisbasedonintegratinglightinteractioneffectsalongaviewingrayontheopticalmodel

•  Absorption–radiantenergyabsorbedoverdistance•  AlphaBlending–combiningtranslucentforegroundcolorwithbackgroundcolor

Volumeissampledatregularintervalsbetweenstartingandendingpositions Hadwigeretal,2009

Emptyskippingandestablishingboundinggeometry

ProposedMethodCongote,J.,Segura,A.,Kabongo,L.,Moreno,A.,Posada,J.,&Ruiz,O.(2011,June).Interactivevisualizationofvolumetricdatawithwebgl inreal-time. InProceedingsofthe16thInternationalConferenceon3DWebTechnology(pp.137-146).ACM.

•  ApproachreliesonLevoyandHadwigeretalworkonvolumerenderingalgorithms

•  AdaptedVolumeRay-castingalgorithmsforWebGL

•  Volumeinterpolationfrom2Dimageslices•  Avoidsuseofserver-sideforremoterendering,enablingbetterscalability

forlarge-userenvironments(suchasinmedicalimaging)•  Doesnotrequirepreprocessing

•  Realismisaccomplishedthroughtheuseofblendingfunctions,transferfunctions,andPhongillumination

ComposingTexturefromImageslices•  Constructa“texture”bytransformingindividualimageslicesintomosiac(matrix)ofimages•  UsesinterpolationtofillinthegapsinbetweentheslicesIdentificationofRaycoordinates•  Usesacubeastheboundinggeometryfortheimagebeingrendered•  Rendersthecubeintosceneusingviewpoint•  Firstrendersthecolorusingdepthfunctionchange

–  1stpasspresentsclosestregionofthecubetothecamera–  2ndpasspresentsfarregion

Raygeneration•  Rayiscalculatedfromfrontregiontobackregion

–  Colorinterpolationisused•  Cartesianx,y,zcoordinatesdefinedwhereonedimensionisanimagesliceselectorandthe

othertwoisacoordinatewithinthatimage•  RayisdividedbySsteps(numberofsamples)

–  Computex,y,zforeachsampleinthevolume–  Valueofpositionisinterpolatedfromthetexture

TransferFunction•  Usevalueoftextureat(x,y,z)toidentifycolor•  Usecolorincompositionfunction•  Rayfinisheswhencompositionfunctionreaches

theendinthecube•  Assignscolorvaluetoeachpixelalongtheray

MethodDetails

CTImageSlicesusedforVolumeRendering

Congoteetal,2011

PerformanceResults

DemoofCTScan3DRendering

Congote, J., Segura,A.,Kabongo, L.,Moreno,A.,Posada, J.,&Ruiz,O. (2011, June).Interactivevisualizationofvolumetricdatawithwebglinreal-time.InProceedingsofthe16thInternationalConferenceon3DWebTechnology(pp.137-146).ACM.http://demos.vicomtech.org/volren/

RelatedWork

Haehn,D.,Rannou,N.,Ahtam,B.,Grant,E.,&Pienaar,R.(2014).NeuroimaginginthebrowserusingtheXToolkit.FrontiersinNeuroinformatics,101.

Qiao,L.,Chen,X.,Zhang,Y.,Zhang,J.,Wu,Y.,Li,Y.,...&Qiu,M.(2017).AnHTML5-BasedPureWebsiteSolutionforRapidlyViewingandProcessingLarge-Scale3DMedicalVolumeReconstructiononMobileInternet.Internationaljournaloftelemedicineandapplications,2017.

•  Combinesurfacerenderingcombinedwithremotevolumerendering

•  WebGLbasedonclientsideandnativeprocessforvolumerendering

•  Solutionfor3Dmedicalvolumereconstruction•  Claimitis“purewebenvironment”,butreliesonremote

renderingcalled“master-slave”

•  XTK–“TheXToolkit”•  ClaimstobefirstWebGL/JavaScript-based

visualizationformedicalimaging•  http://socr.umich.edu/HTML5/BrainViewer/

Review

•  Latencyforloadingtheoriginalimages(whichcanbeverylargeinmedicalimaging)–  Considercompressionoptimizations,videocompressionschemesmayhelp

eliminatetemporalredundancy

•  Surfacerenderingfortexturescanbeveryhelpfultoradiologists•  ExploreuseofThreeJSframeworktoabstractfromWebGL(suchasXTKand

ATM)•  ReducedResolution

–  MedicalImagesuselargesamplebitdepth(vscommon8bit)

•  IssueswithreliabilityandconsistencyofJavascriptsetTimeout()functionhavinga10msoverheadpercall,causingperformanceissues

•  Exposingadditionalcontrolsformanipulatinglighting,maybehelpful

•  Ageneralconcernontheeffectsofinterpolationandothersimulationsthatcandistortormisleadanexpert

ReferencesReviewedPaper:Congote,J.,Segura,A.,Kabongo,L.,Moreno,A.,Posada,J.,&Ruiz,O.(2011,June).Interactivevisualizationofvolumetricdatawithwebglinreal-time.InProceedingsofthe16thInternationalConferenceon3DWebTechnology(pp.137-146).ACM.Congote,J.,Segura,A.,Kabongo,L.,Moreno,A.,Posada,J.,&Ruiz,O.–DemoofReal-TimeInteractiveVisualizationofVolumetricDatawithWebGL.http://demos.vicomtech.org/volren/.AdditionalpapersHADWIGER,M.,LJUNG,P.,SALAMA,C.R.,ANDROPINSKI,T.2009.Advancedilluminationtechniquesforgpu-basedvolumeraycasting.InACMSIGGRAPH2009Courses,ACM,1–166.Haehn,D.,Rannou,N.,Ahtam,B.,Grant,E.,&Pienaar,R.(2014).NeuroimaginginthebrowserusingtheXToolkit.FrontiersinNeuroinformatics,101.LEVOY,M.1988.Displayofsurfacesfromvolumedata.IEEEComput.Graph.Appl.8,3,29–37.Min,Q.,Wang,Z.,&Liu,N.(2018).AnEvaluationofHTML5andWebGLforMedicalImagingApplications.Journalofhealthcareengineering,2018.