Batch Downloading of LiDAR DEM Data via the …€¦ · · 2013-02-20Batch Downloading of LiDAR...
10
1 TUTORIAL: Batch Downloading of LiDAR DEM Data via the OpenTopography Portal Christopher Crosby ([email protected]) Last updated: July 16, 2009 INTRODUCTION: This tutorial is designed to aid OpenTopography Portal users who wish to make bulk downloads of LiDAR DEM tiles via the interface located at: http://opentopo.sdsc.edu/gridsphere/gridsphere?cid=bulkdl The batch download capability of the OpenTopography Portal is designed for advanced users seeking to download large amounts of data quickly and who have the bandwidth, expertise, and software necessary to manage the gigabytes of data this method delivers. For most users we recommend the Google Map‐based tile download interface found in the Portal at: http://opentopography.org/dems METHOD: The OpenTopography Portal provides batch download capability for datasets where standard Digital Elevation Models are available. Under each dataset is a list of specific "targets" or acquisition areas. Clicking on the target link loads a page that lists referring URLs for all available 1km 2 DEM tiles for a data set on a single page to facilitate easy batch downloading. It is up to users to choose a method to retrieve these data in bulk. This tutorial provides a basic introduction to free software options to retrieve these data. A Google search for “download manager” will quickly reveal alternative options if the ones below do not work for you. We recommend and will highlight the following download managers in this tutorial: I. DownThemAll! extension for the Firefox browser (http://www.downthemall.net/ ) II. Free Download Manager (http://www.freedownloadmanager.org/) For users who are comfortable with commandline utilities, batch downloading of the tile DEMs can also be easily accomplished via the free GNU Wget program (http://www.gnu.org/software/wget/ ). Please address questions or comments on the OpenTopography Portal batch download capability to [email protected]
-
Upload
truongthien -
Category
Documents
-
view
216 -
download
1
Transcript of Batch Downloading of LiDAR DEM Data via the …€¦ · · 2013-02-20Batch Downloading of LiDAR...
1
TUTORIAL:BatchDownloadingofLiDARDEMDataviatheOpenTopographyPortal
ChristopherCrosby([email protected])
Lastupdated:July16,2009
INTRODUCTION:ThistutorialisdesignedtoaidOpenTopographyPortaluserswhowishtomakebulkdownloadsofLiDARDEMtilesviatheinterfacelocatedat:http://opentopo.sdsc.edu/gridsphere/gridsphere?cid=bulkdlThebatchdownloadcapabilityoftheOpenTopographyPortalisdesignedforadvancedusersseekingtodownloadlargeamountsofdataquicklyandwhohavethebandwidth,expertise,andsoftwarenecessarytomanagethegigabytesofdatathismethoddelivers.FormostuserswerecommendtheGoogleMap‐basedtiledownloadinterfacefoundinthePortalat:http://opentopography.org/demsMETHOD:TheOpenTopographyPortalprovidesbatchdownloadcapabilityfordatasetswherestandardDigitalElevationModelsareavailable.Undereachdatasetisalistofspecific"targets"oracquisitionareas.ClickingonthetargetlinkloadsapagethatlistsreferringURLsforallavailable1km2DEMtilesforadatasetonasinglepagetofacilitateeasybatchdownloading.Itisuptouserstochooseamethodtoretrievethesedatainbulk.Thistutorialprovidesabasicintroductiontofreesoftwareoptionstoretrievethesedata.AGooglesearchfor“downloadmanager”willquicklyrevealalternativeoptionsiftheonesbelowdonotworkforyou.Werecommendandwillhighlightthefollowingdownloadmanagersinthistutorial:
I. DownThemAll!extensionfortheFirefoxbrowser(http://www.downthemall.net/)
II. FreeDownloadManager(http://www.freedownloadmanager.org/)Foruserswhoarecomfortablewithcommandlineutilities,batchdownloadingofthetileDEMscanalsobeeasilyaccomplishedviathefreeGNUWgetprogram(http://www.gnu.org/software/wget/).PleaseaddressquestionsorcommentsontheOpenTopographyPortalbatchdownloadcapabilitytoinfo@opentopography.org
2
I.DownThemAll!
DownThemAll!(http://www.downthemall.net/)isapopularandfreedownloadmanagerfortheFirefoxwebbrowser.Onceyou’veinstalledDownThemAll!(dTa),usethefollowingstepstodownloaddatafromOpenTopography:
Fromthebatchdownloadspage(http://opentopo.sdsc.edu/gridsphere/gridsphere?cid=bulkdl)
Clickinga“target”(i.e.adatasetsubset)linksuchasthe“OldFaithful”portionoftheIntermountainSeismicBeltGeoEarthScopedatasetloadsapagethatlistsreferringURLsforallavailable1km2DigitalElevationModel(DEM)tilesforadatasetonasinglepagetofacilitateeasybatchdownloading.
3
EachURLonthepagebelowreturnsa.zipfilethatcontainsbareearth(ground)andunfiltered(allreturns)DEMs.Youcoulddownloadthetilesone‐by‐one,butadownloadmanagerprogramcanpullthemallwithminimalworkonyourpart.Notethateachpageliststhe#offilesandthetotaldatavolumeyouaregoingtogetback.
Aboveisthepageyou’llwanttorunthedownloadmanagertosequentiallydownloadalloftheURLslistedonthepage.
4
IMPORTANT:BeforeyourundTatodownloadthefiles,werecommendyoumakeachangetothedefaultsettingsoftheprogram.Navigateto“Tools”menuinFirefox>“DownThemAll!Tools”>“Preferences”andselectthe“Advanced”tab.Onthe“Advanced”tabyou’llwanttodisable“MultipartDownload”bymovingthesliderbarallthewaytotheleftuntilthedialogreads“1/Disabled”.Thischangeseemstoleadtofewerfailuresduringdownload.
5
Onceyou’vemodifiedthepreferencesofdTaasnotedabove,navigatebackto“Tools”>“DownThemAll!Tools”>butthistimeselect“DownThemAll!...”DoingsoloadsathedTadialogthatlookslikethis:
dTawillrecognizeeverylinkonthepagebutsinceallwewantareURLsthatpointto“.zip”files,weselectonlythe“Archives(zip,rar,…)”filteroptionandleaveallotherboxesunchecked.Inthe“Savefilesin:”box,definewhereyouwanttoputthedownloads.Onceyou’vegoteverythingset,click“Start!”
6
OncedTastartsrunningyougetadialogboxthatshowsthestatusofthedownloads.Dependinguponthenumberoffiles,serverload,andyourbandwidth,itmaytakeawhileforthedownloadstocomplete.
Insomecases,adownloadwillhangup‐forexampletheyellowbarintheexampleabovelistedasa“Servererror”.Inthiscaseyoucanselectthatfileandclickonthe“resume”buttonatthetoptorestartthedownload.Ifyouhaveafewfaileddownloads,youcanrightclickinthedTaboxandselect“Removecompletedtasks”toremovethefilesthatsucceeded.Then,selectalloftheremainingfilesandrestartthem.
7
II. FreeDownloadManagerTheFreeDownloadManager(http://www.freedownloadmanager.org/)isafree‐standingapplicationthatisagoodoptionforuserswhodonotusetheFirefoxbrowserandthereforecan’tuseDownThemAll!.FDMisawindowsonlyapplication.Onceyou’vedownloadedandinstalledFDM,followthefirsttwostepsoftheDownThemAll!tutorialabovetogettothewebpagethatliststhereferringURLsfortheDEMtilesforagivendataset:
8
TheFreeDownloadManagerinterfacelookssimilartothedTainterfacediscussedabovebutfunctionsslightlydifferentlysinceitisnotembeddedinthebrowser.
FDMhaslotsofoptions,butforthesakeofthissimpletutorial,wewillstickwiththebasics.First,itwillplacedownloadedfilesinC:\Downloads\bydefault‐tochangethedefaultdownloaddirectoryclickon“Options”>“Settings”>thenselect“NewDownload”fromthe“Downloads”menuandclickthe“Setdefaultdownloadfolder”.
9
SinceFDMisnotpartofthebrowser,itdoesn’tdiscovertheURLsonthepagethesamewaythatdTadoes.ThesimplestwaytoprovideFDMalistofURLstodownloadistoclickanddragtoselectalloftheURLsonthedownloadpagelikethis:
OncetheURLshavebeenselected,copythem(control+C)thengototheFDMinterfaceandselect“File”>“Import”>“ImportlistofURLsfromclipboard”.Youwillbepromptedto“SelectaGroup”,leavethedefaultandclickOK.
10
LikedTa,FDMwilldisplaystatusofeachdownloadasitruns:
Ifdownloadsgetstuck,theycanbestoppedandrestartedinasimilarmannertodTaasdiscussedabove.
