The Project Gutenberg eBook of Spiritual Torrents, by Madame Guyon.pdf
Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science...
Transcript of Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science...
![Page 1: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/1.jpg)
Academic TorrentsAcademic TorrentsScalable Distribution for Science
Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z LoUMass Boston Computer Science Ph.D Candidates
![Page 2: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/2.jpg)
Entire Presentation
Datasets-Searchable central index-Dynamic hosting locations-Ability to cache on campuses-Long term persistence-Aggregate sources
Publications-Long term persistence-New publication model: distributed publishing-Library Smart Nodes
![Page 3: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/3.jpg)
NSF Data Sharing Policy“Investigators are expected to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections and other supporting materials created or gathered in the course of work under NSF grants. Grantees are expected to encourage and facilitate such sharing. See” Award & Administration Guide (AAG) Chapter VI.D.4.
NIH Data Sharing Policies“Expects investigators seeking more than $500K in direct support in any given year to submit a data sharing plan with their application or to indicate why data sharing is not possible.”“Requires data for all NIDA-funded human genetics studies to be available for sharing, independent of direct costs, membership in the NIDA Genetics Consortium, or the type of genetics data generated.” http://www.nlm.nih.gov/NIHbmic/nih_data_sharing_policies.html
We need to share!
![Page 4: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/4.jpg)
Stick figures taken from xkcd
![Page 5: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/5.jpg)
Sharing is Hard
Considerations:● Maintenance - how much work?● Bandwidth - how scalable?● Speed - how fast are downloads?● Robustness - susceptible to failure?● Cost - how much will it be?
![Page 6: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/6.jpg)
Stick figures taken from xkcd
![Page 7: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/7.jpg)
One machine hosts a file from one location● Benefits
○ Simple (relatively)
● Pains○ Single point of failure (hard drive/network/power outage)○ Limited bandwidth (one machine serving the world)
Single Server Model
![Page 8: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/8.jpg)
Stick figures taken from xkcd
![Page 9: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/9.jpg)
Multiple machines host copies of a fileA central point sends the file to each mirror node (via scp, rsync)
A central index publishes hash of file to verify correctness
● Benefits○ Solves the single point of failure○ Might be faster if you download from a closer node
● Pains○ Each mirror must have high bandwidth○ Verification of each file is responsibility of the users
Apache Mirroring
![Page 10: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/10.jpg)
Maintains list of data locations dynamically (via API)Supports HTTP, FTP, and BitTorrent mirrors
● Benefits○ Long term preservation of data○ Automatic verification of data to ensure consistency○ Can extend existing data dissemination systems○ Download from multiple at once (on campus CDN!)
● Pains○ Clients are not designed for research (until now)○ Network firewalls (HTTP and FTP not blocked)
Academic Torrents
![Page 11: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/11.jpg)
Stick figures taken from xkcd
![Page 12: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/12.jpg)
Method ComparisonMaintenance Bandwidth limits Speed Robustness Cost
Single Server Moderate Somewhat Slow No Moderate
Multiple Servers High Somewhat Moderate Somewhat Moderate
Mailing Disks High No High No Low
Free Repositories Low Yes Moderate Somewhat Free
Proprietary Repositories Low Moderate Moderate Somewhat High
![Page 13: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/13.jpg)
Method ComparisonMaintenance Bandwidth limits Speed Robustness Cost
Single Server Moderate Somewhat Slow No Moderate
Multiple Servers High Somewhat Moderate Somewhat Moderate
Mailing Disks High No High No Low
Free Repositories Low Yes Moderate Somewhat Free
Proprietary Repositories Low Moderate Moderate Somewhat High
Academic Torrents Moderate No High Yes Low
![Page 14: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/14.jpg)
Academic Torrents
Peers get torrent from AT
Upload torrent to Academic
Torrents
Create torrent from data
Share data with peers
Transmission torrent client
![Page 15: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/15.jpg)
Academic Torrents Portal
![Page 16: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/16.jpg)
Each entry contains:
Bibtex Metadata (keys->values)
File listing with hashes (verify authenticity)
Listing of hosting locations (global mirror locations)
![Page 17: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/17.jpg)
Curated collections
![Page 18: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/18.jpg)
Each collection is:
Curated by a user (allows trust)
An updatable folder of entries (modifiable)
accessible via APIs (RSS, CSV, RESTful)
![Page 19: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/19.jpg)
![Page 20: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/20.jpg)
![Page 21: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/21.jpg)
![Page 22: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/22.jpg)
Command Line Interface (atdown)https://github.com/AcademicTorrents/AcademicTorrents-Downloader
![Page 23: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/23.jpg)
Command Line Interface (atdown)
![Page 24: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/24.jpg)
Use Case:Wikipedia XML Offline Version
10GB of DataCommunity Hosted
766 Downloads in 2014 (7.66TB!)~15 Persistent mirror locations
![Page 25: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/25.jpg)
Wikipedia data 10GBglobal mirror locations
![Page 26: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/26.jpg)
Speeds vary
Bytes!
At UMass Boston Campus, Boston, MA
At XSEDE14, Atlanta, GA
Different Mirror Access Mirrors have
different speeds
![Page 27: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/27.jpg)
Use Case:Direct Numerical Simulation of Turbulent Flows
5TB of Datain 63 files
Able to use AT infrastructure as management tool.
![Page 28: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/28.jpg)
Direct Numerical Simulation of Turbulent Flows 250GB/5TB in 2 Locations
![Page 29: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/29.jpg)
Entire Presentation
Datasets-Searchable central index-Dynamic hosting locations-Ability to cache on campuses-Long term persistence-Aggregate sources
Publications-Long term persistence-New publication model: distributed publishing-Library Smart Nodes
![Page 30: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/30.jpg)
Questions
Why can you expect papers to be accessible?
What is the cost of a research paper?
![Page 31: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/31.jpg)
![Page 32: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/32.jpg)
![Page 33: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/33.jpg)
Current Publishing Model,Elsevier, IEEE/ACM Journal
Distributed publishing model,Academic Torrents Library Smart Node
: ( IEEE/ACM Conference
Current Open Access Model,PLOS, F1000 Journals
Subscribers Everyone
Rea
der/L
ibra
ry P
ays
Aut
hor P
aysC
ost
Who can access
![Page 34: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/34.jpg)
![Page 35: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/35.jpg)
![Page 36: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/36.jpg)
Library Smart Node Overview
StudentLibrary Database,
OpenURL, orAtoZ Server
Elsevier
Springer
IEEE Explore
Academic Torrents Curated SmartNode
ScholarWorks
PLOS
JLMR
$$$$$$$$$$
![Page 37: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/37.jpg)
Smart Node
Management software for dealing with data
Deals with:Bandwidth Limits
Space LimitsContent (subscriptions)
![Page 38: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/38.jpg)
Smart Node V1
CS410 - Software DesignTeam of UndergraduatesGPL/C++
V2 will be in Java
https://github.com/AcademicTorrents/AcademicTorrents-SmartNodeV1
![Page 39: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/39.jpg)
Open Journal System IntegrationSimon Fraser University Library
![Page 40: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/40.jpg)
Academic Torrents
![Page 41: Academic Torrents - XSEDEAcademic Torrents Academic Torrents Scalable Distribution for Science Joseph Paul Cohen (NSF Graduate Fellow) and Henry Z Lo UMass Boston Computer Science](https://reader030.fdocuments.us/reader030/viewer/2022040107/5e46e79dab158a06955dfd94/html5/thumbnails/41.jpg)
Is this my dissertation topic? No.->Object detection in remote sensed imagery using machine learning +Ad-Hoc pervasive mobile networks +Semi-structured information extraction+CS and Cyber Security Education
blucat Throw Platform Feature Selection
Building Detection
Crater Detection