A Journey through New Languages - Guru Sorocaba 2017
-
Upload
fabio-akita -
Category
Technology
-
view
477 -
download
0
Transcript of A Journey through New Languages - Guru Sorocaba 2017
![Page 1: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/1.jpg)
LANGUAGES:A JOURNEY
@akitaonrails
![Page 2: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/2.jpg)
LANGUAGES:A JOURNEY
GURU SOROCABA 2017
@akitaonrails
![Page 3: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/3.jpg)
@akitaonrails
![Page 4: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/4.jpg)
www.theconf.club
![Page 5: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/5.jpg)
![Page 6: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/6.jpg)
Languages Syntax are EASY
Architectures (PATTERNS) are HARD
![Page 7: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/7.jpg)
![Page 8: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/8.jpg)
![Page 9: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/9.jpg)
![Page 10: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/10.jpg)
![Page 11: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/11.jpg)
![Page 12: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/12.jpg)
![Page 13: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/13.jpg)
![Page 14: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/14.jpg)
![Page 15: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/15.jpg)
git checkout -b old_version remotes/origin/old_version
![Page 16: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/16.jpg)
time bin/manga-downloadr -t
![Page 17: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/17.jpg)
#!/usr/bin/envruby$LOAD_PATH.unshiftFile.join(File.dirname(__FILE__),'..','lib')require'optparse'
options={test:false}option_parser=OptionParser.newdo|opts|opts.banner="Usage:manga-downloadr[options]"
opts.on("-t","--test","Testroutine")do|t|options[:url]="http://www.mangareader.net/onepunch-man"options[:name]="one-punch-man"options[:directory]="/tmp/manga-downloadr/one-punch-man"options[:test]=trueend
opts.on("-uURL","--urlURL","FullMangaReader.netmangahomepageURL-required")do|v|options[:url]=vend
opts.on("-nNAME","--nameNAME","slugtobeusedforthesub-foldertostoreallmangafiles-required")do|n|options[:name]=nend
opts.on("-dDIRECTORY","--directoryDIRECTORY","mainfolderwhereallmangaswillbestored-required")do|d|options[:directory]=dend
opts.on("-h","--help","Showthismessage")doputsoptsexitendend
![Page 18: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/18.jpg)
require'manga-downloadr'generator=MangaDownloadr::Workflow.create(options[:url],options[:name],options[:directory])
generator.fetch_chapter_urls!
generator.fetch_page_urls!
generator.fetch_image_urls!
generator.fetch_images!
generator.compile_ebooks!
![Page 19: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/19.jpg)
require'manga-downloadr'generator=MangaDownloadr::Workflow.create(options[:url],options[:name],options[:directory])
puts"Massiveparallelscanningofallchapters"generator.fetch_chapter_urls!
puts"\nMassiveparallelscanningofallpages"generator.fetch_page_urls!
puts"\nMassiveparallelscanningofallimages"generator.fetch_image_urls!puts"\nTotalpagelinksfound:#{generator.chapter_pages_count}"
puts"\nMassiveparalleldownloadofallpageimages"generator.fetch_images!
puts"\nCompilingallimagesintoPDFvolumes"generator.compile_ebooks!
puts"\nProcessfinished."
![Page 20: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/20.jpg)
require'manga-downloadr'generator=MangaDownloadr::Workflow.create(options[:url],options[:name],options[:directory])unlessgenerator.state?(:chapter_urls)puts"Massiveparallelscanningofallchapters"generator.fetch_chapter_urls!endunlessgenerator.state?(:page_urls)puts"\nMassiveparallelscanningofallpages"generator.fetch_page_urls!endunlessgenerator.state?(:image_urls)puts"\nMassiveparallelscanningofallimages"generator.fetch_image_urls!puts"\nTotalpagelinksfound:#{generator.chapter_pages_count}"endunlessgenerator.state?(:images)puts"\nMassiveparalleldownloadofallpageimages"generator.fetch_images!endunlessoptions[:test]puts"\nCompilingallimagesintoPDFvolumes"generator.compile_ebooks!endputs"\nProcessfinished."
MangaDownloadr::Workflow
![Page 21: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/21.jpg)
MangaDownloadr::WorkflowmoduleMangaDownloadrImageData=Struct.new(:folder,:filename,:url)
classWorkflow
definitialize(root_url=nil,manga_name=nil,manga_root=nil,options={})end
deffetch_chapter_urls!end
deffetch_page_urls!end
deffetch_image_urls!end
deffetch_images!end
defcompile_ebooks!end
defstate?(state)end
privatedefcurrent_state(state)end
endend
fetch_chapter_urls!
![Page 22: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/22.jpg)
moduleMangaDownloadrImageData=Struct.new(:folder,:filename,:url)
classWorkflow
definitialize(root_url=nil,manga_name=nil,manga_root=nil,options={})end
deffetch_chapter_urls!end
deffetch_page_urls!end
deffetch_image_urls!end
deffetch_images!end
defcompile_ebooks!end
defstate?(state)end
privatedefcurrent_state(state)end
endend
fetch_chapter_urls!
![Page 23: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/23.jpg)
fetch_chapter_urls!deffetch_chapter_urls!doc=Nokogiri::HTML(open(manga_root_url))
self.chapter_list=doc.css("#listinga").map{|l|l['href']}self.manga_title=doc.css("#mangapropertiesh1").first.text
current_state:chapter_urlsend
![Page 24: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/24.jpg)
fetch_chapter_urls!deffetch_chapter_urls!doc=Nokogiri::HTML(open(manga_root_url))
self.chapter_list=doc.css("#listinga").map{|l|l['href']}self.manga_title=doc.css("#mangapropertiesh1").first.text
current_state:chapter_urlsend
![Page 25: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/25.jpg)
deffetch_page_urls!
chapter_list.eachdo|chapter_link|
response=Typhoeus.get"http://www.mangareader.net#{chapter_link}"
chapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'
end
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 26: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/26.jpg)
deffetch_page_urls!
chapter_list.eachdo|chapter_link|beginresponse=Typhoeus.get"http://www.mangareader.net#{chapter_link}"
beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endend
rescue=>eputseendend
unlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 27: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/27.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 28: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/28.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 29: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/29.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 30: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/30.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 31: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/31.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 32: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/32.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 33: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/33.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 34: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/34.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 35: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/35.jpg)
deffetch_page_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{chapter_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)pages=chapter_doc.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")chapter_pages.merge!(chapter_link=>pages.map{|p|p['value']})print'.'rescue=>eself.fetch_page_urls_errors<<{url:chapter_link,error:e,body:response.body}print'x'endendhydra.queuerequestrescue=>eputseendendhydra.rununlessfetch_page_urls_errors.empty?puts"\nErrorsfetchingpageurls:"putsfetch_page_urls_errorsend
self.chapter_pages_count=chapter_pages.values.inject(0){|total,list|total+=list.size}current_state:page_urlsend
![Page 36: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/36.jpg)
deffetch_image_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_key|chapter_pages[chapter_key].eachdo|page_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{page_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)image=chapter_doc.css('#img').firsttokens=image['alt'].match("^(.*?)\s\-\s(.*?)$")extension=File.extname(URI.parse(image['src']).path)
chapter_images.merge!(chapter_key=>[])ifchapter_images[chapter_key].nil?chapter_images[chapter_key]<<ImageData.new(tokens[1],"#{tokens[2]}#{extension}",image['src'])print'.'rescue=>eself.fetch_image_urls_errors<<{url:page_link,error:e}print'x'endendhydra.queuerequestrescue=>eputseendendendhydra.rununlessfetch_image_urls_errors.empty?puts"\nErrorsfetchingimageurls:"putsfetch_image_urls_errorsend
current_state:image_urlsend
![Page 37: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/37.jpg)
deffetch_image_urls!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.eachdo|chapter_key|chapter_pages[chapter_key].eachdo|page_link|beginrequest=Typhoeus::Request.new"http://www.mangareader.net#{page_link}"request.on_completedo|response|beginchapter_doc=Nokogiri::HTML(response.body)image=chapter_doc.css('#img').firsttokens=image['alt'].match("^(.*?)\s\-\s(.*?)$")extension=File.extname(URI.parse(image['src']).path)
chapter_images.merge!(chapter_key=>[])ifchapter_images[chapter_key].nil?chapter_images[chapter_key]<<ImageData.new(tokens[1],"#{tokens[2]}#{extension}",image['src'])print'.'rescue=>eself.fetch_image_urls_errors<<{url:page_link,error:e}print'x'endendhydra.queuerequestrescue=>eputseendendendhydra.rununlessfetch_image_urls_errors.empty?puts"\nErrorsfetchingimageurls:"putsfetch_image_urls_errorsend
current_state:image_urlsend
![Page 38: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/38.jpg)
deffetch_images!hydra=Typhoeus::Hydra.new(max_concurrency:hydra_concurrency)chapter_list.each_with_indexdo|chapter_key,chapter_index|chapter_images[chapter_key].eachdo|file|downloaded_filename=File.join(manga_root_folder,file.folder,file.filename)nextifFile.exists?(downloaded_filename)#effectivelyresumesthedownloadlistwithoutre-downloadingeverythingrequest=Typhoeus::Request.newfile.urlrequest.on_completedo|response|begin#downloadFileUtils.mkdir_p(File.join(manga_root_folder,file.folder))File.open(downloaded_filename,"wb+"){|f|f.writeresponse.body}
unlessis_test#resizeimage=Magick::Image.read(downloaded_filename).firstresized=image.resize_to_fit(600,800)resized.write(downloaded_filename){self.quality=50}GC.start#toavoidaleaktoobig(ImageMagickisnotoriousforthat,speciallyonresizes)end
print'.'rescue=>eself.fetch_images_errors<<{url:file.url,error:e}print'#'endendhydra.queuerequestendendhydra.rununlessfetch_images_errors.empty?puts"\nErrorsdownloadingimages:"putsfetch_images_errorsend
current_state:imagesend
![Page 39: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/39.jpg)
defcompile_ebooks!folders=Dir[manga_root_folder+"/*/"].sort_by{|element|ary=element.split("").last.to_i}self.download_links=folders.inject([])do|list,folder|list+=Dir[folder+"*.*"].sort_by{|element|ary=element.split("").last.to_i}end
#concatenatingPDFfiles(250pagespervolume)chapter_number=0while!download_links.empty?chapter_number+=1pdf_file=File.join(manga_root_folder,"#{manga_title}#{chapter_number}.pdf")list=download_links.slice!(0..pages_per_volume)Prawn::Document.generate(pdf_file,page_size:page_size)do|pdf|list.eachdo|image_file|beginpdf.imageimage_file,position::center,vposition::centerrescue=>eputs"Errorin#{image_file}-#{e}"endendendprint'.'end
current_state:ebooksend
![Page 40: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/40.jpg)
time bin/manga-downloadr -t
17.18s user 17.62s system 41% cpu 1:24.04 total
![Page 41: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/41.jpg)
time bin/manga-downloadr -t
17.18s user 17.62s system 41% cpu 1:24.04 total
![Page 42: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/42.jpg)
![Page 43: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/43.jpg)
![Page 44: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/44.jpg)
![Page 45: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/45.jpg)
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
mix.exs
![Page 46: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/46.jpg)
mix.exsdefmoduleExMangaDownloadr.MixfiledouseMix.Project
defprojectdo[app::ex_manga_downloadr,version:"1.0.2",elixir:"~>1.4",build_embedded:Mix.env==:prod,start_permanent:Mix.env==:prod,escript:[main_module:ExMangaDownloadr.CLI],deps:deps()]end
defapplicationdo[applications:[:logger,:httpoison,:porcelain,:observer]]end
defpdepsdo[{:httpoison,"~>0.11"},{:floki,"~>0.17"},{:porcelain,"~>2.0.3"},{:mock,"~>0.2",only::test}]endend
Mixfile
![Page 47: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/47.jpg)
MixfilePoolManagement
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
workflow.ex
![Page 48: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/48.jpg)
Mixfile
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
workflow.ex
![Page 49: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/49.jpg)
workflow.exdefmoduleExMangaDownloadr.Workflowdodefdetermine_source(url)doend
defchapters({url,source})do{:ok,{_manga_title,chapter_list}}=MangaWrapper.index_page(url,source){chapter_list,source}end
defpages({chapter_list,source})dopages_list=chapter_list|>Task.async_stream(MangaWrapper,:chapter_page,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.reduce([],fn{:ok,{:ok,list}},acc->acc++listend){pages_list,source}end
defimages_sources({pages_list,source})dopages_list|>Task.async_stream(MangaWrapper,:page_image,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.map(fn{:ok,{:ok,image}}->imageend)end
defprocess_downloads(images_list,directory)doimages_list|>Task.async_stream(MangaWrapper,:page_download_image,[directory],max_concurrency:@max_demand/2,timeout:@download_timeout)|>Enum.to_list()directoryend
defoptimize_images(directory)do…enddefcompile_pdfs(directory,manga_name)do…end
defpcompile_volume(manga_name,directory,{chunk,index})do…enddefpprepare_volume(manga_name,directory,chunk,index)do…enddefpchunk(collection,default_size)do…endend
:chapter_page
![Page 50: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/50.jpg)
workflow.exdefmoduleExMangaDownloadr.Workflowdodefdetermine_source(url)doend
defchapters({url,source})do{:ok,{_manga_title,chapter_list}}=MangaWrapper.index_page(url,source){chapter_list,source}end
defpages({chapter_list,source})dopages_list=chapter_list|>Task.async_stream(MangaWrapper,:chapter_page,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.reduce([],fn{:ok,{:ok,list}},acc->acc++listend){pages_list,source}end
defimages_sources({pages_list,source})dopages_list|>Task.async_stream(MangaWrapper,:page_image,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.map(fn{:ok,{:ok,image}}->imageend)end
defprocess_downloads(images_list,directory)doimages_list|>Task.async_stream(MangaWrapper,:page_download_image,[directory],max_concurrency:@max_demand/2,timeout:@download_timeout)|>Enum.to_list()directoryend
defoptimize_images(directory)do…enddefcompile_pdfs(directory,manga_name)do…end
defpcompile_volume(manga_name,directory,{chunk,index})do…enddefpprepare_volume(manga_name,directory,chunk,index)do…enddefpchunk(collection,default_size)do…endend
:chapter_page
![Page 51: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/51.jpg)
workflow.exdefmoduleExMangaDownloadr.Workflowdodefdetermine_source(url)doend
defchapters({url,source})do{:ok,{_manga_title,chapter_list}}=MangaWrapper.index_page(url,source){chapter_list,source}end
defpages({chapter_list,source})dopages_list=chapter_list|>Task.async_stream(MangaWrapper,:chapter_page,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.reduce([],fn{:ok,{:ok,list}},acc->acc++listend){pages_list,source}end
defimages_sources({pages_list,source})dopages_list|>Task.async_stream(MangaWrapper,:page_image,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.map(fn{:ok,{:ok,image}}->imageend)end
defprocess_downloads(images_list,directory)doimages_list|>Task.async_stream(MangaWrapper,:page_download_image,[directory],max_concurrency:@max_demand/2,timeout:@download_timeout)|>Enum.to_list()directoryend
defoptimize_images(directory)do…enddefcompile_pdfs(directory,manga_name)do…end
defpcompile_volume(manga_name,directory,{chunk,index})do…enddefpprepare_volume(manga_name,directory,chunk,index)do…enddefpchunk(collection,default_size)do…endend
:chapter_page
![Page 52: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/52.jpg)
workflow.exdefmoduleExMangaDownloadr.Workflowdodefdetermine_source(url)doend
defchapters({url,source})do{:ok,{_manga_title,chapter_list}}=MangaWrapper.index_page(url,source){chapter_list,source}end
defpages({chapter_list,source})dopages_list=chapter_list|>Task.async_stream(MangaWrapper,:chapter_page,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.reduce([],fn{:ok,{:ok,list}},acc->acc++listend){pages_list,source}end
defimages_sources({pages_list,source})dopages_list|>Task.async_stream(MangaWrapper,:page_image,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.map(fn{:ok,{:ok,image}}->imageend)end
defprocess_downloads(images_list,directory)doimages_list|>Task.async_stream(MangaWrapper,:page_download_image,[directory],max_concurrency:@max_demand/2,timeout:@download_timeout)|>Enum.to_list()directoryend
defoptimize_images(directory)do…enddefcompile_pdfs(directory,manga_name)do…end
defpcompile_volume(manga_name,directory,{chunk,index})do…enddefpprepare_volume(manga_name,directory,chunk,index)do…enddefpchunk(collection,default_size)do…endend
:chapter_page
![Page 53: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/53.jpg)
![Page 54: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/54.jpg)
POOL
![Page 55: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/55.jpg)
workflow.exdefmoduleExMangaDownloadr.Workflowdodefdetermine_source(url)doend
defchapters({url,source})do{:ok,{_manga_title,chapter_list}}=MangaWrapper.index_page(url,source){chapter_list,source}end
defpages({chapter_list,source})dopages_list=chapter_list|>Task.async_stream(MangaWrapper,:chapter_page,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.reduce([],fn{:ok,{:ok,list}},acc->acc++listend){pages_list,source}end
defimages_sources({pages_list,source})dopages_list|>Task.async_stream(MangaWrapper,:page_image,[source],max_concurrency:@max_demand)|>Enum.to_list()|>Enum.map(fn{:ok,{:ok,image}}->imageend)end
defprocess_downloads(images_list,directory)doimages_list|>Task.async_stream(MangaWrapper,:page_download_image,[directory],max_concurrency:@max_demand/2,timeout:@download_timeout)|>Enum.to_list()directoryend
defoptimize_images(directory)do…enddefcompile_pdfs(directory,manga_name)do…end
defpcompile_volume(manga_name,directory,{chunk,index})do…enddefpprepare_volume(manga_name,directory,chunk,index)do…enddefpchunk(collection,default_size)do…endend
:chapter_page
![Page 56: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/56.jpg)
manga_wrapper.exdefmoduleMangaWrapperdorequireLogger
defindex_page(url,source)dosource|>manga_source("IndexPage")|>apply(:chapters,[url])end
defchapter_page(chapter_link,source)dosource|>manga_source("ChapterPage")|>apply(:pages,[chapter_link])end
defpage_image(page_link,source)dosource|>manga_source("Page")|>apply(:image,[page_link])end
defpage_download_image(image_data,directory)dodownload_image(image_data,directory)end
defpmanga_source(source,module)docasesourcedo"mangareader"->:"Elixir.ExMangaDownloadr.MangaReader.#{module}""mangafox"->:"Elixir.ExMangaDownloadr.Mangafox.#{module}"endend
defpdownload_image({image_src,image_filename},directory)do
endend
:chapter_page
ChapterPage
![Page 57: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/57.jpg)
manga_wrapper.exdefmoduleMangaWrapperdorequireLogger
defindex_page(url,source)dosource|>manga_source("IndexPage")|>apply(:chapters,[url])end
defchapter_page(chapter_link,source)dosource|>manga_source("ChapterPage")|>apply(:pages,[chapter_link])end
defpage_image(page_link,source)dosource|>manga_source("Page")|>apply(:image,[page_link])end
defpage_download_image(image_data,directory)dodownload_image(image_data,directory)end
defpmanga_source(source,module)docasesourcedo"mangareader"->:"Elixir.ExMangaDownloadr.MangaReader.#{module}""mangafox"->:"Elixir.ExMangaDownloadr.Mangafox.#{module}"endend
defpdownload_image({image_src,image_filename},directory)do
endend
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
:chapter_page
ChapterPage
![Page 58: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/58.jpg)
manga_wrapper.exdefmoduleMangaWrapperdorequireLogger
defindex_page(url,source)dosource|>manga_source("IndexPage")|>apply(:chapters,[url])end
defchapter_page(chapter_link,source)dosource|>manga_source("ChapterPage")|>apply(:pages,[chapter_link])end
defpage_image(page_link,source)dosource|>manga_source("Page")|>apply(:image,[page_link])end
defpage_download_image(image_data,directory)dodownload_image(image_data,directory)end
defpmanga_source(source,module)docasesourcedo"mangareader"->:"Elixir.ExMangaDownloadr.MangaReader.#{module}""mangafox"->:"Elixir.ExMangaDownloadr.Mangafox.#{module}"endend
defpdownload_image({image_src,image_filename},directory)do
endend
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
:chapter_page
ChapterPage
![Page 59: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/59.jpg)
manga_wrapper.exdefmoduleMangaWrapperdorequireLogger
defindex_page(url,source)dosource|>manga_source("IndexPage")|>apply(:chapters,[url])end
defchapter_page(chapter_link,source)dosource|>manga_source("ChapterPage")|>apply(:pages,[chapter_link])end
defpage_image(page_link,source)dosource|>manga_source("Page")|>apply(:image,[page_link])end
defpage_download_image(image_data,directory)dodownload_image(image_data,directory)end
defpmanga_source(source,module)docasesourcedo"mangareader"->:"Elixir.ExMangaDownloadr.MangaReader.#{module}""mangafox"->:"Elixir.ExMangaDownloadr.Mangafox.#{module}"endend
defpdownload_image({image_src,image_filename},directory)do
endend
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
:chapter_page
ChapterPage
![Page 60: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/60.jpg)
manga_wrapper.exdefmoduleMangaWrapperdorequireLogger
defindex_page(url,source)dosource|>manga_source("IndexPage")|>apply(:chapters,[url])end
defchapter_page(chapter_link,source)dosource|>manga_source("ChapterPage")|>apply(:pages,[chapter_link])end
defpage_image(page_link,source)dosource|>manga_source("Page")|>apply(:image,[page_link])end
defpage_download_image(image_data,directory)dodownload_image(image_data,directory)end
defpmanga_source(source,module)docasesourcedo"mangareader"->:"Elixir.ExMangaDownloadr.MangaReader.#{module}""mangafox"->:"Elixir.ExMangaDownloadr.Mangafox.#{module}"endend
defpdownload_image({image_src,image_filename},directory)do
endend
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
:chapter_page
ChapterPage
![Page 61: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/61.jpg)
defmoduleExMangaDownloadr.Mangafox.ChapterPagedorequireLoggerrequireExMangaDownloadr
defpages(chapter_link)doExMangaDownloadr.fetchchapter_link,do:fetch_pages(chapter_link)end
defpfetch_pages(html,chapter_link)do[_page|link_template]=chapter_link|>String.split("/")|>Enum.reverse
html|>Floki.find("div[id='top_center_bar']option")|>Floki.attribute("value")|>Enum.reject(fnpage_number->page_number=="0"end)|>Enum.map(fnpage_number->["#{page_number}.html"|link_template]|>Enum.reverse|>Enum.join("/")end)endend
ChapterPage
![Page 62: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/62.jpg)
ChapterPage
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
cli.ex
![Page 63: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/63.jpg)
cli.exdefmoduleExMangaDownloadr.CLIdoaliasExMangaDownloadr.WorkflowrequireExMangaDownloadrdefmain(args)doargs|>parse_args|>processend...defpparse_args(args)doend
defpprocess(:help)doend
defpprocess(directory,url)doFile.mkdir_p!(directory)File.mkdir_p!("/tmp/ex_manga_downloadr_cache")
manga_name=directory|>String.split("/")|>Enum.reverse|>Enum.at(0)url|>Workflow.determine_source|>Workflow.chapters|>Workflow.pages|>Workflow.images_sources|>Workflow.process_downloads(directory)|>Workflow.optimize_images|>Workflow.compile_pdfs(manga_name)|>finish_processend
defpprocess_test(directory,url)doend
defpfinish_process(directory)doendend
Workflow
![Page 64: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/64.jpg)
mix deps.getmix testmix escript.build
![Page 65: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/65.jpg)
mix deps.getmix testmix escript.build
ex_manga_downloadr - 4.6M
![Page 66: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/66.jpg)
time ./ex_manga_downloadr —test
![Page 67: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/67.jpg)
time ./ex_manga_downloadr —test
32.03s user 57.97s system 120% cpu 1:14.45 total
![Page 68: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/68.jpg)
![Page 69: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/69.jpg)
![Page 70: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/70.jpg)
![Page 71: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/71.jpg)
![Page 72: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/72.jpg)
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
. !"" cr_manga_downloadr !"" libs # !"" ... !"" LICENSE !"" README.md !"" shard.lock !"" shard.yml !"" spec # !"" cr_manga_downloadr # # !"" chapters_spec.cr # # !"" concurrency_spec.cr # # !"" image_downloader_spec.cr # # !"" page_image_spec.cr # # $"" pages_spec.cr # !"" fixtures # # !"" ... # $"" spec_helper.cr $"" src !"" cr_manga_downloadr # !"" chapters.cr # !"" concurrency.cr # !"" downloadr_client.cr # !"" image_downloader.cr # !"" page_image.cr # !"" pages.cr # !"" records.cr # !"" version.cr # $"" workflow.cr $"" cr_manga_downloadr.cr
![Page 73: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/73.jpg)
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
. !"" cr_manga_downloadr !"" libs # !"" ... !"" LICENSE !"" README.md !"" shard.lock !"" shard.yml !"" spec # !"" cr_manga_downloadr # # !"" chapters_spec.cr # # !"" concurrency_spec.cr # # !"" image_downloader_spec.cr # # !"" page_image_spec.cr # # $"" pages_spec.cr # !"" fixtures # # !"" ... # $"" spec_helper.cr $"" src !"" cr_manga_downloadr # !"" chapters.cr # !"" concurrency.cr # !"" downloadr_client.cr # !"" image_downloader.cr # !"" page_image.cr # !"" pages.cr # !"" records.cr # !"" version.cr # $"" workflow.cr $"" cr_manga_downloadr.cr
![Page 74: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/74.jpg)
File.mkdir_p!(directory)File.mkdir_p!("/tmp/ex_manga_downloadr_cache")
manga_name=directory|>String.split("/")|>Enum.reverse|>Enum.at(0)url|>Workflow.determine_source|>Workflow.chapters|>Workflow.pages|>Workflow.images_sources|>Workflow.process_downloads(directory)|>Workflow.optimize_images|>Workflow.compile_pdfs(manga_name)|>finish_processend
![Page 75: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/75.jpg)
[email protected]_directory
pipeSteps.fetch_chapters(@config).>>Steps.fetch_pages(@config).>>Steps.fetch_images(@config).>>Steps.download_images(@config).>>Steps.optimize_images(@config).>>Steps.prepare_volumes(@config).>>unwrap
puts"Done!"end
File.mkdir_p!(directory)File.mkdir_p!("/tmp/ex_manga_downloadr_cache")
manga_name=directory|>String.split("/")|>Enum.reverse|>Enum.at(0)url|>Workflow.determine_source|>Workflow.chapters|>Workflow.pages|>Workflow.images_sources|>Workflow.process_downloads(directory)|>Workflow.optimize_images|>Workflow.compile_pdfs(manga_name)|>finish_processend
![Page 76: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/76.jpg)
#1y=c(b(a))
#2x=b(a)y=c(x)
#ElixirPipesy=a|>b|>c
#CrystalMacroPipesy=pipea.>>b.>>c.>>unwrap
![Page 77: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/77.jpg)
![Page 78: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/78.jpg)
defmoduleExMangaDownloadr.MangaReader.IndexPagedorequireLoggerrequireExMangaDownloadr
defchapters(manga_root_url)doExMangaDownloadr.fetchmanga_root_url,do:collectend
defpcollect(html)do{fetch_manga_title(html),fetch_chapters(html)}end
defpfetch_manga_title(html)dohtml|>Floki.find("#mangapropertiesh1")|>Floki.textend
defpfetch_chapters(html)dohtml|>Floki.find("#listinga")|>Floki.attribute("href")endend
![Page 79: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/79.jpg)
defmoduleExMangaDownloadr.MangaReader.IndexPagedorequireLoggerrequireExMangaDownloadr
defchapters(manga_root_url)doExMangaDownloadr.fetchmanga_root_url,do:collectend
defpcollect(html)do{fetch_manga_title(html),fetch_chapters(html)}end
defpfetch_manga_title(html)dohtml|>Floki.find("#mangapropertiesh1")|>Floki.textend
defpfetch_chapters(html)dohtml|>Floki.find("#listinga")|>Floki.attribute("href")endend
![Page 80: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/80.jpg)
defmoduleExMangaDownloadr.MangaReader.IndexPagedorequireLoggerrequireExMangaDownloadr
defchapters(manga_root_url)doExMangaDownloadr.fetchmanga_root_url,do:collectend
defpcollect(html)do{fetch_manga_title(html),fetch_chapters(html)}end
defpfetch_manga_title(html)dohtml|>Floki.find("#mangapropertiesh1")|>Floki.textend
defpfetch_chapters(html)dohtml|>Floki.find("#listinga")|>Floki.attribute("href")endend
![Page 81: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/81.jpg)
require"./downloadr_client"require"xml"
moduleCrMangaDownloadrclassChapters<DownloadrClientdeffetchhtml=get(@config.root_uri).as(XML::Node)nodes=html.xpath_nodes("//table[contains(@id,'listing')]//td//a/@href")nodes.map{|node|node.text.as(String)}endendend
DownloadrClient
![Page 82: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/82.jpg)
require"./downloadr_client"require"xml"
moduleCrMangaDownloadrclassChapters<DownloadrClientdeffetchhtml=get(@config.root_uri).as(XML::Node)nodes=html.xpath_nodes("//table[contains(@id,'listing')]//td//a/@href")nodes.map{|node|node.text.as(String)}endendend
DownloadrClient
![Page 83: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/83.jpg)
moduleCrMangaDownloadrclassDownloadrClient...defget(uri:String,binary=false)Dir.mkdir_p(@config.cache_directory)unlessDir.exists?(@config.cache_directory)cache_path=File.join(@config.cache_directory,cache_filename(uri))whiletruebeginresponse=if@cache_http&&File.exists?(cache_path)body=File.read(cache_path)HTTP::Client::Response.new(200,body)else@http_client.get(uri,headers:HTTP::Headers{"User-Agent"=>CrMangaDownloadr::USER_AGENT})end
caseresponse.status_codewhen301uri=response.headers["Location"]when200if(binary||@cache_http)&&!File.exists?(cache_path)File.open(cache_path,"w")do|f|f.printresponse.bodyendend
ifbinaryreturncache_pathelsereturnXML.parse_html(response.body)endendrescueIO::Timeoutputs"Sleepingover#{uri}"sleep1endendend...end
DownloadrClient
![Page 84: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/84.jpg)
moduleCrMangaDownloadrclassDownloadrClient...defget(uri:String,binary=false)Dir.mkdir_p(@config.cache_directory)unlessDir.exists?(@config.cache_directory)cache_path=File.join(@config.cache_directory,cache_filename(uri))whiletruebeginresponse=if@cache_http&&File.exists?(cache_path)body=File.read(cache_path)HTTP::Client::Response.new(200,body)else@http_client.get(uri,headers:HTTP::Headers{"User-Agent"=>CrMangaDownloadr::USER_AGENT})end
caseresponse.status_codewhen301uri=response.headers["Location"]when200if(binary||@cache_http)&&!File.exists?(cache_path)File.open(cache_path,"w")do|f|f.printresponse.bodyendend
ifbinaryreturncache_pathelsereturnXML.parse_html(response.body)endendrescueIO::Timeoutputs"Sleepingover#{uri}"sleep1endendend...end
DownloadrClient
![Page 85: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/85.jpg)
moduleCrMangaDownloadrclassDownloadrClient...defget(uri:String,binary=false)Dir.mkdir_p(@config.cache_directory)unlessDir.exists?(@config.cache_directory)cache_path=File.join(@config.cache_directory,cache_filename(uri))whiletruebeginresponse=if@cache_http&&File.exists?(cache_path)body=File.read(cache_path)HTTP::Client::Response.new(200,body)else@http_client.get(uri,headers:HTTP::Headers{"User-Agent"=>CrMangaDownloadr::USER_AGENT})end
caseresponse.status_codewhen301uri=response.headers["Location"]when200if(binary||@cache_http)&&!File.exists?(cache_path)File.open(cache_path,"w")do|f|f.printresponse.bodyendend
ifbinaryreturncache_pathelsereturnXML.parse_html(response.body)endendrescueIO::Timeoutputs"Sleepingover#{uri}"sleep1endendend...end
DownloadrClient
![Page 86: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/86.jpg)
moduleCrMangaDownloadrclassDownloadrClient...defget(uri:String,binary=false)Dir.mkdir_p(@config.cache_directory)unlessDir.exists?(@config.cache_directory)cache_path=File.join(@config.cache_directory,cache_filename(uri))whiletruebeginresponse=if@cache_http&&File.exists?(cache_path)body=File.read(cache_path)HTTP::Client::Response.new(200,body)else@http_client.get(uri,headers:HTTP::Headers{"User-Agent"=>CrMangaDownloadr::USER_AGENT})end
caseresponse.status_codewhen301uri=response.headers["Location"]when200if(binary||@cache_http)&&!File.exists?(cache_path)File.open(cache_path,"w")do|f|f.printresponse.bodyendend
ifbinaryreturncache_pathelsereturnXML.parse_html(response.body)endendrescueIO::Timeoutputs"Sleepingover#{uri}"sleep1endendend...end
DownloadrClient
![Page 87: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/87.jpg)
moduleCrMangaDownloadrclassDownloadrClient...defget(uri:String,binary=false)Dir.mkdir_p(@config.cache_directory)unlessDir.exists?(@config.cache_directory)cache_path=File.join(@config.cache_directory,cache_filename(uri))whiletruebeginresponse=if@cache_http&&File.exists?(cache_path)body=File.read(cache_path)HTTP::Client::Response.new(200,body)else@http_client.get(uri,headers:HTTP::Headers{"User-Agent"=>CrMangaDownloadr::USER_AGENT})end
caseresponse.status_codewhen301uri=response.headers["Location"]when200if(binary||@cache_http)&&!File.exists?(cache_path)File.open(cache_path,"w")do|f|f.printresponse.bodyendend
ifbinaryreturncache_pathelsereturnXML.parse_html(response.body)endendrescueIO::Timeoutputs"Sleepingover#{uri}"sleep1endendend...end
DownloadrClient
![Page 88: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/88.jpg)
moduleCrMangaDownloadrclassDownloadrClient...defget(uri:String,binary=false)Dir.mkdir_p(@config.cache_directory)unlessDir.exists?(@config.cache_directory)cache_path=File.join(@config.cache_directory,cache_filename(uri))whiletruebeginresponse=if@cache_http&&File.exists?(cache_path)body=File.read(cache_path)HTTP::Client::Response.new(200,body)else@http_client.get(uri,headers:HTTP::Headers{"User-Agent"=>CrMangaDownloadr::USER_AGENT})end
caseresponse.status_codewhen301uri=response.headers["Location"]when200if(binary||@cache_http)&&!File.exists?(cache_path)File.open(cache_path,"w")do|f|f.printresponse.bodyendend
ifbinaryreturncache_pathelsereturnXML.parse_html(response.body)endendrescueIO::Timeoutputs"Sleepingover#{uri}"sleep1endendend...end
DownloadrClient
![Page 89: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/89.jpg)
require"fiberpool"
moduleCrMangaDownloadrstructConcurrency(A,B)definitialize(@config:Config,@engine_class:DownloadrClient.class)end
deffetch(collection:Array(A)?,&block:A,DownloadrClient->Array(B)?):Array(B)results=[]ofBifcollectionpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endendendresultsendendend
fetch
Concurrency
![Page 90: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/90.jpg)
require"fiberpool"
moduleCrMangaDownloadrstructConcurrency(A,B)definitialize(@config:Config,@engine_class:DownloadrClient.class)end
deffetch(collection:Array(A)?,&block:A,DownloadrClient->Array(B)?):Array(B)results=[]ofBifcollectionpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endendendresultsendendend
fetch
Concurrency
![Page 91: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/91.jpg)
require"fiberpool"
moduleCrMangaDownloadrstructConcurrency(A,B)definitialize(@config:Config,@engine_class:DownloadrClient.class)end
deffetch(collection:Array(A)?,&block:A,DownloadrClient->Array(B)?):Array(B)results=[]ofBifcollectionpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endendendresultsendendend
fetch
Concurrency
![Page 92: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/92.jpg)
require"fiberpool"
moduleCrMangaDownloadrstructConcurrency(A,B)definitialize(@config:Config,@engine_class:DownloadrClient.class)end
deffetch(collection:Array(A)?,&block:A,DownloadrClient->Array(B)?):Array(B)results=[]ofBifcollectionpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endendendresultsendendend
fetch
Concurrency
![Page 93: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/93.jpg)
require"fiberpool"
moduleCrMangaDownloadrstructConcurrency(A,B)definitialize(@config:Config,@engine_class:DownloadrClient.class)end
deffetch(collection:Array(A)?,&block:A,DownloadrClient->Array(B)?):Array(B)results=[]ofBifcollectionpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endendendresultsendendend
fetch
Concurrency
![Page 94: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/94.jpg)
require"fiberpool"
moduleCrMangaDownloadrstructConcurrency(A,B)definitialize(@config:Config,@engine_class:DownloadrClient.class)end
deffetch(collection:Array(A)?,&block:A,DownloadrClient->Array(B)?):Array(B)results=[]ofBifcollectionpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endendendresultsendendend
fetch
Concurrency
![Page 95: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/95.jpg)
fetchConcurrency
moduleCrMangaDownloadrclassWorkflowend
moduleStepsdefself.fetch_chapters(config:Config)end
defself.fetch_pages(chapters:Array(String)?,config:Config)puts"Fetchingpagesfromallchapters..."reactor=Concurrency(String,String).new(config,Pages)reactor.fetch(chapters)do|link,engine|engine.try(&.fetch(link)).as(Array(String))endend
defself.fetch_images(pages:Array(String)?,config:Config)end
defself.download_images(images:Array(Image)?,config:Config)end
defself.optimize_images(downloads:Array(String),config:Config)end
defself.prepare_volumes(downloads:Array(String),config:Config)endendend
![Page 96: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/96.jpg)
fetchConcurrency
moduleCrMangaDownloadrclassWorkflowend
moduleStepsdefself.fetch_chapters(config:Config)end
defself.fetch_pages(chapters:Array(String)?,config:Config)puts"Fetchingpagesfromallchapters..."reactor=Concurrency(String,String).new(config,Pages)reactor.fetch(chapters)do|link,engine|engine.try(&.fetch(link)).as(Array(String))endend
defself.fetch_images(pages:Array(String)?,config:Config)end
defself.download_images(images:Array(Image)?,config:Config)end
defself.optimize_images(downloads:Array(String),config:Config)end
defself.prepare_volumes(downloads:Array(String),config:Config)endendend
![Page 97: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/97.jpg)
![Page 98: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/98.jpg)
crystal depscrystal speccrystal build src/cr_manga_downloadr.cr --release
![Page 99: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/99.jpg)
crystal depscrystal speccrystal build src/cr_manga_downloadr.cr --release
cr_manga_downloadr 752K
![Page 100: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/100.jpg)
time ./cr_manga_downloadr -t
![Page 101: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/101.jpg)
time ./cr_manga_downloadr -t
5.57s user 6.79s system 14% cpu 1:26.76 total
![Page 102: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/102.jpg)
![Page 103: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/103.jpg)
. !"" _build # $"" ... !"" config # $"" config.exs !"" deps # !"" ... !"" ex_manga_downloadr !"" lib # !"" ex_manga_downloadr # # !"" cli.ex # # !"" mangafox # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" mangareader # # # !"" chapter_page.ex # # # !"" index_page.ex # # # $"" page.ex # # !"" manga_wrapper.ex # # $"" workflow.ex # $"" ex_manga_downloadr.ex !"" mix.exs !"" mix.lock !"" README.md $"" test !"" ex_manga_downloadr # !"" mangafox_test.exs # $"" mangareader_test.exs !"" ex_manga_downloadr_test.exs $"" test_helper.exs
61 directories, 281 files
. !"" cr_manga_downloadr !"" libs # !"" ... !"" LICENSE !"" README.md !"" shard.lock !"" shard.yml !"" spec # !"" cr_manga_downloadr # # !"" chapters_spec.cr # # !"" concurrency_spec.cr # # !"" image_downloader_spec.cr # # !"" page_image_spec.cr # # $"" pages_spec.cr # !"" fixtures # # !"" ... # $"" spec_helper.cr $"" src !"" cr_manga_downloadr # !"" chapters.cr # !"" concurrency.cr # !"" downloadr_client.cr # !"" image_downloader.cr # !"" page_image.cr # !"" pages.cr # !"" records.cr # !"" version.cr # $"" workflow.cr $"" cr_manga_downloadr.cr
![Page 104: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/104.jpg)
. !"" cr_manga_downloadr !"" libs # !"" ... !"" LICENSE !"" README.md !"" shard.lock !"" shard.yml !"" spec # !"" cr_manga_downloadr # # !"" chapters_spec.cr # # !"" concurrency_spec.cr # # !"" image_downloader_spec.cr # # !"" page_image_spec.cr # # $"" pages_spec.cr # !"" fixtures # # !"" ... # $"" spec_helper.cr $"" src !"" cr_manga_downloadr # !"" chapters.cr # !"" concurrency.cr # !"" downloadr_client.cr # !"" image_downloader.cr # !"" page_image.cr # !"" pages.cr # !"" records.cr # !"" version.cr # $"" workflow.cr $"" cr_manga_downloadr.cr
. !"" bin # $"" manga-downloadr !"" Gemfile !"" Gemfile.lock !"" lib # !"" manga-downloadr # # !"" chapters.rb # # !"" concurrency.rb # # !"" downloadr_client.rb # # !"" image_downloader.rb # # !"" page_image.rb # # !"" pages.rb # # !"" records.rb # # !"" version.rb # # $"" workflow.rb # $"" manga-downloadr.rb !"" LICENSE.txt !"" manga-downloadr.gemspec !"" Rakefile !"" README.md $"" spec !"" fixtures # !"" ... !"" manga-downloadr # !"" chapters_spec.rb # !"" concurrency_spec.rb # !"" image_downloader_spec.rb # !"" page_image_spec.rb # $"" pages_spec.rb $"" spec_helper.rb
![Page 105: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/105.jpg)
. !"" cr_manga_downloadr !"" libs # !"" ... !"" LICENSE !"" README.md !"" shard.lock !"" shard.yml !"" spec # !"" cr_manga_downloadr # # !"" chapters_spec.cr # # !"" concurrency_spec.cr # # !"" image_downloader_spec.cr # # !"" page_image_spec.cr # # $"" pages_spec.cr # !"" fixtures # # !"" ... # $"" spec_helper.cr $"" src !"" cr_manga_downloadr # !"" chapters.cr # !"" concurrency.cr # !"" downloadr_client.cr # !"" image_downloader.cr # !"" page_image.cr # !"" pages.cr # !"" records.cr # !"" version.cr # $"" workflow.cr $"" cr_manga_downloadr.cr
. !"" bin # $"" manga-downloadr !"" Gemfile !"" Gemfile.lock !"" lib # !"" manga-downloadr # # !"" chapters.rb # # !"" concurrency.rb # # !"" downloadr_client.rb # # !"" image_downloader.rb # # !"" page_image.rb # # !"" pages.rb # # !"" records.rb # # !"" version.rb # # $"" workflow.rb # $"" manga-downloadr.rb !"" LICENSE.txt !"" manga-downloadr.gemspec !"" Rakefile !"" README.md $"" spec !"" fixtures # !"" ... !"" manga-downloadr # !"" chapters_spec.rb # !"" concurrency_spec.rb # !"" image_downloader_spec.rb # !"" page_image_spec.rb # $"" pages_spec.rb $"" spec_helper.rb
![Page 106: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/106.jpg)
. !"" cr_manga_downloadr !"" libs # !"" ... !"" LICENSE !"" README.md !"" shard.lock !"" shard.yml !"" spec # !"" cr_manga_downloadr # # !"" chapters_spec.cr # # !"" concurrency_spec.cr # # !"" image_downloader_spec.cr # # !"" page_image_spec.cr # # $"" pages_spec.cr # !"" fixtures # # !"" ... # $"" spec_helper.cr $"" src !"" cr_manga_downloadr # !"" chapters.cr # !"" concurrency.cr # !"" downloadr_client.cr # !"" image_downloader.cr # !"" page_image.cr # !"" pages.cr # !"" records.cr # !"" version.cr # $"" workflow.cr $"" cr_manga_downloadr.cr
. !"" bin # $"" manga-downloadr !"" Gemfile !"" Gemfile.lock !"" lib # !"" manga-downloadr # # !"" chapters.rb # # !"" concurrency.rb # # !"" downloadr_client.rb # # !"" image_downloader.rb # # !"" page_image.rb # # !"" pages.rb # # !"" records.rb # # !"" version.rb # # $"" workflow.rb # $"" manga-downloadr.rb !"" LICENSE.txt !"" manga-downloadr.gemspec !"" Rakefile !"" README.md $"" spec !"" fixtures # !"" ... !"" manga-downloadr # !"" chapters_spec.rb # !"" concurrency_spec.rb # !"" image_downloader_spec.rb # !"" page_image_spec.rb # $"" pages_spec.rb $"" spec_helper.rb
![Page 107: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/107.jpg)
[email protected]_directory
pipeSteps.fetch_chapters(@config).>>Steps.fetch_pages(@config).>>Steps.fetch_images(@config).>>Steps.download_images(@config).>>Steps.optimize_images(@config).>>Steps.prepare_volumes(@config).>>unwrap
puts"Done!"end
![Page 108: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/108.jpg)
defself.run(config=Config.new)FileUtils.mkdir_pconfig.download_directory
CM(config,Workflow).fetch_chapters.fetch_pages(config).fetch_images(config).download_images(config).optimize_images(config).prepare_volumes(config).unwrap
puts"Done!"end
[email protected]_directory
pipeSteps.fetch_chapters(@config).>>Steps.fetch_pages(@config).>>Steps.fetch_images(@config).>>Steps.download_images(@config).>>Steps.optimize_images(@config).>>Steps.prepare_volumes(@config).>>unwrap
puts"Done!"end
![Page 109: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/109.jpg)
![Page 110: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/110.jpg)
#concurrency.crpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endend
![Page 111: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/111.jpg)
#concurrency.crpool=Fiberpool.new(collection,@config.download_batch_size)pool.rundo|item|engine=@engine_class.new(@config)ifreply=block.call(item,engine)results.concat(reply)endend
pool=Thread.pool(@config.download_batch_size)mutex=Mutex.newresults=[]
collection.eachdo|item|pool.process{engine=@turn_on_engine?@engine_klass.new(@config.domain,@config.cache_http):nilreply=block.call(item,engine)&.flattenmutex.synchronizedoresults+=(reply||[])end}endpool.shutdown
![Page 112: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/112.jpg)
Fibers
Threads
![Page 113: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/113.jpg)
moduleCrMangaDownloadrclassPages<DownloadrClientdeffetch(chapter_link:String)html=get(chapter_link)nodes=html.xpath_nodes("//div[@id='selectpage']//select[@id='pageMenu']//option")nodes.map{|node|"#{chapter_link}/#{node.text}"}endendend
![Page 114: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/114.jpg)
moduleCrMangaDownloadrclassPages<DownloadrClientdeffetch(chapter_link:String)html=get(chapter_link)nodes=html.xpath_nodes("//div[@id='selectpage']//select[@id='pageMenu']//option")nodes.map{|node|"#{chapter_link}/#{node.text}"}endendend
moduleMangaDownloadrclassPages<DownloadrClientdeffetch(chapter_link)getchapter_linkdo|html|nodes=html.xpath("//div[@id='selectpage']//select[@id='pageMenu']//option")nodes.map{|node|[chapter_link,node.children.to_s].join("/")}endendendend
![Page 115: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/115.jpg)
time bin/manga-downloadr -t
![Page 116: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/116.jpg)
time bin/manga-downloadr -t
19.77s user 10.65s system 33% cpu 1:31.69 total
![Page 117: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/117.jpg)
![Page 118: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/118.jpg)
![Page 119: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/119.jpg)
Ruby/Typhoeus(hydra_concurrency = 50) 41% CPU 1:24 min
![Page 120: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/120.jpg)
Ruby/Typhoeus(hydra_concurrency = 50) 41% CPU 1:24 min
Elixir 1.4.5 (@max_demand=50) 120% CPU 1:14 min
![Page 121: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/121.jpg)
Ruby/Typhoeus(hydra_concurrency = 50) 41% CPU 1:24 min
Elixir 1.4.5 (@max_demand=50) 120% CPU 1:14 min
Crystal 0.23.0 (opt_batch_size = 50) 14% CPU 1:26 min
![Page 122: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/122.jpg)
Ruby/Typhoeus(hydra_concurrency = 50) 41% CPU 1:24 min
Elixir 1.4.5 (@max_demand=50) 120% CPU 1:14 min
Crystal 0.23.0 (opt_batch_size = 50) 14% CPU 1:26 min
Ruby 2.4.1 (opt_batch_size = 50) 33% CPU 1:31 min
![Page 123: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/123.jpg)
![Page 124: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/124.jpg)
Ruby Typhoeus libcurl
![Page 125: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/125.jpg)
Ruby Typhoeus libcurl
Elixir OTP Poolboy
![Page 126: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/126.jpg)
Ruby Typhoeus libcurl
Elixir OTP Poolboy
Crystal Fibers Fiberpool
![Page 127: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/127.jpg)
Ruby Typhoeus libcurl
Elixir OTP Poolboy
Crystal Fibers Fiberpool
Ruby Thread Thread/Pool
![Page 128: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/128.jpg)
![Page 129: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/129.jpg)
![Page 130: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/130.jpg)
manga-downloadr
ex_manga_downloadr
cr_manga_downloadr
![Page 131: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/131.jpg)
manga-downloadr
ex_manga_downloadr
cr_manga_downloadr
fiberpool
cr_chainable_methods
chainable_methods
![Page 132: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/132.jpg)
![Page 133: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/133.jpg)
![Page 134: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/134.jpg)
![Page 135: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/135.jpg)
![Page 136: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/136.jpg)
![Page 137: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/137.jpg)
PREMATUREOPTIMIZATION
The Root of ALL Evil
![Page 139: A Journey through New Languages - Guru Sorocaba 2017](https://reader031.fdocuments.us/reader031/viewer/2022030317/5a66b2ac7f8b9a3c0e8b49d1/html5/thumbnails/139.jpg)
www.theconf.club