Data Loading Made Easy with Mike Nakhimovich DroidCon Italy 2017

Click here to load reader

  • date post

    28-Jan-2018
  • Category

    Technology

  • view

    597
  • download

    3

Embed Size (px)

Transcript of Data Loading Made Easy with Mike Nakhimovich DroidCon Italy 2017

  1. 1. StoreDataLoadingMadeEasy-ish Mike Nakhimovich - NY Times
  2. 2. We Making Apps
  3. 3. We Open Source
  4. 4. Why?Open Source Makes Life Easier
  5. 5. Networking is easier: HTTPURLCONNECTION Volley Retrot/ Okhttp
  6. 6. Storage is Easier SharedPreferences Firebase Realm SqlBrite/SqlDelight
  7. 7. Parsing is Easier JSONObject Jackson Gson Moshi
  8. 8. Fetching, Persisting & Parsing datahas become easy
  9. 9. What's NOT Easy? DATALOADINGEveryone does itdifferently
  10. 10. Whatis DataLoading? TheActofgetting datafroman external systemtoauser'sscreen
  11. 11. Loading is complicated Rotation Handling isa special snowake
  12. 12. NewYorkTimes builtStoreto simplifydataloading github.com/NYTimes/store
  13. 13. OUR GOALS
  14. 14. Datashould survive conguration changes agnostic ofwhere itcomes from or howitis needed
  15. 15. Activitiesand presenters should stopretaining MBs ofdata
  16. 16. Ofine as conguration Cachingas standard, notan exception
  17. 17. API should be simple enough for an internto use,yet robust enough for everydataload.
  18. 18. Howdowework with DataatNY Times?
  19. 19. 80% case: Need data, don'tcare iffresh or cached
  20. 20. Need fresh data background updates & pull to refresh
  21. 21. Requests needto be async & reactive
  22. 22. Datais dependenton each other, DataLoading should betoo
  23. 23. Performance is importantNewcalls should hook into in ight responses
  24. 24. Parsing should be done efcientlyand minimallyParse onceand cachethe resultfor future calls
  25. 25. Let's Use Repository PatternBy creating reactiveand persistent DataStores
  26. 26. Repository Pattern Separatethe logicthatretrievesthe dataand maps ittothe [view] model fromthe [view] logicthatacts onthe model. The repositorymediates betweenthe datalayerandthe [view] layer ofthe application.
  27. 27. WhyRepository? Maximizetheamountof codethatcan betested withautomation by isolatingthe datalayer
  28. 28. Why Repository? Makes iteasierto have multiple devsworking on same feature
  29. 29. Why Repository? Datasource from many locationswillbe centrally managedwith consistent access rulesand logic
  30. 30. Our Implementation https://github.com/NYTimes/Store
  31. 31. WhatisaStore? Aclassthatmanagesthe fetching, parsing,and storage ofaspecic data model
  32. 32. Tell a Store: Whatto fetch
  33. 33. Tell a Store: Whatto fetch Whereto cache
  34. 34. Tell a Store: Whatto fetch Whereto cache Howto parse
  35. 35. Tell a Store: Whatto fetch Whereto cache Howto parse The Store handles ow
  36. 36. Storesare Observable Observable get( V key); Observable fetch(V key) Observable stream() void clear(V key)
  37. 37. Let's see howstores helped usachieve our goalsthrougha PizzaExample
  38. 38. Getis for Handling our 80% Use Case public nal class PizzaActivity { Store pizzaStore; void onCreate() { store.get("cheese") .subscribe(pizza ->.show(pizza)); } }
  39. 39. Howdoes itow?
  40. 40. On Conguration Change You onlyneedto Save/Restoreyour !to get your cached " public nal class PizzaActivity { void onRecreate(String topping) { store.get(topping) .subscribe(pizza ->.show(pizza)); } } Please do notserializeaPizza Itwould beverymessy
  41. 41. Whatwe gain?:Fragments/Presenters don'tneedto be retained NoTransactionTooLargeExceptions Onlykeys needto be Parcelable
  42. 42. Efciencyis Important: for(i=0;i getView() .setData(pizza)); } ManyconcurrentGetrequestswillstillonly hitnetwork once
  43. 43. Fetching NewDatapublic class PizzaPresenter { Store store; void onPTR() { store.fetch("cheese") .subscribe(pizza -> getView().setData(pizza)); } }
  44. 44. Howdoes Store routethe request? Fetchalsothrottles multiple requestsand broadcastresult
  45. 45. Stream listens for events public class PizzaBar { Store store; void showPizzaDone() { store.stream() .subscribe(pizza -> getView().showSnackBar(pizza)); } }
  46. 46. How do We build a Store? By implementing interfaces
  47. 47. Interfaces Fetcher{ Observable fetch(Key key); } Persister{} Observable read(Key key); Observable write(Key key, Raw raw); } Parser extends Func1{ Parsed call(Raw raw); }
  48. 48. Fetcher denes how a Store will get new data Fetcher pizzaFetcher = new Fetcher() { public Observable fetch(String topping) { return pizzaSource.fetch(topping); };
  49. 49. Becoming Observable Fetcher pizzaFetcher = topping -> Observable.fromCallable(() -> client.fetch(topping));
  50. 50. Parsers helpwith Fetchersthatdon't returnViewModels
  51. 51. Some ParsersTransform Parser boxParser = new Parser() { public PizzaBox call(Pizza pizza) { return new PizzaBox(pizza); } };
  52. 52. Others read Streams Parser parser = source -> { InputStreamReader reader = new InputStreamReader(source.inputStream()); return gson.fromJson(reader, Pizza.class); }
  53. 53. MiddleWare is forthe commonJSON cases Parser parser = GsonParserFactory.createSourceParser(gson,Pizza.class) Parser parser = GsonParserFactory.createReaderParser(gson,Pizza.class) Parser parser = GsonParserFactory.createStringParser(gson,Pizza.class) 'com.nytimes.android:middleware:CurrentVersion' 'com.nytimes.android:middleware:jackson:CurrentVersion' 'com.nytimes.android:middleware-moshi:CurrentVersion'
  54. 54. Data FlowwithaParser
  55. 55. Adding Ofine with Persisters
  56. 56. File System Record Persister FileSystemRecordPersister.create( leSystem,key -> "pizza"+key, 1, TimeUnit.DAYS);
  57. 57. File System is KISS storage interface FileSystem { BufferedSource read(String var) throws FileNotFoundException; void write(String path, BufferedSource source) throws IOException; void delete(String path) throws IOException; } compile 'com.nytimes.android:lesystem:CurrentVersion'
  58. 58. Don'tlike our persisters? No Problem! You can implement your own
  59. 59. Persister interfaces Persister { Observable read(Key key); Observable write(Key key, Raw raw); } Clearable { void clear(Key key); } RecordProvider { RecordState getRecordState( Key key);
  60. 60. DataFlowwithaPersister
  61. 61. We have our components! Let's buildand openastore
  62. 62. MultiParser List parsers=new ArrayList(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));
  63. 63. Store Builder List parsers=new ArrayList(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder parsedWithKey()
  64. 64. Add Our Parser List parsers=new ArrayList(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping))
  65. 65. NowOur Persister List parsers=new ArrayList(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .persister( FileSystemRecordPersister.create( leSystem, key -> "prex"+key, 1, TimeUnit.DAYS))
  66. 66. And Our Parsers List parsers=new ArrayList(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .persister( FileSystemRecordPersister.create( leSystem, key -> "prex"+key, 1, TimeUnit.DAYS)) .parsers(parsers)
  67. 67. Timeto OpenaStore List parsers=new ArrayList(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); Store pizzaStore = StoreBuilder parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .persister( FileSystemRecordPersister.create( leSystem, key -> "prex"+key, 1, TimeUnit.DAYS)) .parsers(parsers) .open();
  68. 68. Conguring caches
  69. 69. Conguring MemoryPolicy StoreBuilder .parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .memoryPolicy(MemoryPolicy .builder() .setMemorySize(10) .setExpireAfter(24) .setExpireAfterTimeUnit(HOURS) .build()) .open()
  70. 70. Refresh On Stale - BackFillingthe Cache Store pizzaStore = StoreBuilder .parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .parsers(parsers) .persister(persister) .refreshOnStale() .open();
  71. 71. Network Before Stales Policy Store pizzaStore = StoreBuilder .parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .parsers(parsers) .persister(persister) .networkBeforeStale() .open();
  72. 72. Stores inthewild
  73. 73. Intern Project: BestSellers List public interface Api { @GET Observable getBooks(@Path("category") String category);
  74. 74. @Provides @Singleton Store provideBooks(FileSystem leSystem, Gson gson, Api api) { return StoreBuilder. parsedWithKey() .fetcher(category -> api.getBooks(category)) .persister(SourcePersisterFactory.create(leSystem)) .parser(GsonParserFactory.createSourceParser(gson, Books.class)) .open(); }
  75. 75. public class BookActivity{ ... onCreate(...){ bookStore .get(category) .subscribe(); }
  76. 76. Howdo books getupdated? public class BackgroundUpdater{ ... bookStore .fresh(category) .subscribe(); }
  77. 77. DataAvailablewhen screen needs it UI calls get, background services call fresh
  78. 78. DependentCalls
  79. 79. Simple case Using RxJavaOperators MapStore resulttoanother Store Result public Observable getSectionFront(@NonNull nal String name) { return feedStore.get() .map(latestFeed -> latestFeed.getSectionOrBlog(name)) .atMap(section -> sfStore.get(SectionFrontId.of(section))); }
  80. 80. More Complicated Example Video Store Returns singlevideos, PlaylistStore returns playlist, Howdowe combine?
  81. 81. Relationships by overriding Get/Fetch
  82. 82. Step1: CreateVideo Store Store provideVideoStore(VideoFetcher fetcher, Gson gson) { return StoreBuilder.parsedWithKey() .fetcher(fetcher) .parser(GsonParserFactory .createSourceParser(gson, Video.class)) .open(); }
  83. 83. Step 2: Create PlaylistStore public class VideoPlaylistStore extends RealStore { nal Store videoStore; public VideoPlaylistStore(@NonNull PlaylistFetcher fetcher, @NonNull Store videoStore, @NonNull Gson gson) { super(fetcher, NoopPersister.create(), GsonParserFactory.createSourceParser(gson, Playlist.class)); this.videoStore = videoStore; }
  84. 84. Step 3: Override store.get() public class VideoPlaylistStore extends RealStore { nal Store videoStore; @Override public Observable get(Long playlistId) { return super.get(playlistId) .atMap(playlist -> from(playlist.videos()) .concatMap(video -> videoStore.get(video.id())) .toList() .map(videos -> builder().from(playlist) .videos(videos) .build())); }
  85. 85. Listening for changes
  86. 86. Step1: Subscribeto Store, Filterwhatyou need sectionFrontStore.subscribeUpdates() .observeOn(scheduler) .subscribeOn(Schedulers.io()) .lter(this::sectionIsInView) .subscribe(this::handleSectionChange); Step2: Kick offafresh requesttothatstore public Observable refreshAll(nal String sectionName) { return feedStore.get() .atMapIterable(this::updatedSections) .map(section->sectionFrontStore.fetch(section)) latestFeed -> fetch(latestFeed.changedSections)); }
  87. 87. Bonus: Howwe made Store
  88. 88. RxJavaforAPIand FunctionalNiceties public Observable get(@Nonnull nal Key key) { return Observable.concat( cache(key), fetch(key) ).take(1); } public Observable getRefreshing(@Nonnull nal Key key) { return get(key) .compose(repeatOnClear(refreshSubject, key)); }
  89. 89. GuavaCaches For MemoryCache memCache = CacheBuilder .newBuilder() .maximumSize(maxSize()) .expireAfterWrite(expireAfter(),timeUnit()) .build();
  90. 90. In FlightThrottlerToo inighter.get(key, new Callable() { public Observable call() { return response(key); } }) compile 'com.nytimes.android:cache:CurrentVersion' Why GuavaCaches? Theywillblockacrossthreads saving precious MBs ofdownloads when making same requests in parallel
  91. 91. FileSystem: Builtusing OKIOtoallowstreaming from Networkto disk & diskto parser On Fresh InstallNYTimesappsuccessfullydownloadsand caches dozens ofmbs ofdata
  92. 92. Would love contributions & feedback!thanks for listening