[Hack.Hydrosphere] Project TIDE
-
Upload
bulat-lutfullin -
Category
Data & Analytics
-
view
221 -
download
0
Transcript of [Hack.Hydrosphere] Project TIDE
Team Q-shke-Q
Nikitafront-end
javascript(GoJS) coderhates bad docs
Iskandarback-end
Apache Spark guruPython coder
Bulatteamleadloves Scala anddoesn’t know it
Alexeyfront-endloves Node.js
team’s DJ
But...
Apache Spark Mist Web app
'A data scientist is someone who is better at statistics than any software engineer and better at software engineering than any statistician.' https://www.quora.com/Do-data-scientists-code
Scala, Python, code
WTH is this?
TIDEVisual jobs constructor for
Apache SparkProvides an abstraction over code to make life simpler
Still need to use code snippets in Python
Roadmap
● Auth system● Custom data node● R code snippets support● DataFrames instead of RDD● Job as a subprocedure● Jobs management● UI/UX improvements● Static analysis of data flow● Rewrite backend to Scala
(maybe)● … just for now
Front-end● websockets● Flask● Data flow to pySpark compiler
Back-end● websockets● GoJS● materialize.css● JQuery