Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski
-
Upload
institute-of-contemporary-sciences -
Category
Data & Analytics
-
view
23 -
download
0
Transcript of Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski
![Page 1: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/1.jpg)
Data Science Conference11-12 October 2016Belgrade, Serbia
![Page 2: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/2.jpg)
![Page 3: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/3.jpg)
Aircraft challenge
Marko, don’t forget to show
the aircraft to the audience
![Page 4: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/4.jpg)
![Page 5: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/5.jpg)
Dr David Warren
![Page 6: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/6.jpg)
3400 G shock for 6.5 ms
500 lb. Dropped from 10 ft with a ¼-inch-diameter contact point
1100 ºC flame for 30 minutes.
260 ºC for 10 hours
Immersion in aircraft fluids for 24 hours
Immersion in sea water for 30 days
5,000 pounds crush for 5 minutes on each axis
Pressure equivalent to depth of 20,000 ft.
![Page 7: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/7.jpg)
![Page 8: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/8.jpg)
![Page 9: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/9.jpg)
Avoiding Black Boxes with Data ScienceRaffaele Rainone & Marko Vasiljevski
Data Science Conference11-12 October 2016Belgrade, Serbia
![Page 10: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/10.jpg)
Hkjk;l
Dr Raffaele Rainone
![Page 11: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/11.jpg)
Hkjk;l“I don’t know…, pure mathematician, Python developer, data scientist, pizza lover, feeling a bit home sick for Italy, …”
![Page 12: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/12.jpg)
Chatting about…
Aviation Safety
A story about flight data monitoring (FDM)A story of a flight data analyst
A story of a flight safety statistician
![Page 13: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/13.jpg)
![Page 14: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/14.jpg)
Chatting about…
Data science in (a bit of) action
Jet-engine health - Working with Mr. BayesDetecting safety concerns – PCA, a friend
Finding cuckoo’s eggs – Mr. Markov’s chains
![Page 15: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/15.jpg)
Flight Data Monitoring (FDM)
![Page 16: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/16.jpg)
Flight Data Monitoring (FDM)
![Page 17: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/17.jpg)
Flight Data Monitoring (FDM)
![Page 18: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/18.jpg)
Flight Data Monitoring (FDM) what?!
Part of safety management system (SMS)Airlines worldwide obliged to do FDMSpotting deviations from safe operationNon-punitive – learn from mistakesConfidential
![Page 19: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/19.jpg)
![Page 20: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/20.jpg)
Quick Access Recorder
Flight Data Recorder
Data Acquisition Unit
![Page 21: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/21.jpg)
![Page 22: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/22.jpg)
![Page 23: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/23.jpg)
![Page 24: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/24.jpg)
002AF5C0 00 00 76 07 04 01 64 08 06 01 3C 0C F2 0E 3C 0C ..v...d...<.ò.<.002AF5D0 00 00 58 02 02 0C 00 0C 48 08 00 00 00 00 00 0C ..X.....H.......002AF5E0 00 0C 00 00 00 00 E0 0F 00 00 00 0C 40 0C 00 0C ......à[email protected] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90 01 ...............�002AF600 47 02 77 07 03 00 2B 0E DF 0F 02 0C A5 0F 78 00 G.w...+.ß...¥.x.002AF610 55 04 D3 01 D8 0F EA 03 A4 0F EE 0F FD 0F 00 00 U.Ó.Ø.ê.¤.î.ý...002AF620 08 00 10 00 00 00 08 00 11 00 00 00 06 00 1E 00 ................002AF630 F8 0F 10 00 00 00 8C 0B 0C 00 0A 00 00 00 00 00 ø.....Œ.........002AF640 00 00 77 07 96 0F 64 08 06 01 3C 0C F2 0E 08 08 ..w.–.d...<.ò...002AF650 00 00 00 00 01 00 AC 00 4C 08 01 08 00 00 00 00 ......¬.L.......002AF660 00 00 04 02 D0 02 0D 02 12 02 00 0B 44 01 00 00 ....Ð.......D...002AF670 A4 06 40 01 03 00 00 00 40 02 00 00 00 00 00 00 ¤.@[email protected] 02 00 77 07 00 00 2B 0E DF 0F 04 0C A5 0F 00 08 ..w...+.ß...¥...002AF690 54 04 D4 01 DA 0F EA 03 A4 0F 73 0E 00 00 01 00 T.Ô.Ú.ê.¤.s.....002AF6A0 08 00 10 00 01 00 08 00 00 00 03 00 06 00 1D 00 ................002AF6B0 FB 0F 11 00 02 00 01 00 00 00 03 00 00 00 00 00 û...............002AF6C0 01 00 76 07 00 00 64 08 06 01 3C 0C F2 0E 3C 0C ..v...d...<.ò.<.002AF6D0 6A 02 21 03 03 00 00 00 48 08 00 00 00 00 00 00 j.!.....H.......002AF6E0 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 00 ................002AF6F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 FC 0F ..............ü.002AF700 CC 08 77 07 03 00 2B 0E DF 0F 02 0C A5 0F 79 01 Ì.w...+.ß...¥.y.002AF710 54 04 D4 01 D8 0F EA 03 A7 0F EE 0F FE 0F 01 00 T.Ô.Ø.ê.§.î.þ...002AF720 08 00 00 00 00 00 00 00 11 00 02 00 06 00 00 00 ................002AF730 F8 0F 00 00 00 00 8C 0B 00 00 00 00 00 00 00 00 ø.....Œ.........002AF740 00 00 76 07 94 0F 64 08 06 01 3C 0C F2 0E 08 08 ..v.”.d...<.ò...002AF750 9A 0C 18 00 61 0C 68 01 48 08 F8 0F A4 0C 00 00 š...a.h.H.ø.¤...002AF760 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5B 02 ..............[.002AF770 00 00 03 02 00 00 2B 0F F0 0F 2A 0F 00 00 00 00 ......+.ð.*.....002AF780 00 00 77 07 00 00 2B 0E DF 0F 04 0C A5 0F 2C 00 ..w...+.ß...¥.,.002AF790 54 04 D4 01 D8 0F EB 03 A6 0F 00 0D 00 00 01 00 T.Ô.Ø.ë.¦.......002AF7A0 08 00 17 00 03 00 0F 00 00 00 00 00 06 00 1C 00 ................002AF7B0 F8 0F 10 00 02 00 01 00 00 00 03 00 00 00 00 00 ø...............002AF7C0 00 00 76 07 11 00 64 08 06 01 3C 0C F2 0E 3C 0C ..v...d...<.ò.<.
What now?Data frames
![Page 25: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/25.jpg)
![Page 26: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/26.jpg)
Questions, suggestions, congestions?
![Page 27: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/27.jpg)
The story of a flight data analyst
![Page 28: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/28.jpg)
![Page 29: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/29.jpg)
![Page 30: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/30.jpg)
![Page 31: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/31.jpg)
![Page 32: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/32.jpg)
![Page 33: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/33.jpg)
![Page 34: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/34.jpg)
![Page 35: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/35.jpg)
![Page 36: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/36.jpg)
Questions, suggestions, congestions?
![Page 37: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/37.jpg)
The story of a flight safety statistician
![Page 38: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/38.jpg)
The story of a flight safety statistician
STATISTICS
![Page 39: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/39.jpg)
How do we monitor safety in FDM
012345678
Event Count 3.141592653589793238462643383384
![Page 40: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/40.jpg)
And…
Domain knowledgeExperience
Common sense
![Page 41: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/41.jpg)
Uncertainty - good companion
Normal acceleration - Tdwn Normal acceleration – Lift-off
Statistic ValueTotal count 80,663
Average 1.31(Min, Max) (1.03,
2.40)Range 1.37
Standard deviation 0.10
Statistic ValueTotal count 80,663
Average 1.19(Min, Max) (0.66,
1.63)Range 0.97
Standard deviation 0.05
Wider histogram,
less confidence
in mean value
Narrowerhistogram,
more confidence
in mean value
![Page 42: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/42.jpg)
How apples relate to flight safety
![Page 43: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/43.jpg)
Micromanaging events
0369
Top 5 Events – January 2016
![Page 44: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/44.jpg)
Micromanaging events
High sp
eed t
axiing
High la
teral
g tax
iing
Flap o
versp
eed
Airspe
ed hi
gh 10
000-5
000 f
t
Exces
sive b
reakin
gPu
ll up
Stick
shake
r0369
Top 7 Events – January 2016
![Page 45: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/45.jpg)
Most severe safety event – a riddle
It doesn’t happen in the air
It doesn’t happen at the gate
It happens in the office
![Page 46: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/46.jpg)
Micromanaging events
High sp
eed t
axiing
High la
teral
g tax
iing
Flap o
versp
eed
Airspe
ed hi
gh 10
000-5
000 f
t
Exces
sive b
reakin
gPu
ll up
Stick
shake
r0369
Top 7 Events – January 2016
![Page 47: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/47.jpg)
Not looking at your data!
![Page 48: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/48.jpg)
Trending?
![Page 49: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/49.jpg)
Correlation?
![Page 50: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/50.jpg)
Questions, suggestions, congestions?
![Page 51: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/51.jpg)
Enough, it’s a data science conference!
![Page 52: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/52.jpg)
Improving current state
Jet-engine health with Mr. Bayes
![Page 53: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/53.jpg)
![Page 54: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/54.jpg)
![Page 55: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/55.jpg)
Engine performance monitoring
N1 N2
Fuel
Flow
Exhaust Gas
Temperature
![Page 56: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/56.jpg)
Engine performance monitoring
Fuel
Flow
Exhaust Gas
Temperature
![Page 57: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/57.jpg)
Engine performance monitoring
![Page 58: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/58.jpg)
End of take-off flow
![Page 59: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/59.jpg)
End of take-off flow
![Page 60: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/60.jpg)
End of take-off flow
![Page 61: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/61.jpg)
End of take-off flow
![Page 62: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/62.jpg)
End of take-off flow
![Page 63: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/63.jpg)
End of take-off flow
![Page 64: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/64.jpg)
End of take-off flow
![Page 65: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/65.jpg)
End of take-off – a problem
If not detected – problems with engine healthDifficult to do with classical signal processing
What if a parameter fails (not so rarely)
![Page 66: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/66.jpg)
End of take-off – a solution
Time at vertical navigation mode selected (VNAV)Drop/rise in fuel flow (FF) and gas temperature (EGT)Search for changes around that point in timeGather the knowledgeCalculate probability for EOT for unseen flight
![Page 67: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/67.jpg)
End of take-off – a solution
Time at vertical navigation mode selected (VNAV)Drop/rise in fuel flow (FF) and gas temperature (EGT)Search for changes around that point in timeGather the knowledgeCalculate probability for EOT for unseen flight
![Page 68: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/68.jpg)
End of take-off – a solution
![Page 69: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/69.jpg)
End of take-off – a solution
![Page 70: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/70.jpg)
End of take-off – a solution
![Page 71: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/71.jpg)
End of take-off – a solution
![Page 72: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/72.jpg)
End of take-off – a solution
If crazy about this, search for PyData London 2016 videoM. Vasiljevski & R. Rainone
“Python flying at 40,000 feet”
![Page 73: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/73.jpg)
Questions, suggestions, congestions?
![Page 74: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/74.jpg)
Finding new concerns
Novelty detection with principal component analysis
![Page 75: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/75.jpg)
![Page 76: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/76.jpg)
![Page 77: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/77.jpg)
![Page 78: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/78.jpg)
Standardising
mean = 0standard deviation = 1
![Page 79: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/79.jpg)
![Page 80: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/80.jpg)
![Page 81: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/81.jpg)
![Page 82: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/82.jpg)
Formally…
Previous 700 x 123 matrix is A Multiply transpose of A with A to get covariance matrix, C.
It’s 123 x 123 Do singular value decomposition of C to find eigenvectors
and eigenvalues Eigen vectors coincide with directions of highest variance Keep just first couple of vectors to reduce dimensionality
![Page 83: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/83.jpg)
![Page 84: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/84.jpg)
![Page 85: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/85.jpg)
![Page 86: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/86.jpg)
![Page 87: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/87.jpg)
![Page 88: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/88.jpg)
![Page 89: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/89.jpg)
![Page 90: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/90.jpg)
![Page 91: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/91.jpg)
![Page 92: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/92.jpg)
Questions, suggestions, congestions?
![Page 93: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/93.jpg)
Helping business
Flight data upload monitor based on Markov chains
![Page 94: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/94.jpg)
Flight data upload data data data
![Page 95: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/95.jpg)
Flight data upload data data data
![Page 96: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/96.jpg)
Flight data upload data data data
![Page 97: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/97.jpg)
Flight data upload data data data
![Page 98: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/98.jpg)
Flight data upload data data data
![Page 99: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/99.jpg)
Flight data upload data data data
State1 = M x State0State2 = M x State1 = M x (M x State0) = M2
x State0StateN = MN x State0
Don’t have to know states, just powers of M (probabilities)
![Page 100: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/100.jpg)
Flight data upload data data data
![Page 101: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/101.jpg)
Flight data upload data data data
![Page 102: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/102.jpg)
Flight data upload data data data
![Page 103: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/103.jpg)
Flight data upload data data data
![Page 104: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/104.jpg)
Flight data upload data data data
![Page 105: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/105.jpg)
Flight data upload data data data
![Page 106: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/106.jpg)
Questions, suggestions, congestions?
![Page 107: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/107.jpg)
Credits
![Page 108: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/108.jpg)
![Page 109: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/109.jpg)
CHRIS JESSE
![Page 110: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/110.jpg)
RAFFAELE (PIZZA) RAINONE
![Page 111: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/111.jpg)
MARTA
VASI
LJEV
SKI
![Page 112: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/112.jpg)
MILA
N
BOROTA
![Page 113: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/113.jpg)
MILIC
A IVA
NIŠEV
IĆ+ foeniculumvulgare
![Page 114: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/114.jpg)
ZIZI(paid for my tickets)
![Page 115: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/115.jpg)
Main takeaways from this talk
![Page 116: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/116.jpg)
LEARN
![Page 117: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/117.jpg)
HAVE FUN
![Page 118: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/118.jpg)
SHARE KNOWLEDGE
![Page 119: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/119.jpg)
THINK
![Page 120: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/120.jpg)
PLAY
![Page 121: Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski](https://reader033.fdocuments.us/reader033/viewer/2022051101/586f79bb1a28ab10258b705f/html5/thumbnails/121.jpg)
THANK YOU & SAFE FLYING