What Open Data and Open Source can do for Sri

Lanka?

Srinath Perera, Ph.DVP Research, WSO2

Member, Apache Foundation@srinath_perera

A Day in Your Life

Success Stories• Money Ball ( Baseball drafting) • Nate Silver predicted outcomes

in 49 of the 50 states in the 2008 U.S. Presidential election

• Cancer detection from Biopsy cells ( Big Data find 12 patterns while we only knew 9), http://go.ted.com/CseS

• Bristol-Myers Squibb reduced the time it takes to run clinical trial simulations by 98%

• Xerox used big data to reduce the attrition rate in its call centers by 20%.

• Kroger Loyalty programs ( growth in 45 consecutive quarters)

If you collect data about your business, and feed it to a Big Data system, you will find useful insights

that will provide competitive advantage– (e.g. Analysis of data sets can find new correlations to

"spot business trends, prevent diseases, combat crime and so on”. [Wikipedia])

Putting Analytics to Work

What happened? And Why? ( Hindsight)

What is Happening right now? ( oversight)

What will happen? (Foresight)

Open Source Market Share

• Apache (60%)• Linux (Servers 16%)• Firefox (25%)• Tomcat and most of

middleware • Android (43%)• Even Microsoft

looking favorably at Opensource projects

• There are lot of open source projects bundled inside the proprietary products

Gartner Predicted that by 2012 most systems will use open source

components Copyright kafka4prez and licensed for reuse under CC License ,

http://www.flickr.com/photos/kafka4prez/198465913

What is Open Source?• Most commercial

software does not distribute the source code, and developed and managed in a closed world.

• Idea of open source is to have the code in the open, and to improve it though volunteer contributions using “open development”

• Idea is that the project becomes a eco-system– More ideas– More developers– More Testers– More Bug fixers

“There is no delight in owning anything

unshared.” Seneca (Roman philosopher,

mid-1st century AD)

How does a Open Source Work?• Open code repository (SVN

or Git etc.)• Two parts of the community

– Developer Community– User Community

• Communication through Mailing lists / IRC Channel– Develop mailing list– User mailing list

• Bug tracking database to track errors (Jira, Bugzilla)

• People submit improvements as patches through Jira etc.

Committers have write access to repository Committers review and apply patches, and when you

submit lot of them, they will make you a committer.

History of Opensource• 1970s – UNIX, Emacs• 1984-85 - GNU project

and Free Software Foundation

• 1990 - GNU project almost complete .. well not OS

• 1991 - Linus Torvalds announce Linux, Phython

• 1993 - Net BSD and Free BSD

• 1994-95 - Linux 1.0 released

• 1995 - Apache, KDE, PHP

• 1997 - Genome• 1999 Linux 2.2,

OpenOffice• 2003 - Firefox, Android

http://www.geograph.org.uk/photo/916456http://www.fotopedia.com/items/flickr-3320704544

Why People Contribute?• As a way to improve your

profile (looking for a Job)• To gain experience• To work with “like minded”

People• To be part of something bigger• To be a “Geek”• As a Job – if you a well known

open source developer, chances are that you will get payed for contribution

• As a competitive strategy

Great investments need faith and patience

Copyright U. S. Fish and Wildlife Service and licensed for reuse under CC License , http://www.flickr.com/photos/usfwsnortheast/4754624921 and Copyright WxMom and licensed

for reuse under CC License , http://www.flickr.com/photos/wxmom/1359996991.

• Sahahna• Apache Axis2 and

other projects

http://www.geograph.org.uk/photo/1842872

LKA Success Stories

Why People use Open Source Software?

• It is cheaper• It is better• Because it is open source

(Religiously)• More visibility into the code, better

security, auditing • If there is a problem, I can fix it • More control over releases, roadmap• Patches become available faster • Easy to understand how it works• Can fork the code if needed • Not own by one person, less risk to

depend on it. • Do not have to maintain the code

Big Data and Opensource Most Big data tools are

free Even the state of the

art is being released as opensource

Give countries like a unique opportunity with a level playing field

Open Data

Make the data public Advanced form of the RTI act Opensource idea applied to data

science E.g. programs like “Code for America”

Code Red: US healthcare.gov Rescue

$300M project, that is failing and small group of volunteers go to hackathon mode to fix it, and fix it.

See http://radar.oreilly.com/

2014/03/code-red_-they-have-no-use-for-someone-who-looks-and-dresses-like-me.html

http://content.time.com/time/magazine/article/0,9171,2166770-1,00.html

Filtering Information with Big Data Big Data can filter

information (e.g. SPAM)

Rank Information ( show most relevant articles)

Find Anomalies ( detect Fraud)

Make recommendations ( product recommendations)

Handle reputations (e.g. Ebay, Amazon)

George Caleb Bingham, 1846

Example: Reddit, Hacker News( Ranking)

Keep Your Customers

Get New Customers

Improve Operations

Monetize your data

Example: Amazon, Ebay, Abnb, (reputation)

Urban Planning and Policy Decisions• Urban Planning

– People distribution – Mobility – Waste Management– Parking

• Policy Decision– What if we change

minimum wage? – What are economic

impact of a new law?

By Aqwis - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=6810430

Example: Big Data for Development• Done using CDR data• People density noon vs.

midnight (red => increased, blue => decreased)

From: http://lirneasia.net/2014/08/what-does-big-data-say-about-sri-lanka/

Traffic Lot of us waste time

on traffic Know where is

traffic ( Google traffic does that)

Emergency Response

Know the traffic patterns

Long term planning

Manage Donors and Charities

Sri Lanka donates a lot (even the poorest)

Does the money goes to intended place Can we track how money is spent?

https://iwringer.files.wordpress.com/2015/09/traffic2.jpg?w=656

Day to day Maintenance

Does the news papers are the best way to get day to day things done?

Can crowd sourcing help? How to stop false tickets?

Disease spread Earlier Malaria and now dengue Know current situation Know overall trends ( focus on

problematic areas) Emergency Response

Summary• There are lot Opensource,

Open data, and Big Data can do for Sri Lanka

• Some cases needs money!! And might be beyond us

• But not for many cases – e.g. Sahana– Hackathon to build an app

to decide what topics to take up in the parliament

• What we really need is collaborations between domain experts and computer scientists