ITANA 2016: API Architecture and Implementation
-
Upload
colin-bell -
Category
Technology
-
view
120 -
download
0
Transcript of ITANA 2016: API Architecture and Implementation
ITANA API / Governance Working Group
U N I V E R S I T Y O F W AT E R L O O
API ARCHITECTURE AND IMPLEMENTATIONP O W E R I N G S T U D E N T A N D I N S T I T U T I O N A L I N N O VAT I O N
Colin BellDirector Enterprise ArchitectureEA, ISTBMath2008uwat.ca/ist-ea
Zachary SeguinOpen Data API Incoming Dev Lead Client Services, ISTBCS Computer Science2017uwaterloo.ca/open-data
Kartik TalwarOpen Data API Outgoing Dev Lead Client Services, ISTBSc Physics2016uwaterloo.ca/open-data
Nate VexlerOpen Data API Service LeadClient Services, ISTBASc Systems Design Eng.2012uwaterloo.ca/open-data
ITANA API / Governance Working Group
AGENDA
Background1. Prime Motivators2. Routine Disclosure3. Why Open Data First?4. 5 Star Data5. Open Data License6. History of Open Data
Technical Specifics7. Data Sets8. Data Acquisition9. Technology
ITANA API / Governance Working Group
AGENDA
Growing1. Governance and Policy2. Private Data (PbD)3. Business Process4. Data Warehousing/ESB/iPaaS5. Future Directions
Questions6. Buy vs. Build?7. How to build today?8. Documentation?9. How do you deal with the ERP
culture clash?
ITANA API / Governance Working Group
BACKGROUND
ITANA API / Governance Working Group
ORIGINAL PRIME MOTIVATORS
• Enable Student Development• Enable Homebrew Student Portal
ITANA API / Governance Working Group
PROTECTING PRIVACY
http://eaves.ca/2013/01/07/the-journal-news-gun-map-open-vs-personal-data/
ITANA API / Governance Working Group
MAINTAINING TRANSPARENCYPROTECTING PRIVACY
bit.ly/ipc_abd bit.ly/ipc_pbd
ITANA API / Governance Working Group
ROUTINE DISCLOSURE• Pol icy of Ontar io’s
Information Privacy Commissioner
• 7 pr inciples• bit . ly / ipc_abd
ITANA API / Governance Working Group
ROUTINE DISCLOSURE1. P r o a c t i v e , n o t R e a c t i v e2. A c c e s s E m b e d d e d i n t o D e s i g n3. O p e n n e s s a n d Tr a n s p a r e n c y =
A c c o u n t a b i l i t y4. F o s t e r s C o l l a b o r a t i o n5. E n h a n c e s E f f i c i e n t G o v e r n m e n t6. M a k e s A c c e s s Tr u l y A c c e s s i b l e7. I n c r e a s e s Q u a l i t y o f I n f o r m a t i o n
ITANA API / Governance Working Group
WHY OPEN DATA FIRST?
Personally Identifiable Data
ITANA API / Governance Working Groupbit.ly/ipc_abd
WHY OPEN DATA FIRST?
ITANA API / Governance Working Group
Emerging trend in Government (Toronto, Vancouver)
New Vertical (Higher Ed)
WHY OPEN DATA FIRST?
Open Data License
ITANA API / Governance Working Group
OPEN DATA
5stardata.info
ITANA API / Governance Working Group
IMPROVING OPEN DATA
inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/
ITANA API / Governance Working Group
WHY IS OPEN DATA IMPORTANT
• Economies of Scale • Value of Data • Cost of Using Data
ITANA API / Governance Working Group
WHY IS OPEN DATA IMPORTANT
DO MORE W/ LESS
Fall 2009• Nathan lobbies with
others at Student Technology Advisory Committee re Portal/Open Data
March 2010• Presentation at
High Level Computing Committees UCIST/CTSC
Fall 2010• Jeff Verkoeyen’s
uwdata.ca becomes official
Aug 2011• Kartik Talwar
creates api.youwaterloo.ca
Feb 2012• api.uwaterloo.ca
becomes official
Open Data @ uwaterloo Timeline
2012• Nathan Joins IST full time• Open-Data powered
apps:• - Student Portal (internal)
• - uwflow.com (external)
2013• V2 launches• Drupal-Powered Content
Management System is leveraged to deliver Open Data for use in Student Portal
2014• Open Data-powered
Student Portal launches;• mandate to bring
api.uwaterloo.ca deeper into the enterprise
2015• Campus Map Project
(powered by Open Data) is in development
• Student Portal project continues to further the momentum of Open Data
Open Data @ uwaterloo Timeline
ITANA API / Governance Working Group
IPC CORE DESIGN PRINCIPLES
bit.ly/ipc_abd bit.ly/ipc_pbd
Student PortalAbD/PbD Strategy(Colin Bell)
ITANA API / Governance Working Group
IMPROVING OPEN DATA
inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/
ITANA API / Governance Working Group
API: APPLICATION PROGRAMMING INTERFACE
ITANA API / Governance Working Group
API: APPLICATION PROGRAMMING INTERFACE
CORE AREAS OF SERVICE OF CURRENT API
Open Data API
Student Engagement
(Portal)
Core IT Service (WCMS)
For Students by Students
Emerging Academic
Use
uWaterlooGitHub
github.com/uwaterloo
IRL Community with
DemoCamps
Community Created Toolsuwaterloo.xyz
ITANA API / Governance Working Group
APPLIED EXAMPLE: RESPONDING TO COMMUNITY NEED
GOOSE WATCH
BACK to April 9, 2013
ITANA API / Governance Working Group
Three Days PRIORAppeal For Data
THE STUDENT SUCCESS OFFICE SWINGS IN ACTION TO PROTECT STUDENTS
Three Days PRIOR Sketch/Prototype
ITANA API / Governance Working Group
Mid Afternoon the Day BeforeThe endpoint launches
THE UWATERLOO COMMUNITY S W I N G S I N TO A C T I O N TO R E Q U E S T / D E L I V E R D ATA
Two Days PRIORThe reddit Data Request
ITANA API / Governance Working Group
THE RESPONSE
ITANA API / Governance Working Group
PROGRESS!
5stardata.info
API
PNG
ITANA API / Governance Working Group
TECHNICAL SPECIFICS
ITANA API / Governance Working Group
DATASETS
Food Services
News and Events CO-OP WATpark
New Campus Map Tutors Services Awards
ITANA API / Governance Working Group
Food ServicesLocations and Hours
Daily menuNutritional Information
ITANA API / Governance Working Group
CO-OP DataEmployer Info-Sessions
Career Centre Workshops
ITANA API / Governance Working Group
TutorsFind a Tutor for a courseSee their availability and
get help
ITANA API / Governance Working Group
WATParkSee live parking lot capacity
Get updates on opening and closing hours
ITANA API / Governance Working Group
GithubStudents Contributing Data
Students submitting datasets
ITANA API / Governance Working Group
New Campus Map
ITANA API / Governance Working Group
CURRENT STATE: DATA ACQUISITION
ScrapersGithub• Crowdsourced• Maintained
CMS (Drupal)• Proxy requests• Pub/sub
Other Proxy Requests
Direct Database
Integrations
ITANA API / Governance Working Group
FUTURE STATE: DATA ACQUISITION
ScrapersGithub• Crowdsourced• Maintained
CMS (Drupal)• Proxy requests• Pub/sub
Other Proxy Requests
Direct Database
Integrations
WarehousingESB / iPaaS
ITANA API / Governance Working Group
CURRENT STATE: DATASETS
Food ServicesDB
News/Events/etcPub Sub
CO-OP WATparkProxy
New Campus Map
Curated GH
TutorsProxy
AwardsProxy
ITANA API / Governance Working Group
TECHNICAL ARCHITECTURE
• Student initiated, no fancy ($) tech involved.
• After a number of iterative improvements, this is our stack.
api.uwaterloo.ca
App Server 1 App Server 2
Incoming Webhooks
Workers
Other Databases
Proxy ServicesAPI DB
App Server
Incoming Webhooks Proxy Services Workers
Github sync, scrapers, updaters
Live requests from other internal APIs
New posts from university websites
API DB Other DBs
App Server
API Request Router (PHP Klein)
Tutoring Services Food Services
FS DB API DB
GET METHOD 1GET METHOD 2GET METHOD 3
Serve from FS DB
Campus Events
GET METHOD 1GET METHOD 2GET METHOD 3
Proxied via internal tutoring API
Tutoring API (other department)
Crons
- sync datasets- run scrapers- cache things
Keep data on file system
GET Serve from DB
POST (webhook) Process and update DB
ITANA API / Governance Working Group
DOCUMENTATION / ISSUE TRACKING
• Docs: Github w/ Markdown generator• Issue Tracking
Community Issues: GithubService Desk: RTInternal: Gitlab/Jira
ITANA API / Governance Working Group
GROWING API CAPABILITY
ITANA API / Governance Working Group
GOVERNANCE AND POLICY
• Administrative Information Governance Committee (AIGC)
Vice Presidents, Senior Management, and Associate Provosts2015 decision to consolidate IM related Policies.Procedures and Guidelines to follow Policy. Draft of Policy being moved through review now.
ITANA API / Governance Working Group
PRIVATE DATA
• We have few private data APIs on campus. Faculty developed advising system provides student information backbone.
• They are starting to grow. Principle established prioritizing API integration pathways.
• Real uptake depends on Identity and Access Management implementations.
Shibboleth, SAML, OAUTH2 to come 2016/2017.
ITANA API / Governance Working Group
BUSINESS PROCESS
• Working to change the relationships.Initially we leaned on AbD, many units felt ’bad things’ were happening.Proven that the sky has not fallen.Developing a Steering Committee model to help direct future development merging community needs with institutional goals.Develop SLA with Information Stewards.
ITANA API / Governance Working Group
DATA WAREHOUSING / ENTERPRISE SERVICE BUS / IPAAS• Integrations on campus have been file or DB based.
Times have changed, we are starting to change.• Open Data has provided us with knowledge and
expertise in APIs.• As warehousing improves and we bring either an
ESB or iPaaS solution on campus, our data architecture will start to take shape.
ITANA API / Governance Working Group
DATA WAREHOUSING / ENTERPRISE SERVICE BUS / IPAAS
Data Warehousing
Integration Engine (ESB/iPaaS)
HR
Finance
SIS
Co-Op
…
OR
AbD(Open)
PbD(Priv)
API
ITANA API / Governance Working Group
FUTURE DIRECTIONS
• Composable Microservices• Create Once, Publish Everywhere (COPE) -
NPR Model
University of Waterloo WCMS Create Once Publish Everyone COPE Strategy(Andrew McAlorum)
News and Events
GROWING AREAS OF SERVICE
Open Data API
Student Engagement
(Portal)
Core IT Service (CMS + more)
For Students by Students
Emerging Academic
Use
Campus Map
Research
ITANA API / Governance Working Group
THE OPEN DATA ENGINE
(Demand-First)• Exam Schedule• CO-OP workshops• Tutors• Food Services
(Supply-First)• Services• News Events• Awards• Publication
ITANA API / Governance Working Group
[email protected]@[email protected]@uwaterloo.ca
Buy vs. Build?How to build today?
How do you Document?How to mitigate ERP culture clash?
QUESTIONS?