Developer Toolchain at UC Riverside

Mike KennedyEnterprise Architect

Developer Toolchain at UC Riverside

Mike KennedyEnterprise Architect

Application Deployment

About Me

About Me• Mike Kennedy

About Me• Mike Kennedy

• At UC Riverside since 1995 (Student then Staff)

About Me• Mike Kennedy

• At UC Riverside since 1995 (Student then Staff)

• Enterprise Architect

About Me• Mike Kennedy

• At UC Riverside since 1995 (Student then Staff)

• Enterprise Architect

• Interests lie in web services, application architecture, integration, middleware

About Me• Mike Kennedy

• At UC Riverside since 1995 (Student then Staff)

• Enterprise Architect

• Interests lie in web services, application architecture, integration, middleware

• Major projects - New SIS (Banner), UCPath, ServiceNow, New development environment, ESB

Talk Background

Talk Background• Up until recently, applications developed using

PL/SQL Web Toolkit and Oracle IAS

Talk Background• Up until recently, applications developed using

PL/SQL Web Toolkit and Oracle IAS

Talk Background• Up until recently, applications developed using

PL/SQL Web Toolkit and Oracle IAS

Talk Background• Up until recently, applications developed using

PL/SQL Web Toolkit and Oracle IAS

• Began effort to modernize: .NET vs. Java

Talk Background• Up until recently, applications developed using

PL/SQL Web Toolkit and Oracle IAS

• Began effort to modernize: .NET vs. Java

• Ultimately settled on Grails (Groovy/JVM based framework)

Talk Background• Up until recently, applications developed using

PL/SQL Web Toolkit and Oracle IAS

• Began effort to modernize: .NET vs. Java

• Ultimately settled on Grails (Groovy/JVM based framework)

• Opportunity: Improve on application delivery processes and architecture (my focus today)

Grails

Grails

Grails Application

Tomcat App Server

Java Virtual Machine (JVM)

Operating System

F5

app-prd-1.cis.ucr.edu app-prd-2.cis.ucr.edu

APPPROD

User 1

Tomcat Session Replication

User 2

Database Connection Datab

ase C

onne

ction

HTTP HTTP

Proxy H

TTP Proxy HTTP

Puppet Server

Deployment Server

Configure

Configure

Configure

Source Control Management Server (Git)

ESB

Web Service C

all

Public Network

Private Network

Deploy

Pull

Nagios (Monitoring)

Graphite (Performance

Stats)

Dev

CIS

Commits Code

Starts

Deployments

Writes Scripts

Writes Modules

Writes M

onitoring Checks

Monitor

Col

labo

ratio

n

Grails Application and Support Architecture

https://app.ucr.edu/

Process Improvement

Process Improvement• What are the strengths/weaknesses of current

process?

Process Improvement• What are the strengths/weaknesses of current

process?

• What is the target state?

Process Improvement• What are the strengths/weaknesses of current

process?

• What is the target state?

• Identify technologies/tools that could help

Legacy Deployment Process

Legacy Deployment Process• Developer writes code and deploys

Legacy Deployment Process• Developer writes code and deploys

• Developer has complete control

Legacy Deployment Process• Developer writes code and deploys

• Developer has complete control

• Little to no logging

Legacy Deployment Process• Developer writes code and deploys

• Developer has complete control

• Little to no logging

• Potential issues with coordination

Legacy Deployment Process• Developer writes code and deploys

• Developer has complete control

• Little to no logging

• Potential issues with coordination

• Infrastructure (Dev Ops) not in the loop

New Deployment Process Requirements

New Deployment Process Requirements

• Developers must be empowered to work asynchronously from Dev Ops - BUY IN!

New Deployment Process Requirements

• Developers must be empowered to work asynchronously from Dev Ops - BUY IN!

• Must be logged and auditable

New Deployment Process Requirements

• Developers must be empowered to work asynchronously from Dev Ops - BUY IN!

• Must be logged and auditable

• Automation! Automation! Automation!

New Deployment Process Requirements

• Developers must be empowered to work asynchronously from Dev Ops - BUY IN!

• Must be logged and auditable

• Automation! Automation! Automation!

• Implicit notification of events

New Deployment Process Requirements

• Developers must be empowered to work asynchronously from Dev Ops - BUY IN!

• Must be logged and auditable

• Automation! Automation! Automation!

• Implicit notification of events

• Close collaboration between developers and dev ops teams

Application Delivery Cycle

Application Delivery Cycle

Develop

Build

Deploy

Monitor

Application Delivery Cycle

Develop

Build

Deploy

MonitorGraphite

Develop

Develop• Gitlab

Develop• Gitlab

• Git server (Source Control Management)

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews• Merge requests

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews• Merge requests• Open source

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews• Merge requests• Open source• On premise or cloud

Develop• Gitlab

• Git server (Source Control Management)• Very “Github-like”• Web Front End• Code reviews• Merge requests• Open source• On premise or cloud

Develop

Develop• Slack

Develop• Slack

• Collaboration tool

Develop• Slack

• Collaboration tool• Chat

Develop• Slack

• Collaboration tool• Chat• Integrations galore!

Develop• Slack

• Collaboration tool• Chat• Integrations galore!• Free w/ special EDU pricing for advanced

features

Develop• Slack

• Collaboration tool• Chat• Integrations galore!• Free w/ special EDU pricing for advanced

features

Develop

Developer Git Server (SCM)

Collaboration

Developer

Developer

Develop

Developer Git Server (SCM)

Collaboration

Push

Developer

Developer

Develop

Developer Git Server (SCM)

Collaboration

Push

Notify

Developer

DeveloperReceive

Develop

Developer Git Server (SCM)

Collaboration

Push

Developer

Developer

Develop

Developer Git Server (SCM)

Collaboration

Push

Developer

Developer

Pull

Build

Build• Gitlab, Slack

Build• Gitlab, Slack

• Jenkins

Build• Gitlab, Slack

• Jenkins• Continuous Integration Server (CI) - Builds

Build• Gitlab, Slack

• Jenkins• Continuous Integration Server (CI) - Builds• Can be triggered by branch push

Build• Gitlab, Slack

• Jenkins• Continuous Integration Server (CI) - Builds• Can be triggered by branch push• We use special branch names - release/x.y.z

Build

Build• Gitlab, Slack

Build• Gitlab, Slack

• Nexus

Build• Gitlab, Slack

• Nexus• Maven artifact repository

Build• Gitlab, Slack

• Nexus• Maven artifact repository• Jenkins publishes built wars here

Build• Gitlab, Slack

• Nexus• Maven artifact repository• Jenkins publishes built wars here• Can contain other artifacts (jars, zips)

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

file.groovy

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

file.groovyPush

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

file.groovyPush

NotifyReceive

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

file.groovyPush

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

file.groovyPush

Build Webhook

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration file.groovy

Push

Pull

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

Push

Pull

Build WAR

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

Push

Pull

Notify

Build WAR

Receive

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

Push

Pull

Build WAR

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

Push

Pull

Publish

Build WAR

Build

Developer

Git Server (SCM)

CI (Build) Server

Artifact Repository (Maven)

Collaboration

Push

Pull

Publish

Build WAR

Deploy

Deploy• Slack, Nexus

Deploy• Slack, Nexus

• Rundeck

Deploy• Slack, Nexus

• Rundeck• Job scheduler and runbook automation

Deploy• Slack, Nexus

• Rundeck• Job scheduler and runbook automation• Jobs can be prebuilt and shared

Deploy• Slack, Nexus

• Rundeck• Job scheduler and runbook automation• Jobs can be prebuilt and shared• Auditable

App Server 2

App Server 1

Deploy

Developer

Artifact Repository (Maven)Collaboration

Job Scheduler/Deployment Server

App Server N

App Server 2

App Server 1

Deploy

Developer

Artifact Repository (Maven)Collaboration

Job Scheduler/Deployment Server

App Server N

Run

App Server 2

App Server 1

Deploy

Developer

Artifact Repository (Maven)Collaboration

Job Scheduler/Deployment Server

App Server N

Run Execute

App Server 2

App Server 1

Deploy

Developer

Artifact Repository (Maven)Collaboration

Job Scheduler/Deployment Server

App Server N

Run Execute

Download WAR

App Server 2

App Server 1

Deploy

Developer

Artifact Repository (Maven)Collaboration

Job Scheduler/Deployment Server

App Server N

Run Execute

App Server 2

App Server 1

Deploy

Developer

Artifact Repository (Maven)Collaboration

Job Scheduler/Deployment Server

App Server N

Run Execute

NotifyReceive

Monitor

Monitor• Slack

Monitor• Slack

• Nagios

Monitor• Slack

• Nagios• Ping test

Monitor• Slack

• Nagios• Ping test• Application availability

Monitor• Slack

• Nagios• Ping test• Application availability

Monitor• Slack

• Nagios• Ping test• Application availability

• Graphite

Monitor• Slack

• Nagios• Ping test• Application availability

• Graphite• Application Metrics

Monitor• Slack

• Nagios• Ping test• Application availability

• Graphite• Application Metrics• JVM Performance

Monitor

Monitor• Logging

Monitor• Logging

• ELK stack

Monitor• Logging

• ELK stack• Log Stash (Log Collection)

Monitor• Logging

• ELK stack• Log Stash (Log Collection)• Elastic Search (Log Indexing, Searching)

Monitor• Logging

• ELK stack• Log Stash (Log Collection)• Elastic Search (Log Indexing, Searching)• Kibana (Visualization)

Monitor• Logging

• ELK stack• Log Stash (Log Collection)• Elastic Search (Log Indexing, Searching)• Kibana (Visualization)

• Splunk Killer?

Monitor• Logging

• ELK stack• Log Stash (Log Collection)• Elastic Search (Log Indexing, Searching)• Kibana (Visualization)

• Splunk Killer?• Check out Nick Turley’s talk:

“Security Data Analytics Platform” tomorrow at 9am!

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Graphite

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Log Messages

Graphite

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Log MessagesView

Graphite

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

View

Graphite

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

View

Graphite

Monitor

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Notify

Receive

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Receive

View

Graphite

Monitor

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Notify

Receive

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Receive

View

Graphite

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Notify

Receive

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Receive

View

GraphiteMetrics

Monitor

App Server 2

App Server 1Developer

Collaboration

App Server N

Notify

Receive

Dev Ops

Log Collection and Visualization (ELK)

Monitoring

Performance Metrics

Receive

View

GraphiteView

Metrics

Demo Time

Outcomes

Outcomes• Still early

Outcomes• Still early

• Buy-in was quick

Outcomes• Still early

• Buy-in was quick

• Good feedback from developers and dev ops teams

Outcomes• Still early

• Buy-in was quick

• Good feedback from developers and dev ops teams

• Many of the tools deployed used for non-development projects (New SIS)

Outcomes• Still early

• Buy-in was quick

• Good feedback from developers and dev ops teams

• Many of the tools deployed used for non-development projects (New SIS)

• All tools deployed are free and open source!

QuestionsMike Kennedy

michael.kennedy@ucr.edu

Twitter: @mike_kennedy