Going Enterprise? An Introduction to Enterprise Application Management

Stephen Feldman (sfeldman@blackboard.com)

Director Blackboard Performance Engineering

07/10/07

2

Who is this presentation geared for?

• System Administrators

• Functional Administrators

• Helpdesk/Support Personnel

• Project Champions

• Anyone at the end of the day who is accountable for the success of your institution’s Blackboard implementation.

3

• Better understanding of the various Blackboard architectural components.

• Tips to point you in the right direction to manage various components of the stack.

• Direction of troubleshooting issues that might occur.

• Ideas around planning.

• Tools that could be of useful assistance.

What will you attain from this presentation?

4

• Are race car drivers really professional athletes?• What’s a flying buttress?• Chemical reaction of adding water to baking

soda• Playing Inspector Gadget• Rocky wasn’t a boxing champion, he was a

planning champion• Fire is truly the most essential tool for mankind

Presentation agenda

5

• Managing an enterprise application takes a lot of work, planning and coordination.

• Applications aren’t for techies, but for real users.– Users are the greatest stakeholders that are often forgotten.– Where do they fall from a priority perspective?

• The application should run in a manner that’s seamless to your users.– Your users should be aware, but not invested in your success.– You should be invested in the success of your users.

• It doesn’t require formal training…– You do not have to be an accomplished DBA, network architect or

application administrator to be successful.• These resources should be available to assist you in time of need.

– You need to understand basic premises about the application from both the business and technical perspective.

• Understanding the business objectives can make the technical aspects of the position quite easy.

Going enterprise doesn’t mean going PRO

6

Where does your institution fit?

Level 1 Level 2 Level 3 Level 4 Level 5

Reactive and

Exploratory

MonitorAnd

Instrument

PerformanceOptimizing

BusinessOptimizing

ProcessOptimizing

7

Performance Maturity Model

• Most institutions fall under Level 1– Less then 20% qualify for Level 2

• Mostly an issue over resources

– Less then 2% qualify for Level 3– Less then 1% fall in the range of Level 4 or 5

• Why?– Cost– Capabilities/Resources– Level of Effort

8

Performance Maturity Model

• Implementing most of the Blackboard Reference Architecture components will help clients transcend to Level 2 and 3 taking into account:– Cost– Capabilities/Resources– Level of Effort

• Essentially we have identified components that focus on improving operational performance.

9

What is the Blackboard Reference Architecture?

10

Dell Multi-Purpose Reference Architecture

11

Dell Multi-Purpose Reference Architecture

Disaster Recovery:ML 60204 LTO 3 Drives 36 SlotsSAN Attached

PowerConnect2624

1 3 5 7 9 11 13 15 17 19 21 23SPD/LNK/ACT

FDX/HDX

2 4 6 8 10 12 14 16 18 20 22 24SPD/LNK/ACT

FDX/HDXPOWER

1 3 5 7 9 11 13 15 17 19 21 23

2 4 6 8 10 12 14 16 18 20 22 24 24

Load Balance: F5 BigIP 1500 Application Switch.

PowerConnect 2716 16 Port GigE Switch

Application Servers NFS or UNC mountBlackboard running on top of W2K3, IIS & Tomcat JavaBlackboard on top of Red Hat ES 44 – 1950s with 2 Dual Core 3.00MHz 4MB Cache 8GB RAM2 - 146 GB 10K SAS Raid 1

Database ServerMicrosoft SQL Server 2005 Enterprise Oracle 10g Standard for 4 Socket6850 with 4 Dual Core 3.4MHz 16MB Cache 16GB RAM5 - 300GB 10K SAS/SATA Raid 1/5 Hot spare

iSCSI FC SAN StorageCX3-20 iSCSI Fibre ChannelOptional 4+4 iSCSI Ports15 – 146GB FC DrivesImplementation for 2 directconnected Hosts

PowerVault136T

Backup Server1950 with 2 Dual Core 2.66MHz 4MB Cache 4GB RAM2 - 146 GB 10K SAS Raid 1

12

Sun Multi-Purpose Reference ArchitectureOracle and Oracle RAC on SUN

Reference Architecture

Gig-E Switch(separate subnet)

Gig-E Switch Private Interconnect(separate subnet)

Public Network

Storage Network

Oracle Cluster Interconnect Network

Attempt to bond interconnect NICs

Sun T1000 6 to 8-core

Sun T2000 6 to 8-core

Sun T1000 6 to 8-core

Sun T2000 6 to 8-core

Sun T1000 6 to 8-core

Sun T2000 6 to 8-core

Gig-E Switch

Application Tier

Database Tier Virtual IP addresses

Netscaler7000Citrix Netscaler 7000

NetApp FAS Appliance

Sun Modular Blade Units (Niagara: T1)

Storage Tier

13

Going enterprise means you understand the ingredients…

Apache or IIS

Oracle or SQL Server

Tomcat Application Container (Clusterable)

Mod_JK

Perl_Ex or Mod_Perl

Java 1.5

Blackboard Academic Suite

Oracle or SQL Server

WebLogic Application Container (Clusterable)

Java 1.5

Blackboard Academic Suite: Vista/Campus Edition

14

Alternative Performance Configurations: Multi-Home Clustering

AS Node 1

AS Node n

AS Node 3

AS Node 2

Apache Web Server

Single Physical Server

HTTP Traffic

Vista Admin Node

Vista Node n

Vista Node 2

Vista Node 1

Hardware Load Balancer

Single Physical Server

HTTP Traffic

Academic Suite Vista/CE Learning System

15

Alternative Performance Configurations: Distributed Load-Balancing and Clustering

Physical Server n

Physical Server 3

Physical Server 1

Physical Server 2

Vista Admin Node

Vista Node n

Vista Node 2

Vista Node 1

Hardware Load Balancer

HTTP Traffic

Hardware Load Balancer

AS Instance

Apache

Server 1

AS Instance

Apache

Server 2

AS Instance

Apache

Server 3

AS Instance

Apache

Server n

HTTP Traffic

Academic Suite Vista/CE Learning System

16

Alternative Performance Configurations: Server Virtualization (Linux/Windows)

Virtual M

achine

Virtual M

achine

Virtual M

achine

Virtual M

achine

2x2 Physical Server

Physical Memory Pool

vCP

U

vCP

U

vCP

U

System Hypervisor Kernel

Network Switch

CPU Core CPU Core CPU Core CPU CoreNetwork Attached

StorageVirtual Interfaces Virtual InterfacesVirtual InterfacesVirtual Interfaces

vCP

U

vCP

UvC

PU

vCP

U

vCP

U

Re

served

RA

M

Re

served

RA

M

Re

served

RA

M

Re

served

RA

M

Physical Interfaces

17

Alternative Performance Configurations: Server Virtualization (Solaris)

18

Going enterprise means knowing enough to be dangerous…• Understand the components of the architecture.• Learn about their purpose and function

– All of the vendors have well documented knowledge base resources for both developers and system administrators.

– They have support sites for common issues that reveal themselves in log messages.

• Reminder: Know where these systems log messages.• Tip: Google isn’t always your friend, but often the vendors

who develop our components have great KB areas.

• Start simple…learn basic commands and progress to advance administrative functions.

19

Going enterprise means you build up knowledge in key areas…• How well do you know the traffic behavior on the

system?– What’s the busiest hour of the day?– What’s the busiest day of the week?– What’s the busiest week of the year?– Are there key events to be aware of that affect user behavior?– How long do users spend in the system?– Where do users spend their time?– What operations do they perform the most?

• How does behavior correlate to system resource utilization?

• How well do you understand growth from user generated data and system generated data?

• What is your institution’s policy on retaining data?

20

Going enterprise means you know how to work your way out of a brown paper bag…

• Issues will occur…be prepared.• Identify a methodology for problem resolution• Characteristics of a good methodology for resolving

issues:– Reliable– Deterministic– Practical– Finite– Predictive– Efficient– Impact Aware– Measurable

21

• Operations guide and plan for managing the system– Detailed information about the application

environment.– Summary of integration with external systems– Hardware diagrams, invoices, maintenance numbers.– Personas about roles and responsibilities– Log of activity, work completed, upgrades,

maintenance.– Routine Operations

• Daily, Weekly, Monthly, Quarterly, Annually or Other– Summary of support tickets and issues

Going enterprise means you are a planning champion…

22

• Capacity planning guides should contain…– Manage your deployment architecture– Handling adoption and growth– Archiving data– Maintenance windows and tasks– Integrating with external systems– Redundancy and failover– Business processes– Upgrades– Rolling out new features

Going enterprise means you are a planning champion…

23

• Disasters are unpredictable, but you can be proactive with your planning to minimize the impact

• Put a plan together…Practice the plan (regularly)– Introduce new employees to the plan– Interview exiting employees to address the plan

• Get buy-in from your team and your boss• Put disaster recovery in your team’s budget• Build-in capabilities for redundancy and failover

Going enterprise means you have a disaster recovery plan…

24

• An enterprise system does not take care of itself.• An enterprise system runs 24 x 7 (365 days a

year)– Ask yourself, do you work 24 x 7 (365 days a year)?

• Administrative responsibilities from a time, event and/or milestone perspective should be documented and practiced.– Processes should be repeatable and reliable.– Responsibilities should be clearly defined.– Processes should have workflows– Workflows have decision trees

Going enterprise means you have operating procedures…

25

• Essential operational tools– Ticket/Issue Tracking System– Knowledge Management System

• Essential administrative tools– Application management tools– Database management tools

• Performance management tools– Profiling and debugging tools– Log analytics and behavior modeling tools

• Quality Assurance Process– Development/Test Environment– Test Cases and Test Procedures

• Test Acceptance– Daily/Hourly Checks

• Is the application environment running?

Going enterprise means you have tools to make your job easier…

26

Want More?

• To view my resources and references for this presentation, visit

www.scholar.com• Simply click “Advanced Search” and

search by s_feldman and tag: ‘bbworld07’

27

Questions?

Steve FeldmanDirector, Software Performance Engineering and Architecturesfeldman@blackboard.com